Persistent Memory for Long-Running Agents

This article was revised in October 2026 to describe memorg as it is implemented; an earlier version described memory types and pipelines it does not have.

Most LLM agents are stateless. Each session starts from zero. The user re-explains their preferences, re-establishes context, re-provides background that was already discussed yesterday. The model has no deficit — it can process all of this again. But the user does, and the system is doing redundant work.

The problem is not that LLMs cannot handle long contexts. Modern models accept very long inputs. The problem is that context is ephemeral. When the session ends, the context window is discarded. The next session starts empty. Whatever the agent learned about the user, the project, or the domain during the previous interaction is gone.

memorg is our project exploring what a persistent memory layer for LLMs looks like in practice. This post describes the problem space, the architecture memorg actually uses, and the open questions we are still working through.

Context vs. Memory

These terms are used loosely in the LLM literature, and the conflation causes confusion. We draw a sharp distinction:

Context is the information available to the model within a single inference call. It is the content of the context window — the system prompt, the conversation history, any retrieved documents, and the current user input. Context is ephemeral. It exists for the duration of one API call and is discarded afterwards.

Memory is the information that persists across inference calls and across sessions. It is stored externally to the model — in a database, a file, a vector store — and selectively loaded into context when relevant. Memory has a lifecycle independent of any single conversation.

The relationship between them is straightforward: memory is the persistent store; context is the active working set loaded from that store (plus the current conversation).

┌─────────────────────────────────────────┐
│              Memory Store               │
│  (persistent, structured, queryable)    │
│                                         │
│  ┌──────────────┐  ┌───────────────┐   │
│  │ Conversation │  │ Documents,    │   │
│  │ history      │  │ notes,        │   │
│  │ (exchanges)  │  │ entities      │   │
│  └──────────────┘  └───────────────┘   │
│         │                  │            │
│         └────────┬─────────┘            │
│                    │ retrieval           │
│                    ▼                    │
│  ┌─────────────────────────────────┐    │
│  │        Context Window           │    │
│  │  (system prompt + retrieved     │    │
│  │   memories + conversation +     │    │
│  │   current query)                │    │
│  └─────────────────────────────────┘    │
└─────────────────────────────────────────┘

This distinction matters because the design constraints are different. Context is bounded by the model’s context window and priced per token. Memory is bounded by storage and retrieval quality. Mixing them up leads to architectures that either stuff everything into context (expensive, eventually exceeds the window) or store nothing persistently (amnesic agents).

What memorg Stores

The cognitive-science literature divides memory into semantic memory (facts), episodic memory (experiences) and procedural memory (skills). It is a useful lens for thinking about agents, and it is tempting to build three stores to match. memorg does not. It makes a plainer, more conservative choice, and it is worth being precise about what that choice is.

Conversation history as a hierarchy. Chat is organised four levels deep:

Session (a user's workspace)
└── Conversation (a chat thread)
    └── Topic (a specific subject)
        └── Exchange (a user message and the system's reply)

Each level carries metadata. Conversations and topics have titles, summaries and embeddings; topics record key entities; each exchange has an embedding and an importance score, and its messages carry parsed content — entities, intents and sentiment — alongside the raw text. The hierarchy matters for retrieval: a search can be scoped to a session, a conversation, a topic, or everything.

Everything else as memory items. For material that is not a chat turn — documents, notes, entities, or a custom type of your own — memorg has a generic memory abstraction. A memory item has content, a type, tags, optional parent/child links, an importance score and an embedding:

doc = await system.create_memory_item(
    content="Meeting notes: decided to use PostgreSQL for prod",
    item_type="document",
    tags=["meeting", "database", "decisions"],
)

results = await system.search_memory(
    query="database decisions",
    item_types=["document", "note"],
    tags=["decisions"],
)

Both worlds share the same storage and the same vector index, so one database holds a user’s conversations and their notes side by side.

If you want semantic, episodic and procedural distinctions, you can express them on top of this — as item types, tags or topics — but the library does not enforce them, extract them or retrieve them differently. That is a deliberate scope decision, and one of the open questions below.

Storage Architecture

memorg’s storage is two components in one place.

SQLite for records. The chat hierarchy is stored in four tables — sessions, conversations, topics and exchanges — each holding JSON documents. Each table has an FTS5 full-text mirror kept in sync by triggers, so every record is keyword-searchable as well as readable with ordinary SQL.

USearch for vectors. Embeddings come from OpenAI and are indexed with USearch: by default 1,536-dimensional f32 vectors with a cosine metric. The index is persisted to a .usearch file next to the database, and the vector metadata — the id, the raw vector and the text — lives in a table inside the same SQLite file.

┌───────────────────────────────────────┐
│             memorg storage            │
├───────────────────┬───────────────────┤
│ SQLite            │ USearch           │
│ sessions,         │ HNSW index        │
│ conversations,    │ (cosine, f32)     │
│ topics, exchanges │                   │
│ + FTS5 mirrors    │ vector metadata   │
│ + memory items    │ kept in SQLite    │
└───────────────────┴───────────────────┘

We chose an embedded database over a dedicated vector service on purpose. For a single user’s memory — a long history, but not millions of records — a file you can copy, inspect and back up is worth more than horizontal scale. memorg publishes no benchmark of retrieval speed or quality at any scale, so if scale matters to you, measure it on your own data.

Both components sit behind small protocols — a StorageAdapter with write, read, query, delete and get_stats, and a VectorStore with add_vector, search_nearest, delete_vector and get_stats — so either can be swapped for another backend without touching the rest.

The Retrieval Problem

Storage is the easy part. Retrieval is where the interesting problems live.

Given an incoming message, which memories should be loaded into context? Loading too many wastes the context budget and can degrade answers, as the model attends to irrelevant material. Loading too few means the agent behaves as if it has forgotten things the user expects it to remember.

memorg’s search_context works in four steps:

1. Embed the query. A query processor turns the user’s text into an embedding plus parsed metadata.
2. Vector search. USearch returns nearest neighbours, over-fetching to twice the number of results requested to leave headroom for re-ranking.
3. Keyword fallback. If the vector search returns fewer results than requested, an FTS5 keyword search runs across the four chat tables to fill the gap.
4. Multi-factor ranking. The merged candidates are ranked by a scorer that combines semantic similarity, temporal relevance and importance; by default the weights are 0.4, 0.3 and 0.3.

Each result records how it matched — keyword, semantic, temporal, hybrid or importance — which makes it possible to see why a memory was surfaced rather than guess.

User query
    │
    ▼
┌────────────────┐
│  Embed query   │
└───────┬────────┘
        ▼
┌────────────────┐     too few results?    ┌──────────────┐
│ USearch nearest│ ───────────────────────▶│ FTS5 keyword │
│  neighbours    │                         │   search     │
└───────┬────────┘                         └──────┬───────┘
        └──────────────────┬──────────────────────┘
                           ▼
                ┌────────────────────┐
                │ Multi-factor score │
                │ (semantic, recency,│
                │  importance)       │
                └────────────────────┘

Fitting Memory into Context

Retrieval returns candidates; something still has to decide what fits. memorg separates this into two components.

The context manager prioritises and compresses. Its default prioritisation strategy weights items by recency; its default compression strategy is extractive summarisation; and it maintains a token-budgeted working memory, which defaults to 4,096 tokens in MemorgSystem.

The context-window optimiser builds the prompt. Progressive summarisation collapses long content while keeping its head and tail intact, and a token optimiser uses tiktoken’s cl100k_base encoding to fit the result under a budget.

Both prioritisation and compression are protocols, so a different strategy — say, one that boosts items tagged as decisions — is a class with one method rather than a fork of the library.

Using It

memorg runs three ways. As a library, as above. As an interactive CLI — run memorg with an OpenAI key set and chat, with commands to search conversation history (search), search all memory items (memsearch), and add tagged notes (addnote). And as an MCP server, memorg-mcp, which exposes sessions, conversations, exchanges and search as tools, so that an MCP client such as Claude Desktop can use memorg as its external memory.

Memory Formation and Forgetting

Retrieval assumes memories exist. In memorg they are created explicitly: your application adds exchanges and memory items, and memorg embeds, indexes and scores them. There is no background process that reads finished sessions and decides what to remember.

That is a conscious limitation, and it is where the interesting research sits:

Extraction. Turning a conversation into durable facts — “the project uses PostgreSQL” — needs a model, and the extraction can be wrong. Misattributed facts are worse than missing ones, because they are retrieved with confidence.

Contradiction. When a new fact contradicts an old one, the old one should be superseded rather than silently deleted, so the agent can still answer “what did we used to think?”. memorg’s records carry timestamps, which makes this possible; it does not do it for you.

Forgetting. Recency already lowers the weight of old material at ranking time. Deliberate decay, and consolidation of many related episodes into one summary, are open design problems rather than features.

Open Questions

Privacy and access control. Sessions belong to a user, but shared memory spaces — team knowledge — with access boundaries are an open design problem.

Memory accuracy. Even without automatic extraction, summaries and parsed entities can be wrong. We do not have a systematic way to validate a memory store’s accuracy beyond user correction.

Evaluation. How do you measure whether a memory system is working? Retrieval precision and recall against human judgements is the obvious answer, and the ground truth is expensive to build. We have not published such an evaluation for memorg.

Typed memory. Whether semantic, episodic and procedural memory deserve different storage and retrieval — or whether tags and hybrid search are enough — is a question we would rather answer with measurements than with an architecture diagram.

Cross-agent memory. When several agents write to one store, consistency becomes a concern: two agents can record contradictory facts about the same user.

Conclusion

Persistent memory turns agents from stateless tools into something closer to collaborators. The user does not need to re-establish context every session, and the agent accumulates knowledge about their projects, preferences and decisions. The agent gets more useful over time not because the model improves, but because the memory layer supplies more relevant context.

memorg is our attempt to build that layer with plain, inspectable parts: a conversation hierarchy, generic memory items, SQLite with full-text search, a USearch vector index, and hybrid ranking you can read. It is a research project, not a finished product, and the open questions outnumber the answered ones. But the core proposition — that memory and context are distinct concerns requiring distinct architectures — is one we are confident in.

The code is available at github.com/Skelf-Research/memorg.