Why Agent Memory Is Broken (And the Path to Fix It)

Three months into running a personal AI assistant, you notice the same thing every week: it doesn’t know your timezone. You told it. Twice. It’s gone.

This is not a small bug. It’s an architectural choice. The standard agent loop has two memory primitives, and both are broken for anything that has to last past a single session.

The two flavors of broken

Conversation buffer memory holds the current turn plus a sliding window of recent turns. When the window rolls, the agent forgets. A 200k context buys you longer recall — it does not buy you memory. The instant the user closes the tab, what they said is gone.

Vector memory stores embeddings of past content and retrieves the top-k similar chunks at the start of each turn. This is fine for “find a document like this.” It is not fine for “is this true.” A user’s timezone is a fact, not a similarity. Asking “is this user in UTC?” by computing the cosine distance between sentences is a category error.

Most production agents stack both: a buffer for the current session plus a vector store for retrieval-augmented generation. The result is an agent that knows what you said five minutes ago and can find documents you wrote last year, but cannot tell you what your job title is. You told it. It re-derives.

What memory actually has to do

Step back from how agent SDKs implement memory and ask what memory has to do in a long-running assistant. Four things:

1. Hold facts that survive sessions. “Alice is in UTC.” “Alice prefers mornings.” Not chat history. Facts.
2. Carry provenance. Where did this fact come from? Which session? Which message? Which extraction prompt?
3. Carry confidence and decay. “Alice said she’s in Lyon” decays faster than “Alice’s home country is the UK.” The system has to know which.
4. Carry scopes. Some facts are PHI. Some are MNPI. Some are scoped to a tenant. The fact has to know what consent reads and writes need.

A conversation buffer does none of these. A vector store does one (it carries the source chunk, sort of). Neither is enough.

Typed memory: facts as first-class objects

The fix is older than agents: store assertions, not chunks. A typed memory layer holds rows like:

Assertion(
    subject="alice",
    predicate="timezone",
    object="UTC",
    confidence=0.95,
    provenance=Source(
        session_id="s_2026_05_01",
        message_id="m_42",
        extraction_prompt="user-message-extraction-v3",
    ),
    decay=Decay.SLOW,                  # timezones don't change daily
    consent_scopes=["memory:write"],
    tags=["pii"],                      # routing for compliance
    created_at=...,
    superseded_by=None,
)

This row answers the question the buffer and the vector store can’t: “is this true now, and how confident are we.” It also has the provenance an auditor needs and the tags a compliance system needs.

You can store a million of these. You can sort by recency, filter by tag, decay by curve, and supersede by a newer assertion. You can encrypt the [pii]-tagged rows at rest. You can ship the whole thing to Postgres and bound it by tenant_id.

This is what we mean by typed memory. It is not a clever embedding trick. It is just storing the actual fact instead of storing the words around the fact.

Vector memory tells you what’s similar. Conversation memory tells you what’s recent. Neither tells you what’s true. That’s the gap typed memory closes.

Two ontologies, two lifecycles

ONTO splits typed memory into two layers because facts have two lifecycles.

UPO — User Personalization Ontology. Durable. Survives sessions. Has decay but tends to live. Examples: timezone, name, employer, account tier, preferred language. The agent loads UPO at the start of every interaction.

SMO — Session Memory Ontology. Transient. Lives inside the session. Built up turn by turn. Examples: “current trip city is Lyon,” “user is debugging the auth flow,” “intent is rebooking.” Most SMO assertions evaporate when the session ends. Some — the ones that earn durability through repetition or explicit signal — promote to UPO.

The split matters because durability is expensive. Every UPO row is something an auditor can ask about. You do not want every passing comment to land there. SMO is the scratchpad; UPO is the ledger.

Four levels of extraction

The other half of the fix is how facts get into memory. The standard agent stack is “user said something → put it in the buffer → maybe later RAG retrieves it.” That’s not extraction. That’s storage.

ONTO routes natural language to one of four extraction levels:

• session: transient context (current city, current intent, current document)
• profile: durable identity (timezone, name, preferences)
• domain: app-scoped, multi-tenant facts (this customer’s tier, this clinic’s protocols)
• tool: tool-local state (a saved query, a search filter)

The LLM proposes assertions and labels each with a level. The runtime routes by the label, writes through Policy Guard, and audits the decision. The user sees their typed graph grow. The auditor sees the source prompt and the routing decision.

This is the difference between “the model put something in memory” and “the system stored a typed fact, at a known level, under known consent, with a known source.” The first is opaque. The second is a system.

Why this matters more in 2026

Two trends make this urgent. First, agents are running longer. Personal AI, customer-facing copilots, regulated workflows — every one of these has session lengths measured in months, not turns. Conversation buffers were never built for that.

Second, the regulators arrived. HIPAA, GDPR, the EU AI Act, the FINOS AI Governance Framework. Every one of these asks “show me the data the model used and the decision it made.” A model that re-derives facts from a buffer cannot answer that. A typed memory layer answers it by construction.

The “but vectors are useful” caveat

To be clear: vectors are useful. ONTO ships vector search. The right model is both:

• Typed memory for facts you can name (timezone, tier, preferences, schedules).
• Vector memory for content you can’t pre-typify (notes, emails, documents you index for retrieval).

The mistake is using vectors for the facts. Storing “Alice’s timezone is UTC” as an embedding and retrieving it later by similarity is what’s broken. Storing it as an Assertion with a decay curve and a consent scope is what’s right.

What you give up

Typed memory has costs. You need to define a schema. You need to write extraction prompts that target the schema. You need to handle conflicts when the user changes their mind. You need to deal with decay. None of these are free.

What you get: an agent that knows who you are. An audit trail. A migration path when the schema evolves. A compliance story when the regulator calls. A foundation under whatever model you swap in next.

That’s the trade. Sometimes the right thing is also the harder thing.

Where this leaves us

The standard agent loop is good at one-shot tasks. It is bad at long-running ones because its memory primitives are mismatched to the problem. Conversation buffer memory is a side effect of how transformers see input. Vector memory is a side effect of how search engines compare documents. Neither was designed to answer “what is true about this user.”

Typed memory was designed for that. UPO + SMO are one way to do it; you could roll your own. What matters is that you stop treating the buffer as memory and the vector store as a fact ledger. They are not.

If you build an agent that is going to live longer than a session, you are going to invent typed memory eventually. The choice is whether you invent it in a hurry, the day the auditor calls, or whether you start with it.

Start with it.

---

Want to see typed memory in action? Install ONTO and run the extraction example — typed facts land in UPO with one flag, and you can query them with one line. The features page has the deeper architecture and the comparison page shows how the other SDKs handle memory differently.

Frequently asked questions

What's wrong with conversation buffer memory?

It only holds the current session. When the context window rolls, the agent forgets. There is no notion of a fact that is true now, only of recent turns.

Why aren't vector embeddings enough?

Embeddings answer 'what looks similar.' They do not answer 'is this true right now.' A user's timezone or a customer's tier is a fact, not a similarity. Storing typed assertions with provenance is the right primitive for that class of memory.

What is UPO+SMO?

User Personalization Ontology (UPO) is durable typed memory — facts with provenance, confidence, decay, and consent scopes. Session Memory Ontology (SMO) is transient context that summarizes and promotes to UPO when it earns durability. Together they replace the conversation buffer + RAG store stack with one typed memory.