The Rise of Agentic Memory: Beyond Context Windows
For most of the past two years the headline number on every model release was the context window. A million tokens, then two million. The message underneath was that a model that could hold more of the conversation at once was a better model. The quieter claim, that a big enough window makes memory infrastructure unnecessary, does not survive contact with how these models actually behave.
A long context is not memory. Even when a full history fits, the model degrades as the window fills. LangGraph’s own documentation puts it plainly: long-context models get distracted by stale or off-topic content, respond more slowly, and cost more per call. A model can technically read a 300,000-token transcript. That is not the same as remembering what in it mattered.
The real frontier is a memory layer that behaves like human memory, split into two tiers. A small working memory for the current task, and a durable store for everything the agent should carry across sessions. I am not imposing this metaphor on the field. It is the design vocabulary the frameworks themselves use, and it traces back to an actual paper.
The vocabulary comes from cognitive science
CoALA, short for Cognitive Architectures for Language Agents, is the academic anchor. Published in 2023 by Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths, it proposes a language agent with modular memory components, a structured action space, and a generalized decision loop. The memory components map onto the ones cognitive science already uses: semantic memory for facts, episodic memory for experiences and past actions, procedural memory for instructions and the system prompt.
LangGraph’s documentation adopts this taxonomy nearly verbatim. Facts, experiences, instructions. It cites CoALA as precedent and then adds the honest caveat that the analogy is not perfect. That caveat matters, and I will come back to it. For now the narrower point is that the words the frameworks throw around, episodic and semantic, are borrowed from a research tradition rather than invented by a marketer.
MemGPT invented the hierarchy
The first working implementation of the idea was MemGPT, out of UC Berkeley in October 2023. Its contribution was a specific mechanism called virtual context management. Treat the model’s context window as RAM and an external store as disk, and when something needs to move between the two, use an interrupt, the way an operating system pages memory.
MemGPT was renamed Letta, and the rename tells you how the idea matured. Letta now describes itself as a platform for stateful agents, and its interface splits memory into core memory blocks, which stay in context, and archival memory, which lives outside it and gets paged in on demand.
The frameworks converged on two layers
MemGPT’s RAM-and-disk split became the default shape across the field. Almost every framework now separates short-term, thread-scoped state from long-term, cross-session knowledge. The names differ. The structure does not.
LangGraph calls the first layer checkpointers: they persist the state of a single thread so an agent can resume where it stopped. The second layer is stores, cross-thread data that outlives any one conversation. CrewAI did something more interesting. It replaced four separate memory classes, short-term, long-term, entity, and external, with one unified Memory API. An LLM inspects each saved item and infers its scope and importance, and recall is scored on semantic similarity plus recency plus importance.
OpenAI’s path is the clearest tell. Its Swarm library was deliberately stateless, specifying that it saves no state between calls. The production successor, the Agents SDK, ships a feature called Sessions, described as a persistent memory layer. A company that sold statelessness as a design virtue now ships memory as a first-class feature.
The substrate war
Underneath the two-layer split sits a real design fight about what the durable store should be. There are two camps.
The first uses vector embeddings. Mem0, at roughly 64,000 GitHub stars, describes itself as a universal memory layer. It extracts salient facts from conversations, consolidates them, and retrieves them by semantic similarity. Its paper reports a 26% relative improvement over OpenAI on the LOCOMO long-conversation benchmark, 91% lower p95 latency, and more than 90% token savings compared to feeding the full context back in. LangMem is the same idea wired into LangGraph’s store: search and manage tools on the hot path, plus a background manager that consolidates memories offline.
The second uses a knowledge graph. Microsoft’s GraphRAG, at around 36,000 stars, builds a graph index over private data for retrieval, but the project is in maintenance mode and its own README warns that indexing is expensive. The more interesting entry is Graphiti, from Zep. Graphiti is a temporal knowledge graph: it records facts along with when each was true, so an agent can answer what changed rather than only what is. It preserves provenance and supports an ontology that can be prescribed or learned.
The temporal graph is the synthesis. Flat embedding stores are fast and fuzzy. Knowledge graphs are relational and auditable but expensive to build. A temporal graph tries to keep the relational structure while answering the question the others dodge: how do facts change over time.
One detail keeps the two-camp framing honest. Mem0’s own paper describes a graph-based variant that scores a couple of points higher than the base vector approach. The flagship embedding store is already reaching for graph structure. The war is not as clean as the labels suggest.
Memory is the part that persists
For the reader this site is written for, a self-hosted operator building agents, memory is where the lock-in concentrates. An orchestration framework is something you can swap. A memory store is not. It is the part of the agent that persists across sessions, which makes it the part you have to back up, secure, and, when something goes wrong, explain.
That is why this post fits the arc we have been building here. We wrote about single-prompt LLMs hitting a ceiling, about orchestration frameworks multiplying, and about audit trails becoming the new security boundary. Memory is the missing middle. It is what lets a long-running agent accumulate competence instead of restarting from scratch, and it is where the compliance question from those earlier posts gets a concrete answer: the record of what the agent remembers, held in a store you can point to and audit.
The caveats
Several claims in this post deserve skepticism, and I want to name which ones.
The human-memory analogy is imperfect, and LangGraph says so itself. Episodic and semantic are a useful vocabulary, not a model of how cognition works. When a vendor says an agent has episodic memory, read it as it keeps a record of past actions, not as a claim about neuroscience.
Mem0’s headline benchmark numbers come from its managed platform, which includes optimizations the open-source SDK does not ship. Self-hosted users should expect the same direction, not the same digits.
The knowledge-graph side costs real money to index, and GraphRAG is in maintenance mode. Zep’s story is a separate caution: its open-source Community Edition is deprecated, and the actively developed project is now Graphiti, which powers the commercial Zep Cloud. Open-source memory has a habit of becoming a funnel for a managed service.
Statelessness was not always a bug. Swarm’s pitch was that being stateless made coordination lightweight and easy to test. Memory-first agents trade that simplicity for durability, and it is a real trade, not a one-sided win.
CrewAI’s claim of being used by 65% of the Fortune 500 is marketing from its own homepage. And every star count here is a point-in-time signal of attention, read on August 31, 2026, not a measure of adoption or quality.
Bottom line
The context window was never going to be the thing that made agents durable. A model that can read a million tokens still forgets what it did last Tuesday unless something writes it down. The field has converged on the fix: two layers of memory, a working store and a durable one, borrowed from cognitive science and implemented either as embeddings or as a knowledge graph.
The fight that matters now is over the durable substrate, and it is unresolved. Vector stores are fast and cheap. Knowledge graphs are structured and auditable. Temporal graphs are the attempt to get both. Whichever wins, the memory layer is where an agent’s behavior stops being stateless, which is why it is also where governance and auditability have to live.
For a builder the practical takeaway is simpler than the vendor marketing suggests. Treat memory as infrastructure you own rather than a feature you bolt on. If you would not want to lose it, or would not want an auditor asking how it was built, it belongs in a store you control.
Sources
- Cognitive Architectures for Language Agents — Sumers, Yao, Narasimhan & Griffiths, arXiv, September 2023
- MemGPT: Towards LLMs as Operating Systems — Packer et al., UC Berkeley, arXiv, October 2023
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — Chhikara et al., arXiv, April 2025
- LangGraph: Memory — LangChain docs
- LangGraph: Persistence — LangChain docs
- LangMem — LangChain
- mem0ai/mem0 — GitHub
- letta-ai/letta — GitHub
- Letta: Memory — Letta docs
- CrewAI: Memory — CrewAI docs
- microsoft/graphrag — GitHub
- getzep/graphiti — GitHub
- getzep/zep — GitHub
- openai/swarm — GitHub
- OpenAI Agents SDK — OpenAI