Every team I work with has felt their AI go blank. The assistant that nailed a multi-step task on Monday cannot remember the customer it spoke to on Friday. The legal copilot that read the contract twenty minutes ago hallucinates the indemnity clause. The agent that started the day knowing your product loses the thread by 4pm.
The market’s answer arrives on cue: bigger context windows. 4K became 128K became 200K became 1M. Every release deck pitches the new ceiling as a memory upgrade. Vendors will tell you the same model can read a novel and remember your last conversation. The two are sold as one feature.
Long context and memory both look like cures for forgetfulness. They solve different kinds of forgetting, and the failure to separate them is what makes most production AI systems brittle. Long context helps with what the model can use while producing an answer. Memory helps with what survives between sessions, so a returning user does not have to start again. You can buy a bigger desk and still have no filing system.
Your 200K Window Starts Failing at 50K
In 2025, Kelly Hong, Anton Troynikov, and Jeff Huber at Chroma Research tested 18 frontier models — GPT-4.1, Claude Opus 4, Gemini 2.5, the lot — and measured how performance changed as input length grew. Every single model degraded as the input got longer, even on simple tasks.
A model with a 200K-token context window can exhibit significant degradation at just 50K tokens. That is a quarter of the advertised ceiling. Empirical tests of 1M-token models show they rarely sustain high-accuracy reasoning across more than 30 to 60 percent of capacity.
There is a position bias on top of all that. Stanford’s “Lost in the Middle” work — and Chroma’s replication — found that information buried in the middle of context gets attended to poorly, with accuracy drops of 30 percent or more. The model reads everything. It uses the start, it uses the end, and it loses the middle. Anyone who has buried a critical clause at line 247 of a 400-line prompt has seen this happen.
The advertised window is a capacity limit, not a guarantee of accurate recall throughout it.
Memory Is a Different Layer, Not a Bigger Layer
A memory system keeps information across sessions and selects what to bring back when it is relevant. Context is what the model sees while producing an answer; memory is what survives until the next interaction. For the person using the assistant, the difference is whether they have to explain the same situation again.
Researchers at UC Berkeley described this distinction in the 2023 MemGPT paper. The paper made a direct analogy: a context window is RAM — fast, limited, volatile. External memory is disk — slower, persistent, large. The LLM itself manages what to keep in active context and what to page out, using tool calls to read and write its own memory tier. The system was productionized as Letta, and it does what no context window alone can do: maintain coherent conversations across sessions, accumulate knowledge over weeks, distinguish the recent from the durable.
Enterprise architectures are catching up. The emerging two-layer pattern separates working memory — current task context, aggressively curated — from persistent knowledge: episodic memory (what happened in past interactions), semantic memory (facts and entities), and procedural memory (learned workflows). These are not optional categories. They are what “memory” actually contains in any system that operates beyond a single conversation.
Most teams call something else memory: a vector database, chunked documents, a similarity search on top. That is RAG. It is a useful component. It is not memory in the sense above. RAG captures one slice — semantic retrieval from a static store — and misses the rest. A system with RAG can look up a fact. It cannot remember what happened, learn what worked, or decide what to forget. No one would confuse RAM with disk storage. Yet that is the trade the AI market has been making — selling capacity as if capacity were architecture.
The Systems That Got It Right
The research gives teams a reason to test a separate memory layer: these systems outperform approaches that try to hold everything in context.
Google’s Titans architecture, presented at NeurIPS 2025, introduced a neural long-term memory module that is structurally distinct from the attention mechanism. It uses a “surprise” metric to decide what to store permanently — novel, unexpected information is more likely to be retained. A rough analogy to how human memory encodes the events that break our predictive models. On the BABILong benchmark, Titans outperformed GPT-4 on long-range reasoning while using far fewer parameters.
The result supports testing a separate memory mechanism rather than relying only on a larger context window.
GAM, a dual-agent memory architecture profiled by VentureBeat in 2025, made a different bet on the same intuition. One agent — the memorizer — captures and structures information during interactions, deciding what matters and how it relates to existing knowledge. A second agent — the researcher — retrieves on demand using just-in-time assembly, not precomputed summaries. On the RULER benchmark, which tracks variables over many reasoning steps, GAM hit over 90 percent accuracy where conventional RAG approaches largely failed.
The pattern repeats. Memory is not a passive store; it is an active process of selective encoding and contextual recall. The systems that win treat encoding and retrieval as separate problems, with their own agents, criteria, and quality metrics. Context engineering — careful curation of what enters the window — turns out to matter more than raw context expansion.
Three Kinds of Forgetting
I distinguish three kinds of forgetting, each requiring a different fix.
Compression forgetting happens within a session. Information drops or degrades because the context window cannot hold it well — or because rot kicks in before it fills. If doubling the window would fix the problem, this is what you have.
Session forgetting happens between sessions. The user comes back tomorrow and the AI has no idea what happened today. Doubling the window does nothing here. The next session starts fresh anyway. This requires memory across sessions.
Relevance forgetting is sneakier. The information exists somewhere — in a document, in a past chat, in a CRM field — but the system fails to surface it at the right moment. The fix is curation: a memory layer that decides what to remember, when to retrieve, and what to demote.
Most teams treat all three as one problem and reach for the most visible lever: window size. That is how you end up with a 200K-token model that degrades at 50K — more capacity you cannot effectively use, because the actual problem was different.
Limits of the distinction
Context windows are scaling so fast the distinction will collapse on its own. If attention mechanisms continue improving and Titans-style memory modules become standard in frontier models, today’s separate-memory infrastructure may look quaint in 18 months. It is a fair bet. The empirical trend points the other way: context rot has worsened with scale, not improved, across model generations. And Titans, the architecture most likely to “solve” long context, validates the thesis by adding a separate memory module rather than expanding the window. The bet against architectural separation has been on the table for three years now — three years of production systems quietly accumulating technical debt from that confusion.
RAG already solves this. Retrieval-augmented generation is widely deployed and effectively gives models access to persistent external knowledge. If RAG is “memory,” then the architectural separation argument is restating a known engineering tradeoff under a new label. For many current use cases, RAG is sufficient. However, RAG addresses semantic retrieval from a static store. It does not capture episodic memory (what happened), procedural memory (what worked), or active curation (what to forget). GAM’s 90 percent accuracy on RULER where RAG fails is the empirical wedge. As AI systems shift from single-query tools to persistent agents, RAG’s limitations become structural.
Organizations buy AI as one bundled product. This is the strongest practical objection. Most CTOs do not architect “context layers” and “memory layers” separately — they buy an Assistants API and call it done. True, today. But this is the same arc we have seen before. Databases unbundled from monolithic stores into purpose-built systems. Compute unbundled from servers into microservices. Security unbundled from perimeters into zero-trust. Each took years. Each was resisted on procurement grounds until the bundled solution started to crack. AI is at the early end of that arc, not the late end.
Another research finding complicates this distinction: in some tasks, smaller, better-selected context outperforms longer context. LongLLMLingua showed up to 21.4 percent accuracy gains on NaturalQuestions with roughly 4x fewer tokens. The boundary between “working context” and “durable artifact” is not always a clean line. The architectural separation is the right default for systems that need to operate beyond a single session. It is not a universal law.
Your Desk Is Not Your Filing System
Start with a failure someone using the system can describe. Did the assistant miss information in the current task, forget an earlier interaction, or fail to find something it had stored? The answer helps you decide which layer to change.
Classify the forgetting. Audit the AI failures in your system over the past month. For each one, ask: would doubling the context window fix this? If yes, it is compression. If no, it is session or relevance. Tally the proportions. The mix tells you where to invest. Teams that run this audit often discover that the majority of their AI failures are not context problems at all.
Design the layers. Working context (RAM) needs aggressive curation: summarize prior steps, drop stale information, shape what the model sees on each pass. The Chroma finding — that context engineering matters more than raw window size — is the guiding principle. Memory (disk) needs three sub-components: storage (what to keep), retrieval (how to find it), and curation (what to forget or demote).
Engineer what persists. The hard decision in any memory architecture is not what to store. It is what to promote and what to let decay. Use three signals: surprise (is this novel?), relevance (does it connect to ongoing goals?), and frequency (has this pattern appeared before?). Titans’ surprise metric is one concrete reference. Memory without selectivity is just a database. A growing database eventually rots the same way context does.
Measure and mature. Track context metrics within sessions: effective utilization, the threshold at which quality degrades, the density of curated input. Track memory metrics across sessions: cross-session recall accuracy, retrieval precision, knowledge accumulation rate. Without per-layer metrics, debugging is guesswork. Regulated industries — healthcare, finance, policy — have a particular interest here: when the system cannot reproduce the context behind a recommendation, it cannot support review or audit.
This is the organizational counterpart to what I wrote about in Fine-Tune Your Organization. Institutions also have working context and durable memory. The same confusion plays out at the human layer.
A Test You Can Run This Afternoon
Run the same task at 25, 50, and 75 percent of your model’s context window. Measure accuracy. If quality drops, you have a context problem. Long context is the lever.
Then come back tomorrow. Ask the system to recall something from yesterday’s session. If it cannot, you have a memory problem. Bigger windows will not help.
These tests describe two different needs. The assistant must use the material in front of it accurately, and it must retain what a returning user needs from earlier work. Measure both before treating a larger window as the answer.
Alongside context-window size, ask the vendor what persists when a user returns to the system.
References
- Hong, K., Troynikov, A., & Huber, J. (2025). “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Chroma Research. https://research.trychroma.com/context-rot
- Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). “MemGPT: Towards LLMs as Operating Systems.” arXiv / NeurIPS 2023 Workshop. https://arxiv.org/abs/2310.08560
- Behrouz, A., et al. (2025). “Titans: Learning to Memorize at Test Time.” Google Research / NeurIPS 2025. https://arxiv.org/abs/2501.00663
- VentureBeat (2025). “GAM Takes Aim at Context Rot: A Dual-Agent Memory Architecture That Outperforms Long-Context LLMs.” https://venturebeat.com/ai/gam-takes-aim-at-context-rot-a-dual-agent-memory-architecture-that
- arXiv (2025). “From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs.” https://arxiv.org/html/2504.15965v1
- Tribe AI (2025). “Beyond the Bubble: How Context-Aware Memory Systems Are Changing the Game in 2025.” https://www.tribe.ai/applied-ai/beyond-the-bubble-how-context-aware-memory-systems-are-changing-the-game-in-2025
- Fluid AI (2025). “AI Memory Systems Explained: Short-Term, Long-Term, and Working Memory in LLMs.” https://www.fluid.ai/blog/ai-memory-systems-explained
- Liu, N. F., et al. (2023). “Lost in the Middle: How Language Models Use Long Contexts.” Stanford / arXiv. https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf
- “LongLLMLingua: Accelerating and Enhancing LLMs in Long Context via Prompt Compression.” ACL 2024. https://aclanthology.org/2024.acl-long.91/
- arXiv (2025). “Contextual Memory Intelligence: A Foundational Paradigm.” https://arxiv.org/html/2506.05370v1