MemoryTax: The Bitter Lesson of Agent Memory

Zedong Peng

cs.CLOct 21, 2026

MemoryTax: The Bitter Lesson of Agent Memory

Authors: Zedong Peng

Organizations: Shanghai Jiao Tong University

Abstract

Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives; coding agents keep the raw history and let the model search it with its own tools. We ask whether designed memory still helps once the model is strong enough to search for itself. On one LoCoMo conversation (419 turns, 152 questions), we run four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, and record what every rollout costs. This is a short empirical study, with one conversation and one run per cell. Agent memory is a costly patch for a weak model: Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than a whole full-context rollout. Agent memory also has a ceiling, set by the model that writes it: a stronger reader does not recover what a weaker writer left out. With Claude Opus 5.5, Pi tops the cost-accuracy frontier, scoring 99% for less than half the cost of full context; its one miss is a wrong answer key. Part of that lead comes from photo captions that only the agents could read: on the 88 questions whose evidence has no photo, Pi and full context differ by one answer.

Figures & tables

Figure 1: Cost and correct rate per rollout. Each badge is one cell’s whole run: the build, then all 152 questions. Colour is the model, the badge the method’s mark, and a letter in the badge marks a method without one (F full context, B BM25). The stepped line is the highest correct rate reached at or below each cost, and the labelled cells lie on it. Cost is on a log axis.

Method	Stored	Model sees
Full context	raw history	all of it
BM25	raw turns	top-8 turns, $\pm$ 2
Mem0	extracted facts	top-200 facts
Zep	temporal graph	top-20 edges
Pi, Codex, Claude Code	raw history, one JSON file	what its tools read

Table 1: Methods: what each stores and what the model sees when it answers.

Figure 2: Cumulative cost–correct curves. Each curve is one rollout: the build, then the questions from cheapest to most expensive (equal costs grouped), adding up cost and correct answers; correct answers are divided by 152, so a wrong answer adds cost only. Colour is the model, the badge the method, and agents are dotted. The cost axis is linear to $2 and logarithmic above. Curves use a nine-point moving average, and the badges mark the unsmoothed totals of Table 2 . Strong lines are cells that no other cell beats on both cost and correct answers.

Figure 3: Writer and reader. Each colour is one store, written by that model. The filled badge is the writer reading its own store, and arrows run from it to the other two readers.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Model	Effort	Build	Per q.	Single	Multi	Temp.	Open	All
			Cost ($)		Correct (%)
Questions					70	32	37	13	152
Full context	GPT-4o-mini	–	$\circ$	0.0028	89	75	51	62	74
	GPT-6 Luna	high	$\circ$	0.0019	90	97	95	85	92
	Claude Opus 5.5	high	$\circ$	0.117	93	94	97	100	95
BM25	GPT-4o-mini	–	$\circ$	0.00028	86	50	73	69	74

Appendix

Table 2: Per method and model. Cost is the list price of every model call except the judge: Build is paid once, before any question; Per q. is everything one question costs, retrieval included, averaged over the 152 questions.

\circ

: nothing is built. Effort: the model’s reasoning effort; GPT-4o-mini has none. Correct: share of answers judged correct, in %, per LoCoMo category (single-hop, multi-hop, temporal, open-domain); the first row gives each category’s size.

	Mem0, read by			Zep, read by
Written by	GPT-4o-mini	GPT-6 Luna	Opus 5.5	GPT-4o-mini	GPT-6 Luna	Opus 5.5
GPT-4o-mini	78	83	84	43	43	45
GPT-6 Luna	74	80	81	52	51	55
Claude Opus 5.5	89	91	95	66	71	87

Appendix

Table 3: Writer and reader. Correct rate (%), the reader answering from the writer’s store and its retrievals. Bold: one model writes and reads (Table 2 ).

	GPT-4o-mini		GPT-6 Luna		Claude Opus 5.5
Method	Photo (64)	Other (88)	Photo (64)	Other (88)	Photo (64)	Other (88)
Full context	66	81	91	93	89	99
BM25	61	83	78	84	84	94
Mem0	70	84	72	86	91	99
Zep	33	50	53	50	77	94
Pi	53	69	88	94	98	100

Appendix

Table 4: Correct rate (%) on the 64 questions with evidence in a turn that shares a photo, whose caption only the agents could read, and on the other 88.

Figure 4: The curves of Figure 2 against turns (LLM calls, top) and tokens (bottom), on linear axes, the questions ordered by that amount. Strong lines are cells that no other cell beats on both that amount and correct answers.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 444 papers

MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation
Jun 29, 2026Kuan WangAgent Memory LLM Agent Evaluation
Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents
Mar 5, 2026Natchanon Pollertlam, Witchayut KornsuwannawitPersistent Memory for Language Models Long-Term Conversational Memory
Bricks Technology
Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents
Mar 20, 2026Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall +2Persistent Memory for Language Models LLM Memory
Memori Labs Inc.

Jun 29, 2026J/K move · Enter open · S save

MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation

Kuan Wang

Agent Memory LLM Agent Evaluation Memory Agent Benchmarks RAG Evaluation

Open paper AskPDF arXiv

Agent memory systems are increasingly evaluated against RAG and full-context baselines, but reported gains often mix changes in the memory method with changes in the language model, embedding model, or retrieval pipeline, making it unclear what is actually being measured. We present MemDelta, a controlled evaluation protocol that varies one component at a time on LongMemEval-S (500 questions, 50+ sessions, three model families). Four findings emerge: (1) verbatim RAG matches full-context GPT-4o-mini (47.2% vs. 49.8%, p = 0.34), but the ranking reverses across models: Gemini gains +14pp from full context, while Sonnet gains +31pp from RAG, partly because it refuses 63% of full-context queries; (2) swapping only the embedding model in an identical pipeline shifts accuracy by +6.2pp at n = 500 (p = 0.004), and Mem0 beats MiniLM-RAG by +11pp but loses to cloud-RAG by 1.2pp, so one variable flips the conclusion; (3) agent self-memory (42%) underperforms basic retrieval (47%); (4) on 2 of 6 question types (n = 88), Mem0 matches cloud RAG (72.7% vs. 73.9%, p = 1.0) at 50x the cost, suggesting narrow rather than general gains. We recommend memory evaluations fix embedding models across comparisons, stratify by model family, and report write-path cost before attributing gains to architecture.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation

Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents