Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

Wei-Chieh Huang; Weizhi Zhang; Yuchen Wu; Yankai Chen; Eric Hanchen Jiang; Wooseong Yang; Yiwei Yang; Henry Peng Zou

cs.CLAug 15, 2026

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

Authors: Wei-Chieh Huang, Weizhi Zhang, Yuchen Wu, Yankai Chen, Eric Hanchen Jiang, Wooseong Yang, Yiwei Yang, Henry Peng Zou,

Organizations: University of Illinois Chicago · University of Washington · McGill University · MBZUAI ·

Abstract

Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate, namely the underlying medium in which memory is represented and stored, should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability introduces a further routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents. Code will be made available upon acceptance.

Figures & tables

Figure 1: Evaluation landscape of 52 memory-augmented LLM systems (2023–2026). (a) Benchmark adoption: 62% of usage concentrates on LoCoMo and LongMemEval (b) Metric coverage: every system reports accuracy but only 21% report any efficiency metric; 81% of systems use GPT-family backbones. (c) Breadth and efficiency: half of systems use a single benchmark with no efficiency metrics; no system simultaneously achieves broad coverage and comprehensive efficiency reporting.

Figure 2: Overview of the harness evaluation . 11 memory methods (M1–M11) spanning seven substrate families are evaluated on identical interaction histories across four benchmarks, instrumented with 26 metrics in two families (performance and efficiency).

			LoCoMo		LME-S		MAB LRU		MAB TTL		MAB CR
Model	Sub.	Family	P4 $\uparrow$	$E_{15}$ $\downarrow$	P4 $\uparrow$	$E_{15}$ $\downarrow$	P4 $\uparrow$	$E_{15}$ $\downarrow$	P4 $\uparrow$	$E_{15}$ $\downarrow$	P4 $\uparrow$	$E_{15}$ $\downarrow$
Qwen3-8B	External
	M1	Flat	0.540	0.77	0.478	7.30	0.578	9.94	0.560	0.92	0.270	1.65
	M2	Flat	0.470	0.33	0.480	1.75	0.620	3.23	0.470	0.33	0.270	1.39
	M3	Text	0.562	4.18	0.495	55.58	0.563	146.65	0.640	27.15	0.240	2.30
	M4	Struct	0.435	6.36	0.430	225.67	0.493	274.06	0.480	12.65	0.270	1.50

Table 1: Memory substrates on LoCoMo , LME-S , and MAB (LRU, TTL, CR). P4

\uparrow

: LLM-judge score;

E_{15}

\downarrow

: per-query latency (s). Within each model band, green and red mark the per-column best and worst; bold marks the per-model P4 winner on each benchmark.

			ALFWorld-unseen					BigCodeBench-Hard
Model	Sub.	Family	TSR $\uparrow$	$\mathrm{Steps}\!\mid\!\mathcal{S}$ $\downarrow$	$P_{\mathrm{avg}}$ $\uparrow$	$E_{2}^{\mathrm{p90}}$ $\downarrow$	$E_{15}$ $\downarrow$	Pass@1 $\uparrow$	$P_{\mathrm{avg}}$ $\uparrow$	$E_{2}^{\mathrm{avg}}$ $\downarrow$	$E_{15}$ $\downarrow$
Qwen3-8B	-	NoMem	5.7	12.2	13.5	13.2	488	8.1	12.4	3.8	4
	M1	Flat	5.2	5.9	6.6	16.1	672	9.5	20.1	5.0	5
	M2	Flat	6.7	6.0	9.4	15.7	647	10.1	11.9	4.9	5
	M3	Text	5.2	8.3	8.4	16.3	653	8.1	12.4	5.3	5
	M4	Struct	7.5	7.9	8.5	14.8	605	10.1	20.4	4.6	5

Table 2: Memory substrates evaluated on ALFWorld-unseen and BigCodeBench-Hard . Within each model band, green marks the best value on a metric, red marks the worst.

P_{\mathrm{avg}}

is the mean per-task partial-success score. Bold marks the per-model performance winner (TSR for ALFWorld, Pass@1 for BCB). M10 is omitted from both agent-centric benchmarks because the cumulative context exceeds the tested context windows; M9 is omitted from the Gemma-4 band because it is incompatible with the A4B MoE routing.

Figure 3: Performance vs. latency across four benchmarks on Qwen3-32B-AWQ . (a) LoCoMo and (b) LongMemEval-S: absolute

P_{4}

. (c) ALFWorld-unseen and (d) BigCodeBench-Hard:

\Delta

-performance vs. latency overhead anchored at NoMem (No memory baseline).

Figure 4: Retrieval breadth $k$ scales oppositely on long-context QA versus sequential decision-making. Top: LoCoMo ( Qwen3-8B , $n{=}1986$ , $k{\in}\{1,2,5,10,20\}$ ). Bottom: ALFWorld-unseen ( Qwen3-32B-AWQ , $n{=}134$ , $k{\in}\{1,2,3,4,5\}$ ). green when the trend is in the preferred direction, red otherwise. No Mem stands for no memory baseline.

Figure 5: Response-token attention under M1 retrieval for (a) LoCoMo (b) ALFWorld.

Figure 6: MAB CR scalability on Qwen3-32B-AWQ :

P_{4}

(bars, left) and per-query latency (lines, right, log) at 6K, 32K, 262K tokens.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

System

Year

Bench.

Metrics

Base

System

Year

Bench.

Metrics

Base

Generative Agents [ 37 ]

2023

Other

Mem0 [ 32 ]

2025

A, P, T, L

MemGPT [ 33 ]

2023

DMR

Mem-

\alpha

[ 43 ]

2025

Other

A, P, M

MemoChat [ 53 ]

2023

Other

MemAgent [ 51 ]

2025

Other

MemoryBank [ 4 ]

2023

Other

MemOS [ 7 ]

2025

LC, LME

A, P

RSum [ 54 ]

2023

Other

Memoria [ 46 ]

2025

LME

A, T, L

SCM [ 55 ]

2023

Other

MemoryOS [ 56 ]

2025

A, P

Appendix

Table 3: Landscape of 52 memory-augmented LLM systems (2023–2026). Benchmark codes: LC = LoCoMo, LME = LongMemEval, MH = Multi-hop QA, DS = DialSim, DMR = DMR. Metric codes: A = accuracy, P = per-type, T = token, L = latency, R = runtime, M = memory size, K = API calls. #B : distinct benchmarks used. #E : efficiency dimensions reported (bold

{\geq}2

). Base : G = GPT-series, O = open-source. Summary : 81% GPT-family; 62% of benchmark pairs on LC + LME; only 21% report any efficiency metric; no system reaches

{\geq}3

benchmarks

\times

{\geq}3

efficiency dimensions.

Family	Method	TF	LLM	Write	Read	Mgmt
External Memory
Flat Index	M1 Dense Vector [ 20 ]	✓	✗	Encode; add to index	Dense top- $k$	—
	M2 Sparse Vector [ 21 ]	✓	✗	Tokenize; append to BM25	Sparse top- $k$	—
Text Record	M3 Gist Index [ 22 ]	✓	✓	LLM gist per page; index gists	LLM selects pages to expand	Text de-dup
Structural	M4 Evolving Notes [ 23 ]	✓	✓	LLM note per write; build links	ANN $+$ link expansion	Evolution-triggered neighbour rewrites
	M5 Dual-Level Graph [ 24 ]	✓	✓	LLM extracts entities $+$ relations; build KG	KG $+$ vector hybrid (mix mode)	Lazy batch indexing; cached graph

Appendix

Table 4: Memory substrate configurations. TF : training-free ( ✓ ) or requires weight modification ( ✗ ). LLM : auxiliary LLM required ( ✓ ) beyond final-answer generation. ‡ attention-guided utterance selection with re-prefill (Qwen3 hybrid-attention adaptation). § zero-shot LLM controller in place of PPO.

Substrate	Method	Auxiliary LLM calls	Execution time
Text Record	MemGPT [ 33 ]	2,739	16.3 h
Text Record	Mem0 [ 32 ]	8,984	11.8 h
Structural	Zep [ 31 ]	7,624	23.1 h

Appendix

Table 5: Auxiliary-LLM cost on a full LoCoMo run (ten conversations, 1,986 queries; GPT-4o-mini aux model, embedding excluded). Costs are one to two orders of magnitude above the lightest retrieval baselines (e.g. M2 BM25 issues zero aux-LLM calls and finishes in roughly ten minutes).

ID	Metric	Definition	Benchmark
Performance : answer accuracy and retrieval quality
P1 $\uparrow$	Exact Match	$\mathbb{1}[\mathtt{norm}(\hat{y})=\mathtt{norm}(y)]$ ; lowercase, strip articles, punctuation, whitespace	LC
P2 $\uparrow$	Token F1	Harmonic mean of token level precision and recall between $\mathtt{norm}(\hat{y})$ and $\mathtt{norm}(y)$	LC
P3 $\uparrow$	BLEU 1	Clipped unigram precision with brevity penalty: $\mathrm{BP}\cdot\sum_{w}\min(c_{w}^{\hat{y}},c_{w}^{y})/\|\hat{y}\|$	LC
P4 $\uparrow$	LLM Judge	LLM labels the answer CORRECT/WRONG given $(q,y,\hat{y})$ ; fraction correct in $[0,1]$	LC , MA
P5	Compression Ratio	$\|\text{input tokens}\|/\|\text{stored tokens}\|$ ; larger values indicate more lossy compression (unsigned)	All

Appendix

Table 6: Metric taxonomy for evaluating memory substrates. Two families: Performance (P1–P6, P8–P12) captures answer accuracy and retrieval quality; Efficiency (E1–E15) captures latency, storage, LLM call costs, and aggregate totals. The Benchmark column records which benchmarks officially report each metric. Abbreviations: LC = LoCoMo, MA = MemoryAgentBench, ALF = ALFWorld, BC = BigCodeBench-Hard; All denotes substrate-level metrics applicable across all four. Arrows indicate the preferred direction.

Model	Sub.	Family	P1 EM	P2 F1	P3 BLEU	P4 Judge	P6 Recall@k
Qwen3-8B	M1	Flat	0.273	0.466	0.285	0.540	0.696
	M2	Flat	0.283	0.436	0.241	0.470	0.595
	M3	Text	0.190	0.366	0.269	0.562	0.620
	M4	Struct	0.226	0.355	0.216	0.435	0.465
	M5	Struct	0.196	0.381	0.252	0.648	0.455
	M6	Hier	0.251	0.436	0.276	0.556	0.797

Appendix

Table 7: Table for different substrate and model LoCoMo Performance.

Model	Sub.	Family	$E_{2}$ (ms/q)	$E_{3}$ (ms/wr)	$E_{4}$ (ms/rd)	$E_{14}$ (s)	$E_{15}$ (s/q)	$P_{5}$
Qwen3-8B	M1	Flat	522	199	226	1,534	0.77	1.00
	M2	Flat	302	0	1	660	0.33	1.00
	M3	Text	2,292	1,663	781	8,293	4.18	5.29
	M4	Struct	169	6,210	9	12,638	6.36	0.71
	M5	Struct	20,680	5,116	9,610	55,290	27.84	0.44
	M6	Hier	683	641	195	2,758	1.39	0.62

Appendix

Table 8: Table for different substrate and model LoCoMo Latency & Quality.

Sub.	Family	$E_{1}$	$E_{5}$	$E_{6}$	$E_{7}$	$E_{8}$	$E_{9}$	$E_{10}$	$E_{11}$	$E_{12}$	$E_{13}$
M1	Flat	3.59M	0	0	889.8K	0	0	0	0	0	0
M2	Flat	173.7K	0	0	806.6K	0	0	0	0	0	0
M3	Text	257.4K	311.8K	390	11.52M	8.23M	2,040	0	0	8.54M	2,430
M4	Struct	177.6K	4.59M	3,268	96.2K	0	0	3.16M	2,312	7.75M	5,580
M5	Struct	167.8K	33.0M	9,860	6.00M	17.8M	3,980	0	0	50.8M	13,840
M6	Hier	4.33M	381.8K	910	1.87M	0	0	127.8K	190	509.6K	1,100

Appendix

Table 9: Table for different substrate LoCoMo Storage, Tokens & Calls.

			LME-S			MAB LRU			MAB TTL			MAB CR
Model	ID	Family	P12	P2	P3	P12	P2	P3	P12	P2	P3	P12	P2	P3
Qwen3-8B	M1	Flat	0.390	0.170	0.113	0.592	0.491	0.417	0.570	0.476	0.405	0.330	0.211	0.171
	M2	Flat	0.340	0.170	0.114	0.606	0.527	0.448	0.480	0.399	0.340	0.280	0.188	0.155
	M3	Text	0.123	0.047	0.029	0.549	0.479	0.407	0.840	0.544	0.462	0.270	0.192	0.163
	M4	Struct	0.303	0.145	0.094	0.465	0.419	0.356	0.490	0.468	0.397	0.290	0.121	0.103
	M5	Struct	0.380	0.160	0.103	0.549	0.491	0.417	0.580	0.468	0.397	0.500	0.362	0.308

Appendix

Table 10: P12 (SubEM), P2 (F1), P3 (BLEU-1) per (substrate, generator, sweep) for LongMemEval-S and MemoryAgentBench.

Model	ID	Family	$E_{1}$	$E_{2,\mathrm{mean}}$	$E_{2,\mathrm{p90}}$	$E_{14}$	$E_{15}$
Qwen3-8B	M1	Flat	9.7M	2,075	4,602	2,191	7.30
	M2	Flat	2.7M	1,747	3,580	525	1.75
	M3	Text	4.5M	4,500	9,000	16,674	55.58
	M4	Struct	1.6M	1,828	4,233	67,702	225.67
	M5	Struct	32.0M	8,500	18,000	56,162	187.21
	M6	Hier	27.5M	19,956	5,195	5,987	19.96

Appendix

Table 11: LongMemEval-S storage and latency per substrate.

Model	ID	Family	$E_{5}$	$E_{6}$	$E_{7}$	$E_{12}$	$E_{13}$	$P_{5}$
Qwen3-8B	M1	Flat	0	0	101K	0	0	1.00
	M2	Flat	0	0	153K	0	0	1.00
	M3	Text	437K	671	5.0M	5.3M	731	4.50
	M4	Struct	4.4M	1310	108K	4.4M	1310	6.20
	M5	Struct	10.3M	2371	375K	10.4M	2431	3.80
	M6	Hier	280K	80	46K	280K	80	2.80

Appendix

Table 12: LongMemEval-S token usage (

E_{5}

–

E_{13}

Model	Sub.	Family	P8 TSR	Steps	P10 Steps $\mid\mathcal{S}$	$P_{\mathrm{avg}}$
Qwen3-8B	–	NoMem	5.7	47.0	12.2	13.5
	M1	Flat	5.2	47.7	5.9	6.6
	M2	Flat	6.7	47.0	6.0	9.4
	M3	Text	5.2	47.8	8.3	8.4
	M4	Struct	7.5	46.9	7.9	8.5
	M5	Struct	9.0	47.1	17.4	12.5

Appendix

Table 13: Table for different substrate and model ALFWorld-unseen Performance.

Model	Sub.	Family	$E_{1}$	$E_{2}$	$E_{2,\text{p90}}$	$E_{3}$	$E_{4}$	$E_{14}$	$E_{15}$
Qwen3-8B	–	NoMem	0	12,728	13,200	–	–	65,392	488
	M1	Flat	2.17M	14,086	16,104	188	220	90,048	672
	M2	Flat	1.49M	13,747	15,692	0	1	86,698	647
	M3	Text	1.43M	13,655	16,318	1,620	754	87,502	653
	M4	Struct	942K	12,903	14,843	6,134	9	81,070	605
	M5	Struct	532K	13,167	15,702	5,021	9,412	83,080	620

Appendix

Table 14: Table for different substrate and model ALFWorld-unseen Latency & Storage.

Sub.	Family	$E_{5}$	$E_{6}$	$E_{7}$	$E_{8}$	$E_{9}$	$E_{10}$	$E_{11}$	$E_{12}$	$E_{13}$
M1	Flat	0	0	391K	0	0	0	0	0	0
M2	Flat	0	0	3.66M	0	0	0	0	0	0
M3	Text	372K	200	8.14M	1.84M	1,340	0	0	2.21M	1,540
M4	Struct	1.62M	199	5.05M	0	0	480K	380	2.10M	579
M5	Struct	1.42M	385	261	1.21M	770	0	0	2.63M	1,155
M6	Hier	357K	460	235K	0	0	84.6K	92	442K	552

Appendix

Table 15: Table for different substrate ALFWorld-unseen Token Accounting.

Model	Sub.	Family	Pass@1 $\uparrow$	$P_{\mathrm{avg}}$ $\uparrow$	$P_{5}$
Qwen3-8B	–	NoMem	8.1	12.4	–
	M1	Flat	9.5	20.1	1.000
	M2	Flat	10.1	11.9	1.000
	M3	Text	8.1	12.4	1.964
	M4	Struct	10.1	20.4	0.982
	M5	Struct	15.5	17.1	0.817

Appendix

Table 16: Table for different substrate and model BigCodeBench-Hard Performance & Quality.

Model	Sub.	Family	$E_{1}$	$E_{2}$	$E_{2,\text{p90}}$	$E_{3}$	$E_{4}$	$E_{14}$	$E_{15}$
Qwen3-8B	–	NoMem	150K	3,800	4,940	–	–	592	4
	M1	Flat	1.32M	4,976	6,631	199	183	776	5.2
	M2	Flat	245K	4,853	6,600	0.1	3	718	4.9
	M3	Text	305K	5,300	6,890	6,853	46	784	5.3
	M4	Struct	150K	4,642	6,432	137	138	747	5.0
	M5	Struct	239K	28,273	36,700	8,779	12,432	4,184	28.3

Appendix

Table 17: Table for different substrate and model BigCodeBench-Hard Latency & Storage.

Model	Sub.	Family	$E_{5}$	$E_{6}$	$E_{7}$	$E_{8}$	$E_{9}$	$E_{10}$	$E_{11}$	$E_{12}$	$E_{13}$
Qwen3-8B	–	NoMem	–	–	4.75M	–	–	–	–	–	–
	M1	Flat	0	0	2.29M	0	0	0	0	0	0
	M2	Flat	0	0	2.26M	0	0	0	0	0	0
	M3	Text	55.9K	36	2.80M	2.84M	148	0	0	2.90M	184
	M4	Struct	0	0	2.16M	0	0	0	0	0	0
	M5	Struct	0	0	40.7K	116K	148	0	0	116K	148

Appendix

Table 18: Table for different substrate and model BigCodeBench-Hard Token Accounting.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Mar 8, 2026Pengfei DuLLM Agent Memory Agent Memory Management
Hong Kong Research Institute of Technology, HongKong, China
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
Feb 26, 2026Yujie Zhao, Boqin Yuan, Junbo Huang +9Agent Memory Long-Horizon Agent Evaluation
UCSD · Recursive.

Mar 8, 2026J/K move · Enter open · S save

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

Pengfei Du

Hong Kong Research Institute of Technology, HongKong, China

LLM Agent Memory Agent Memory Management Memory-Augmented Language Model Agents LLM Context Management Memory Agent Benchmarks

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly operate in settings where a single context window is far too small to capture what has happened, what was learned, and what should not be repeated. Memory -- the ability to persist, organize, and selectively recall information across interactions -- is what turns a stateless text generator into a genuinely adaptive agent. This survey offers a structured account of how memory is designed, implemented, and evaluated in modern LLM-based agents, covering work from 2022 through early 2026. We formalize agent memory as a write--manage--read loop tightly coupled with perception and action, then introduce a three-dimensional taxonomy spanning temporal scope, representational substrate, and control policy. Five mechanism families are examined in depth: context-resident compression, retrieval-augmented stores, reflective self-improvement, hierarchical virtual context, and policy-learned management. On the evaluation side, we trace the shift from static recall benchmarks to multi-session agentic tests that interleave memory with decision-making, analyzing four recent benchmarks that expose stubborn gaps in current systems. We also survey applications where memory is the differentiating factor -- personal assistants, coding agents, open-world games, scientific reasoning, and multi-agent teamwork -- and address the engineering realities of write-path filtering, contradiction handling, latency budgets, and privacy governance. The paper closes with open challenges: continual consolidation, causally grounded retrieval, trustworthy reflection, learned forgetting, and multimodal embodied memory.

Abstract

LLM Agent Memory

Explore similar work

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications