MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends

Chaoqun Zhan; Qiang Zhou; Guannan Li; Zhenqiang Huang; Qianjin Wang

cs.AIAug 14, 2026

MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends

Authors: Chaoqun Zhan, Qiang Zhou, Guannan Li, Zhenqiang Huang, Qianjin Wang

Organizations: Zbyte Inc.

Abstract

Most agent-memory benchmarks test post-hoc recall, whereas MemoryArena evaluates whether memory supports interdependent, multi-session task completion. We compare MemoryLake, a structured multi-track memory backend, with Mem0, text-embedding-3-small vector RAG, and a long-context control across all five MemoryArena domains. The systems share the same agent framework, requested gpt-5-mini model alias, task samples, and scoring code; the memory integration is the intentionally changed component. Because each backend bundles write, retrieval, consolidation, budgeting, and prompt-assembly choices, the study is a matched system-level comparison, not a representation-only ablation or a cost-matched experiment. On the shared evaluation sets, MemoryLake has the highest observed success rate (SR) in mathematics (9/40), physics (12/20), and progressive retrieval (4/20). Every system has zero SR in travel planning, and web shopping yields a single bundle-level success (long context, 1/150); MemoryLake ranks third on both the travel soft process score and shopping step match. Following MemoryArena's suite-level convention, a post-hoc equal-weight average over the five SRs is 20.5% for MemoryLake versus 13.6% for the best comparator. These are point estimates: sample sizes are modest, confidence intervals overlap, and we do not report paired significance tests. A separate MemoryLake-only run over all 221 progressive queries yields a failure-counted SR of 26.7% (59/221) and is not a baseline comparison. The results support a workload-dependent view of memory backends and an observed lead among the four evaluated systems on the shared sets; they do not establish benchmark-wide state of the art or a causal advantage of representation structure.

Figures & tables

System	Stored representation	Recall and prompt behavior
Long Context	No external memory; prior within-instance trajectory retained verbatim	No retrieval; full available trajectory is flattened into the next prompt
Mem0	Extractive fact entries	Similar-fact recall through the Mem0 backend
Vector RAG	Raw trajectory chunks embedded with text-embedding-3-small	Similar-chunk retrieval
MemoryLake	Confirmed conclusions, supporting evidence, and reusable experience in separate tracks	Prioritized conclusion presence, on-demand evidence retrieval (dense encoding via bge-m3 ), and bounded assembly

Table 1: System-level differences. The experiment compares complete backend configurations rather than isolating representation structure.

Domain	Shared scale	Process metric	End-to-end metric
Formal mathematics	40 papers / 354 subproblems	Mean within-paper subproblem accuracy (PS)	Final subproblem correct for each paper (SR)
Formal physics	20 papers / 86 subproblems	Mean within-paper subproblem accuracy (PS)	Final subproblem correct for each paper (SR)
Group travel planning	30 groups / 208 member plans	Added-constraint slot satisfaction (SPS)	Every member plan in a group passes (SR)
Bundled web shopping	150 bundles / 900 purchases	Exact ASIN step match	All six purchases in a bundle match (SR)
Progressive retrieval	20 queries / 142 slots	All-slots macro-average pass rate (PS)	Final combined query correct (SR)

Table 2: Shared comparison sets and metrics. Travel and progressive retrieval use subsets of the released benchmark.

Task	Metric	MemoryLake	Mem0	Vector RAG	Long Context
Math reasoning	PS / SR	29.5 / 22.5	28.4 / 20.0	28.9 / 17.5	27.6 / 12.5
Physics reasoning	PS / SR	64.5 / 60.0	45.9 / 30.0	57.4 / 40.0	59.1 / 45.0
Travel planning	SPS / SR	43.0 / 0.0	27.2 / 0.0	45.7 / 0.0	53.1 / 0.0
Web shopping	Step match / SR	29.6 / 0.0	24.3 / 0.0	29.7 / 0.0	30.0 / 0.7
Progressive retrieval	PS / SR	6.7 / 20.0	8.9 / 15.0 ∗	5.6 / 10.0	5.3 / 10.0

Table 3: Main results in percent. Bold indicates the highest point estimate for each displayed metric. SR is zero for every system in travel planning; in web shopping the only bundle-level success is long context’s 1/150.

System	Math	Physics	Travel	Shopping	Progressive	Macro Avg SR
MemoryLake	9/40	12/20	0/30	0/150	4/20	20.5%
Mem0	8/40	6/20	0/30	0/150	3/20 ∗	13.0%
Vector RAG	7/40	8/20	0/30	0/150	2/20	13.5%
Long Context	5/40	9/20	0/30	1/150	2/20	13.6%

Table 4: Exact end-to-end success counts and the post-hoc equal-weight average of the five domain SRs. The average is descriptive and is computed on the shared sets, not the complete benchmark. Mem0’s starred progressive count is the capped re-run defined under Table 3 ; its unmodified run scores 0/20, which lowers its average to 10.0%.

Task	System	SR	95% Wilson interval
Math ( $n=40$ )	MemoryLake	9/40 (22.5%)	[12.3, 37.5]
	Mem0	8/40 (20.0%)	[10.5, 34.8]
	Vector RAG	7/40 (17.5%)	[8.7, 32.0]
	Long Context	5/40 (12.5%)	[5.5, 26.1]
Physics ( $n=20$ )	MemoryLake	12/20 (60.0%)	[38.7, 78.1]
	Mem0	6/20 (30.0%)	[14.5, 51.9]

Table 5: Exact SR fractions and marginal Wilson intervals for all five comparison domains. Rows with identical results across systems are reported in compact form. Overlapping marginal intervals do not constitute a paired significance test. The 221-query row is MemoryLake-only and is not comparable to the 20-query baselines. Mem0’s starred progressive row is the capped re-run defined under Table 3 ; its unmodified run is 0/20 (0.0%), 95% Wilson interval [0.0, 16.1].

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Issue	Observed release ambiguity	Treatment in this study
Model identifier	The MemoryArena paper uses both GPT-5.1-mini and GPT-5-mini in different sections.	We report the requested alias used in our runs, gpt-5-mini , and do not use cross-paper absolute score matching.
Progressive count	The paper cites 256 progressive tasks, while the released dataset contains 221 progressive decompositions.	The shared comparison uses a fixed 20-query list. The separate full MemoryLake run uses all 221 released decompositions.
Progressive reproduction	The paper does not provide a reproducible query list and evaluator version sufficient for us to reconstruct its reported 20-query values.	We use only scores generated inside our four-system experiment and do not import the original paper’s progressive values.

Appendix

Table 6: Release-alignment observations that directly affect protocol interpretation.

System	Slot coverage	PS (answered slots)	PS (all slots)	SR
MemoryLake	133/142 (93.7)	6.7	6.7	4/20
Mem0, unmodified	32/142 (22.5)	3.3	1.9	0/20
Mem0, capped re-run	120/142 (84.5)	10.7	8.9	3/20
Vector RAG	92/142 (64.8)	8.2	5.6	2/20
Long Context	134/142 (94.4)	6.0	5.3	2/20

Appendix

Table 7: Progressive retrieval on the shared 20-query subset: slot coverage and both PS denominators, in percent unless a fraction is shown. Cross-system comparisons use the all-slots denominator. Both Mem0 runs are listed; no cell is bolded, because the capped re-run is not measured under the same code as the other rows.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
Feb 18, 2026Zexue He, Yu Wang, Churan Zhi +11Agent Memory LLM Agent Memory
Stanford University · UCSD · UIUC +3
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0
Are We Ready For An Agent-Native Memory System?
Jun 23, 2026Wei Zhou, Xuanhe Zhou, Shaokun Han +5Agent Memory LLM Agent Memory
Shanghai Jiao Tong University · Tsinghua University · MemTensor (Shanghai) Technology Co., Ltd

Feb 18, 2026J/K move · Enter open · S save

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu+10

Stanford University · UCSD · UIUC+3

Agent Memory LLM Agent Memory Memory Agent Benchmarks Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory. MemoryArena is now released at https://memoryarena.github.io/.

Abstract

LLM Agent Memory

Explore similar work

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

DolphinBench: Mapping the Pareto Frontier of Agent Memory

Are We Ready For An Agent-Native Memory System?