AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Yujie Zhao; Boqin Yuan; Junbo Huang; Haocheng Yuan; Zhongming Yu; Haozhou Xu; Lanxiang Hu; Abhilash Shankarampeta

cs.AIFeb 26, 2026

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Authors: Yujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan, Zhongming Yu, Haozhou Xu, Lanxiang Hu, Abhilash Shankarampeta, +4 more

Organizations: UCSD · Recursive.

Abstract

Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet existing memory benchmarks are largely dialogue-centric, while real agent memory consists of continuous agent-environment interaction trajectories composed of states, actions, observations, and tool outputs. To address this gap, we introduce AMA-Bench (Agent Memory with Any length), a benchmark for evaluating long-horizon memory in realistic agentic settings. AMA-Bench combines real-world agent trajectories from representative applications with expert-curated QA, as well as synthetic trajectories that scale to arbitrary horizons with rule-based QA. Our study shows that existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval. We further propose AMA-Agent, a memory system based on causality-graph construction and tool-augmented retrieval. AMA-Agent achieves 57.22% accuracy on AMA-Bench, outperforming the strongest baseline by 11.16%. Resources are available at: https://ama-bench.github.io/.

Figures & tables

Figure 1: Comparison of memory across reasoning, chatbots, and agent applications. Agent trajectories exhibit unique properties, including being causally grounded, diverse symbolic artifacts, and dense objective information.

Figure 2: Model performance across agent task families in AMA- Bench.

Figure 3: Domain and question type distribution in AMA- Bench.

Category	Benchmark	Interaction	Content	Average Length	Representation	Memory
		Paradigm	Source	(tokens)	Types	Organization
Dialogue-Centric	LoCoMo ( Maharana et al., 2024 )	Dialogue	Real + Synthetic	9K	NL + Vision	Episodic
	LongMemEval ( Wu et al., 2025 )	Dialogue	Synthetic	115K	NL	Multi-session
	MemoryAgentBench ( Hu et al., 2025b )	Dialogue	Real + Synthetic	100K–500K	NL	Multi-domain
	MemoryBench ( Ai et al., 2025 )	Dialogue	Real + Synthetic	$\sim$ 30–380K	NL	Multi-session + Feedback
	RealTalk ( Lee et al., 2025 )	Dialogue	Real	17K	NL	Multi-day

Table 1: Comparison of memory benchmarks. NL: Natural Language.

Figure 4: Formalizing memory system and capability for agentic applications.

Mechanism	Capability	Description
Memory Retrieval	A. Recall	Identification of Temporal and sequential information.
Memory Retrieval	B. Causal Inference	Verification of action preconditions and dependency relations between states.
Memory Evolution	C. State Updating	Tracking updates to states, including explicit observations and hidden states.
Memory Condensation	D. State Abstraction	Filtering redundant content while extracting precise and condensed key information.

Table 2: Memory capability described in Sec 3.2 . We group evaluation dimensions into three mechanisms and four capabilities.

Figure 5: Synthetic subset construction pipeline. We synthesize an executable environment backend with explicit latent states and transitions, render machine-generated observations to form trajectories, and programmatically generate trajectory-grounded QA pairs.

Figure 6: LLM-as-a-judge accuracy across different memory systems using Qwen3-32B as the base model.

Figure 7: Impact of Model Scale vs. Memory Architecture. While scaling the backbone yields marginal gains, the choice of memory system accounts for the majority of performance variance.

Method	Full Observation w/ Needle ACC	Consturcted Memory w/ Needle ACC	End to End System ACC
HippoRAG2	0.46	0.37 (−19.6%)	0.21 (−43.2%)
Mem1		0.29 (−37.0%)	0.20 (−31.0%)
AMem		0.29 (−37.0%)	0.24 (−17.2%)
MemoryBank		0.27 (−41.3%)	0.26 (−3.7%)

Table 3: Ablation study on performance bottlenecks under the needle retrieval protocol in BabyAI, evaluated by accuracy (ACC) . Dark red values denote relative decrease vs. the previous column.

Figure 8: Overview of the AMA- agent. (A) Illustrates the transition from trajectories to a structured causality graph. (B) depicts the retrieval mechanism, utilizing tool-augment search.

Method	Recall	Causal Inference	State Updating	State Abstraction	Average
Claude Haiku 3.5 ( Anthropic, 2025 )	0.4943 (0.3510)	0.4507 (0.2792)	0.4287 (0.3015)	0.3090 (0.2648)	0.4361 (0.3067)
GPT-5-mini ( OpenAI, 2025 )	0.6951 (0.4010)	0.7157 (0.3027)	0.6575 (0.3288)	0.6235 (0.3262)	0.6784 (0.3464)
GPT 5.2 ( Wailgum, 2025 )	0.7741 (0.4758)	0.8047 (0.3512)	0.6563 (0.3686)	0.6037 (0.3582)	0.7226 (0.3988)
Gemini 2.5 flash ( Gemini Team, 2025 )	0.5834 (0.3682)	0.5087 (0.2628)	0.5000 (0.2395)	0.4196 (0.2361)	0.5168 (0.2878)
Qwen2.5-14B-1M ( Yang et al., 2025b )	0.5570 (0.4157)	0.4111 (0.3209)	0.4728 (0.3348)	0.3368 (0.3560)	0.4638 (0.3622)
Qwen3-32B ( Yang et al., 2025a )	0.6149 (0.4074)	0.5178 (0.3289)	0.4903 (0.3334)	0.3657 (0.3172)	0.5181 (0.3545)

Table 4: Performance of different models on real-world subset.

Method	Recall	Causal Inference	State Updating	State Abstraction	Average
RAG
BM25	0.3301 (0.1465)	0.4264 (0.1549)	0.3450 (0.1325)	0.2498 (0.1623)	0.3436 (0.1475)
Qwen3-Emb-4B ( Zhang et al., 2025b )	0.4843 (0.1590)	0.4974 (0.1549)	0.3520 (0.1353)	0.3011 (0.1610)	0.4227 (0.1522)
GraphRAG ( Edge et al., 2025 )	0.3077 (0.2769)	0.3905 (0.2634)	0.3140 (0.2551)	0.2879 (0.2588)	0.3258 (0.2650)
HippoRAG2 ( Gutiérrez et al., 2025 )	0.4579 (0.2356)	0.5080 (0.1966)	0.4403 (0.1892)	0.3538 (0.1785)	0.4480 (0.2048)
Agent Memory Methods

Table 5: Performance comparison of Agent Memory and RAG methods using base model Qwen-32B on real-world subset.

Figure 9: Performance Benchmarking. We evaluate 15 memory methods across Qwen 8B and 32B backbones.

Method	TW E2E	TW QA	Sp2 E2E	Sp2 QA
AMA- Agent (AMA)	51.5	40.4	26.2	57.4
LongContext	47.5	33.0	23.5	50.9
HippoRAG ( Gutiérrez et al., 2025 )	27.5	11.6	23.8	50.5
Embedding (Qwen3-Emb-4B) ( Zhang et al., 2025b )	37.5	13.9	21.4	45.4
MemoryBank ( Zhong et al., 2023 )	31.5	11.9	17.2	26.0
Mem0 ( Chhikara et al., 2025 )	30.0	11.8	15.6	12.9

Table 6: End-to-end task success and matched QA accuracy on TextWorld and Spider2, with the same Qwen3-32B execution agent and only the memory mechanism varied. Across both environments, QA accuracy and end-to-end success are strongly correlated, and AMA-Agent leads on every column.

Method	Recall	Causal Inference	State Updating	State Abstraction	Avg.
AMA-agent	0.62	0.61	0.53	0.47	0.57
w/o Causality Graph	0.48 ( $-22.6\%$ )	0.48 ( $-21.3\%$ )	0.36 ( $-32.1\%$ )	0.35 ( $-25.5\%$ )	0.43 ( $-24.6\%$ )
w/o Tool-Augmented Retrieval	0.47 ( $-24.2\%$ )	0.51 ( $-16.4\%$ )	0.42 ( $-20.8\%$ )	0.31 ( $-34.0\%$ )	0.44 ( $-22.8\%$ )
Note: Values in dark red indicate the relative performance decrease compared to the full AMA agent.

Table 7: Ablation results for AMA- Agent

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Benchmark	Trace Source	Total	Selected
Embodied AI	ALFWorld-verified (seen)	ALFRED ( Li et al., 2025 )	140	33
Gaming	BALROG / lmgame bench	BALROG / GamingAgent	367	30
Web Task Execution	WebArena ( Zhou et al., 2023 )	WebArena ( Zhou et al., 2023 )	162	31
Software Engineering	SWE-bench ( Jimenez et al., 2023 )	OpenHands ( Wang et al., 2024 )	162	34
Open World Tool QA	GAIA ( Mialon et al., 2023 )	CoSight ( Zhang et al., 2025a )	100	30
Text 2 SQL	Spider 2.0 ( Lei et al., 2024 )	Spider2 agent ( Lei et al., 2024 )	120	51

Appendix

Table 8: Implementation details of collected agent trajectories across task families.

Field	#Samples	#QA	Evaluation Type				Avg. Turns	Avg. Tokens	Max Tokens
Field	#Samples	#QA	Type A	Type B	Type C	Type D	Avg. Turns	Avg. Tokens	Max Tokens
Text 2 SQL	51	612	223	153	134	102	21.80	6,049	10,718
Open World Tool QA	30	360	98	95	107	60	41.40	288,651	996,826
Web Task Execution	31	372	125	93	93	61	24.77	34,265	166,260
Gaming	30	360	120	90	90	60	149.87	14,909	33,360
Embodied AI	30	360	61	90	150	59	130.33	26,306	60,717

Appendix

Table 9: Statistics of QA pairs, evaluation type distribution, and interaction complexity.

Metric	Value
Mean normalized evidence position	0.475
Mean hop count	3.38
Multi-hop ratio	72.2%
Mean evidence span (turns, multi-hop only)	16.1
Mean normalized span (multi-hop only)	0.260

Appendix

Table 10: Long-horizon reasoning statistics on a 720-QA sample of the real-world subset. Evidence positions are normalized to

[0,1]

over each trajectory.

Split	$[0\%,20\%)$	$[20\%,80\%)$	$[80\%,100\%]$
Overall	21.4%	62.9%	15.7%

Appendix

Table 11: Position coverage of supporting evidence across the full real-world trajectory.

Trajectory range	Turn %	Token %
$[0\%,20\%)$	15.1%	15.7%
$[20\%,40\%)$	26.8%	27.3%
$[40\%,60\%)$	42.8%	41.6%
$[60\%,80\%)$	10.3%	10.7%
$[80\%,100\%]$	0 5.0%	0 4.8%

Appendix

Table 12: Evidence distribution across normalized trajectory bins on the synthetic subset.

Metric	Overall	BabyAI	TextWorld
Mean normalized evidence position	0.424	0.476	0.420
Mean hop count	4.22	11.39	3.64
Multi-hop ratio	62.6%	100.0%	59.7%

Appendix

Table 13: Per-environment multi-hop statistics on the synthetic subset.

Split	First 20%	Middle 60%	Last 20%
BabyAI	9.3%	89.4%	1.3%
TextWorld	15.2%	79.7%	5.0%

Appendix

Table 14: Position coverage of supporting evidence across the full synthetic trajectory.

Model	Max context tokens	Notes
Claude 3.5 Haiku	200,000	API context window
OpenAI GPT 5 mini	400,000	API context window
OpenAI GPT 5.2	400,000	API context window
Gemini 2.5 Flash	1,048,576	Max input tokens
Qwen2.5 14B Instruct 1M	1,010,000	Long context checkpoint
Qwen3 32B	32,768	Native

Appendix

Table 15: Maximum context lengths used for long context baselines.

Judge pair	Agreement
Qwen3-32B vs. GPT-5.4	92.8%
Qwen3-32B vs. GPT-5.2	91.8%
Qwen3-32B vs. Claude-4.6	88.8%
Qwen3-32B vs. DeepSeek-v3.2	84.7%

Appendix

Table 16: Pairwise agreement (%) of the primary Qwen3-32B judge against four alternative judges from different model families, measured on the same 300-instance human-validated sample.

	Human Label
Judge Label	Correct	Incorrect	Total
Correct	190 (TP)	7 (FP)	197
Incorrect	15 (FN)	88 (TN)	103
Total	205	95	300

Appendix

Table 17: Confusion matrix (left) and performance metrics (right) for LLM-as-judge vs. human annotations.

Method	Avg. Acc	Construct	Retrieve+Ans	E2E
Mem0 ( Chhikara et al., 2025 )	0.2104	1.12	1.73	2.85
MemAgent ( Yu et al., 2025 )	0.2768	4.86	0.68	5.54
AMA- Agent (AMA)	0.5722	6.80	1.70	8.50
HippoRAG2 ( Gutiérrez et al., 2025 )	0.4480	10.90	1.20	12.10
Mem1 ( Zhou et al., 2025 )	0.1229	21.57	0.46	22.03
SimpleMem ( Liu et al., 2026 )	0.1811	23.55	0.80	24.35

Appendix

Table 18: Average per-trajectory latency (seconds) on the real-world subset, broken down into memory construction and retrieve+answer. AMA-Agent is more accurate than every faster baseline and faster than every more-accurate baseline.

Method	Easy	Medium	Hard	Very hard	E2E
AMA- Agent (AMA)	84	61	35	22	50.5
LongContext	85	55	32	18	47.5
HippoRAG	63	30	12	5	27.5
Embedding	68	48	26	8	37.5
MemoryBank	53	36	23	14	31.5
Mem0	50	35	20	12	30.0

Appendix

Table 19: TextWorld end-to-end task success (%) by difficulty, same Qwen3-32B execution agent with only the memory mechanism varied.

Method	Avg / QA	Avg / task	Max / QA	Max / task
LongContext	1.4	2,176	2.2	2,932
HippoRAG	12.1	3,660	47.2	7,290

Appendix

Table 20: Average and maximum wall-clock latency (seconds) per QA vs. per end-to-end task on TextWorld. End-to-end rollouts are

\sim\!10^{3}\times

slower per task because the agent must execute

80

–

100

sequential interaction steps with the environment.

Behavior pattern	RAG-based memory	Agentic-summary memory	Causality-graph memory (Ours)
Reduce repetitive actions	Retrieves semantically related clues such as “antidote” or “alchemy room” but misses the prerequisite chain (retrieve key $\to$ unlock basement $\to$ enter room), leading to repeated invalid open door attempts.	Compresses the trajectory into a coarse goal (“craft antidote in basement”), omitting executable preconditions; the agent repeatedly retries futile actions.	Preserves action dependencies explicitly, so the agent first retrieves the key, then unlocks the basement, then crafts the antidote, with no wasted attempts.
Recover from earlier mistakes	Retrieval still surfaces stale-but-relevant fragments, causing the agent to keep following an already-invalidated plan.	A summary may preserve the original plan but fail to reflect an irreversible state change (e.g., a consumed item).	Updates the consequence chain on each transition, so the agent abandons the invalid branch and re-plans.

Appendix

Table 21: Qualitative comparison of memory categories on TextWorld behavior. RAG retrieves semantically relevant but causally incomplete evidence; agentic-summary compresses away preconditions; causality-graph preserves prerequisite chains and recovers from errors.

Method	Qwen3-32B	Qwen3-8B	GPT-5-mini (syn.)
Mem0	0.2095	0.2770	0.1141
HippoRAG2	0.4378	0.3836	0.0732
Mem1	0.1218	0.1429	0.0928
MemoryBank	0.3417	0.2648	0.0725
A-mem	0.3208	0.3498	0.1158
AMA- Agent (AMA)	0.5629	0.4814	0.2075

Appendix

Table 22: Average accuracy across base models. Real-world numbers use the full real-world subset; the GPT-5-mini column uses the synthetic subset. The relative ordering among methods is preserved across all three settings.

Method	LoCoMo Acc.
A-mem ( Xu et al., 2025 )	60.84
Mem0 ( Chhikara et al., 2025 )	63.51
GraphRAG ( Edge et al., 2025 )	65.39
HippoRAG ( Gutiérrez et al., 2025 )	72.10
AMA- Agent (AMA)	76.20

Appendix

Table 23: Out-of-benchmark generalization on LoCoMo ( Maharana et al., 2024 ) (Qwen3-32B). AMA-Agent outperforms the strongest baseline HippoRAG by 4.1 absolute points.

Explore similar work

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations
Feb 22, 2026Dongming Jiang, Yi Li, Songtao Wei +8Agent Memory Agent Memory Management
University of Texas at Dallas · University of California, Davis · Texas A&M University
Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework
Apr 2, 2026Yanchen Wu, Tenghui Lin, Yingli Zhou +7LLM Agent Memory Agent Memory Management
CUHK-Shenzhen · CUHK · HITSZ +2

May 18, 2026J/K move · Enter open · S save

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu+6

Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology+1

LLM Agent Memory LLM Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks Agent Benchmarks

Open paper AskPDF arXiv

Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce EvoMemBench, a unified benchmark organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. execution-oriented). We compare 15 representative memory methods with strong long-context baselines under a standardized protocol. Results show that current memory systems are still far from a general solution: long-context baselines remain highly competitive, memory helps most when the current context is insufficient or tasks are difficult, and no single memory form works consistently across all settings. Retrieval-based methods remain strong for knowledge-intensive settings, whereas procedural and long-term memory methods are more effective for execution-oriented tasks when their stored experience matches the task structure. We hope EvoMemBench facilitates future research on more effective memory systems for LLM-based agents. Our code is available at https://github.com/DSAIL-Memory/EvoMemBench.

Abstract

Explore similar work

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework