Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet existing memory benchmarks are largely dialogue-centric, while real agent memory consists of continuous agent-environment interaction trajectories composed of states, actions, observations, and tool outputs. To address this gap, we introduce AMA-Bench (Agent Memory with Any length), a benchmark for evaluating long-horizon memory in realistic agentic settings. AMA-Bench combines real-world agent trajectories from representative applications with expert-curated QA, as well as synthetic trajectories that scale to arbitrary horizons with rule-based QA. Our study shows that existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval. We further propose AMA-Agent, a memory system based on causality-graph construction and tool-augmented retrieval. AMA-Agent achieves 57.22% accuracy on AMA-Bench, outperforming the strongest baseline by 11.16%. Resources are available at: https://ama-bench.github.io/.
Figures & tables
Figure 1: Comparison of memory across reasoning, chatbots, and agent applications. Agent trajectories exhibit unique properties, including being causally grounded, diverse symbolic artifacts, and dense objective information.
Figure 2: Model performance across agent task families in AMA- Bench.
Figure 3: Domain and question type distribution in AMA- Bench.
Category
Benchmark
Interaction
Content
Average Length
Representation
Memory
Paradigm
Source
(tokens)
Types
Organization
Dialogue-Centric
LoCoMo ( Maharana et al., 2024 )
Dialogue
Real + Synthetic
9K
NL + Vision
Episodic
LongMemEval ( Wu et al., 2025 )
Dialogue
Synthetic
115K
NL
Multi-session
MemoryAgentBench ( Hu et al., 2025b )
Dialogue
Real + Synthetic
100K–500K
NL
Multi-domain
MemoryBench ( Ai et al., 2025 )
Dialogue
Real + Synthetic
∼ 30–380K
NL
Multi-session + Feedback
RealTalk ( Lee et al., 2025 )
Dialogue
Real
17K
NL
Multi-day
Table 1: Comparison of memory benchmarks. NL: Natural Language.
Figure 4: Formalizing memory system and capability for agentic applications.
Mechanism
Capability
Description
Memory Retrieval
A. Recall
Identification of Temporal and sequential information.
B. Causal Inference
Verification of action preconditions and dependency relations between states.
Memory Evolution
C. State Updating
Tracking updates to states, including explicit observations and hidden states.
Memory Condensation
D. State Abstraction
Filtering redundant content while extracting precise and condensed key information.
Table 2: Memory capability described in Sec 3.2 . We group evaluation dimensions into three mechanisms and four capabilities.
Figure 5: Synthetic subset construction pipeline. We synthesize an executable environment backend with explicit latent states and transitions, render machine-generated observations to form trajectories, and programmatically generate trajectory-grounded QA pairs.
Figure 6: LLM-as-a-judge accuracy across different memory systems using Qwen3-32B as the base model.
Figure 7: Impact of Model Scale vs. Memory Architecture. While scaling the backbone yields marginal gains, the choice of memory system accounts for the majority of performance variance.
Method
Full Observation w/ Needle ACC
Consturcted Memory w/ Needle ACC
End to End System ACC
HippoRAG2
0.46
0.37 (−19.6%)
0.21 (−43.2%)
Mem1
0.29 (−37.0%)
0.20 (−31.0%)
AMem
0.29 (−37.0%)
0.24 (−17.2%)
MemoryBank
0.27 (−41.3%)
0.26 (−3.7%)
Table 3: Ablation study on performance bottlenecks under the needle retrieval protocol in BabyAI, evaluated by accuracy (ACC) . Dark red values denote relative decrease vs. the previous column.
Figure 8: Overview of the AMA- agent. (A) Illustrates the transition from trajectories to a structured causality graph. (B) depicts the retrieval mechanism, utilizing tool-augment search.
Method
Recall
Causal Inference
State Updating
State Abstraction
Average
Claude Haiku 3.5 ( Anthropic, 2025 )
0.4943 (0.3510)
0.4507 (0.2792)
0.4287 (0.3015)
0.3090 (0.2648)
0.4361 (0.3067)
GPT-5-mini ( OpenAI, 2025 )
0.6951 (0.4010)
0.7157 (0.3027)
0.6575 (0.3288)
0.6235 (0.3262)
0.6784 (0.3464)
GPT 5.2 ( Wailgum, 2025 )
0.7741 (0.4758)
0.8047 (0.3512)
0.6563 (0.3686)
0.6037 (0.3582)
0.7226 (0.3988)
Gemini 2.5 flash ( Gemini Team, 2025 )
0.5834 (0.3682)
0.5087 (0.2628)
0.5000 (0.2395)
0.4196 (0.2361)
0.5168 (0.2878)
Qwen2.5-14B-1M ( Yang et al., 2025b )
0.5570 (0.4157)
0.4111 (0.3209)
0.4728 (0.3348)
0.3368 (0.3560)
0.4638 (0.3622)
Qwen3-32B ( Yang et al., 2025a )
0.6149 (0.4074)
0.5178 (0.3289)
0.4903 (0.3334)
0.3657 (0.3172)
0.5181 (0.3545)
Table 4: Performance of different models on real-world subset.
Method
Recall
Causal Inference
State Updating
State Abstraction
Average
RAG
BM25
0.3301 (0.1465)
0.4264 (0.1549)
0.3450 (0.1325)
0.2498 (0.1623)
0.3436 (0.1475)
Qwen3-Emb-4B ( Zhang et al., 2025b )
0.4843 (0.1590)
0.4974 (0.1549)
0.3520 (0.1353)
0.3011 (0.1610)
0.4227 (0.1522)
GraphRAG ( Edge et al., 2025 )
0.3077 (0.2769)
0.3905 (0.2634)
0.3140 (0.2551)
0.2879 (0.2588)
0.3258 (0.2650)
HippoRAG2 ( Gutiérrez et al., 2025 )
0.4579 (0.2356)
0.5080 (0.1966)
0.4403 (0.1892)
0.3538 (0.1785)
0.4480 (0.2048)
Agent Memory Methods
Table 5: Performance comparison of Agent Memory and RAG methods using base model Qwen-32B on real-world subset.
Figure 9: Performance Benchmarking. We evaluate 15 memory methods across Qwen 8B and 32B backbones.
Method
TW E2E
TW QA
Sp2 E2E
Sp2 QA
AMA- Agent (AMA)
51.5
40.4
26.2
57.4
LongContext
47.5
33.0
23.5
50.9
HippoRAG ( Gutiérrez et al., 2025 )
27.5
11.6
23.8
50.5
Embedding (Qwen3-Emb-4B) ( Zhang et al., 2025b )
37.5
13.9
21.4
45.4
MemoryBank ( Zhong et al., 2023 )
31.5
11.9
17.2
26.0
Mem0 ( Chhikara et al., 2025 )
30.0
11.8
15.6
12.9
Table 6: End-to-end task success and matched QA accuracy on TextWorld and Spider2, with the same Qwen3-32B execution agent and only the memory mechanism varied. Across both environments, QA accuracy and end-to-end success are strongly correlated, and AMA-Agent leads on every column.
Method
Recall
Causal Inference
State Updating
State Abstraction
Avg.
AMA-agent
0.62
0.61
0.53
0.47
0.57
w/o Causality Graph
0.48 ( −22.6% )
0.48 ( −21.3% )
0.36 ( −32.1% )
0.35 ( −25.5% )
0.43 ( −24.6% )
w/o Tool-Augmented Retrieval
0.47 ( −24.2% )
0.51 ( −16.4% )
0.42 ( −20.8% )
0.31 ( −34.0% )
0.44 ( −22.8% )
Note: Values in dark red indicate the relative performance decrease compared to the full AMA agent.
Table 7: Ablation results for AMA- Agent
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Benchmark
Trace Source
Total
Selected
Embodied AI
ALFWorld-verified (seen)
ALFRED ( Li et al., 2025 )
140
33
Gaming
BALROG / lmgame bench
BALROG / GamingAgent
367
30
Web Task Execution
WebArena ( Zhou et al., 2023 )
WebArena ( Zhou et al., 2023 )
162
31
Software Engineering
SWE-bench ( Jimenez et al., 2023 )
OpenHands ( Wang et al., 2024 )
162
34
Open World Tool QA
GAIA ( Mialon et al., 2023 )
CoSight ( Zhang et al., 2025a )
100
30
Text 2 SQL
Spider 2.0 ( Lei et al., 2024 )
Spider2 agent ( Lei et al., 2024 )
120
51
Appendix
Table 8: Implementation details of collected agent trajectories across task families.
Field
#Samples
#QA
Evaluation Type
Avg. Turns
Avg. Tokens
Max Tokens
Type A
Type B
Type C
Type D
Text 2 SQL
51
612
223
153
134
102
21.80
6,049
10,718
Open World Tool QA
30
360
98
95
107
60
41.40
288,651
996,826
Web Task Execution
31
372
125
93
93
61
24.77
34,265
166,260
Gaming
30
360
120
90
90
60
149.87
14,909
33,360
Embodied AI
30
360
61
90
150
59
130.33
26,306
60,717
Appendix
Table 9: Statistics of QA pairs, evaluation type distribution, and interaction complexity.
Metric
Value
Mean normalized evidence position
0.475
Mean hop count
3.38
Multi-hop ratio
72.2%
Mean evidence span (turns, multi-hop only)
16.1
Mean normalized span (multi-hop only)
0.260
Appendix
Table 10: Long-horizon reasoning statistics on a 720-QA sample of the real-world subset. Evidence positions are normalized to [0,1] over each trajectory.
Split
[0%,20%)
[20%,80%)
[80%,100%]
Overall
21.4%
62.9%
15.7%
Appendix
Table 11: Position coverage of supporting evidence across the full real-world trajectory.
Trajectory range
Turn %
Token %
[0%,20%)
15.1%
15.7%
[20%,40%)
26.8%
27.3%
[40%,60%)
42.8%
41.6%
[60%,80%)
10.3%
10.7%
[80%,100%]
0 5.0%
0 4.8%
Appendix
Table 12: Evidence distribution across normalized trajectory bins on the synthetic subset.
Metric
Overall
BabyAI
TextWorld
Mean normalized evidence position
0.424
0.476
0.420
Mean hop count
4.22
11.39
3.64
Multi-hop ratio
62.6%
100.0%
59.7%
Appendix
Table 13: Per-environment multi-hop statistics on the synthetic subset.
Split
First 20%
Middle 60%
Last 20%
BabyAI
9.3%
89.4%
1.3%
TextWorld
15.2%
79.7%
5.0%
Appendix
Table 14: Position coverage of supporting evidence across the full synthetic trajectory.
Model
Max context tokens
Notes
Claude 3.5 Haiku
200,000
API context window
OpenAI GPT 5 mini
400,000
API context window
OpenAI GPT 5.2
400,000
API context window
Gemini 2.5 Flash
1,048,576
Max input tokens
Qwen2.5 14B Instruct 1M
1,010,000
Long context checkpoint
Qwen3 32B
32,768
Native
Appendix
Table 15: Maximum context lengths used for long context baselines.
Judge pair
Agreement
Qwen3-32B vs. GPT-5.4
92.8%
Qwen3-32B vs. GPT-5.2
91.8%
Qwen3-32B vs. Claude-4.6
88.8%
Qwen3-32B vs. DeepSeek-v3.2
84.7%
Appendix
Table 16: Pairwise agreement (%) of the primary Qwen3-32B judge against four alternative judges from different model families, measured on the same 300-instance human-validated sample.
Human Label
Judge Label
Correct
Incorrect
Total
Correct
190 (TP)
7 (FP)
197
Incorrect
15 (FN)
88 (TN)
103
Total
205
95
300
Appendix
Table 17: Confusion matrix (left) and performance metrics (right) for LLM-as-judge vs. human annotations.
Method
Avg. Acc
Construct
Retrieve+Ans
E2E
Mem0 ( Chhikara et al., 2025 )
0.2104
1.12
1.73
2.85
MemAgent ( Yu et al., 2025 )
0.2768
4.86
0.68
5.54
AMA- Agent (AMA)
0.5722
6.80
1.70
8.50
HippoRAG2 ( Gutiérrez et al., 2025 )
0.4480
10.90
1.20
12.10
Mem1 ( Zhou et al., 2025 )
0.1229
21.57
0.46
22.03
SimpleMem ( Liu et al., 2026 )
0.1811
23.55
0.80
24.35
Appendix
Table 18: Average per-trajectory latency (seconds) on the real-world subset, broken down into memory construction and retrieve+answer. AMA-Agent is more accurate than every faster baseline and faster than every more-accurate baseline.
Method
Easy
Medium
Hard
Very hard
E2E
AMA- Agent (AMA)
84
61
35
22
50.5
LongContext
85
55
32
18
47.5
HippoRAG
63
30
12
5
27.5
Embedding
68
48
26
8
37.5
MemoryBank
53
36
23
14
31.5
Mem0
50
35
20
12
30.0
Appendix
Table 19: TextWorld end-to-end task success (%) by difficulty, same Qwen3-32B execution agent with only the memory mechanism varied.
Method
Avg / QA
Avg / task
Max / QA
Max / task
LongContext
1.4
2,176
2.2
2,932
HippoRAG
12.1
3,660
47.2
7,290
Appendix
Table 20: Average and maximum wall-clock latency (seconds) per QA vs. per end-to-end task on TextWorld. End-to-end rollouts are ∼103× slower per task because the agent must execute 80 – 100 sequential interaction steps with the environment.
Behavior pattern
RAG-based memory
Agentic-summary memory
Causality-graph memory (Ours)
Reduce repetitive actions
Retrieves semantically related clues such as “antidote” or “alchemy room” but misses the prerequisite chain (retrieve key → unlock basement → enter room), leading to repeated invalid open door attempts.
Compresses the trajectory into a coarse goal (“craft antidote in basement”), omitting executable preconditions; the agent repeatedly retries futile actions.
Preserves action dependencies explicitly, so the agent first retrieves the key, then unlocks the basement, then crafts the antidote, with no wasted attempts.
Recover from earlier mistakes
Retrieval still surfaces stale-but-relevant fragments, causing the agent to keep following an already-invalidated plan.
A summary may preserve the original plan but fail to reflect an irreversible state change (e.g., a consumed item).
Updates the consequence chain on each transition, so the agent abandons the invalid branch and re-plans.
Appendix
Table 21: Qualitative comparison of memory categories on TextWorld behavior. RAG retrieves semantically relevant but causally incomplete evidence; agentic-summary compresses away preconditions; causality-graph preserves prerequisite chains and recovers from errors.
Method
Qwen3-32B
Qwen3-8B
GPT-5-mini (syn.)
Mem0
0.2095
0.2770
0.1141
HippoRAG2
0.4378
0.3836
0.0732
Mem1
0.1218
0.1429
0.0928
MemoryBank
0.3417
0.2648
0.0725
A-mem
0.3208
0.3498
0.1158
AMA- Agent (AMA)
0.5629
0.4814
0.2075
Appendix
Table 22: Average accuracy across base models. Real-world numbers use the full real-world subset; the GPT-5-mini column uses the synthetic subset. The relative ordering among methods is preserved across all three settings.
Method
LoCoMo Acc.
A-mem ( Xu et al., 2025 )
60.84
Mem0 ( Chhikara et al., 2025 )
63.51
GraphRAG ( Edge et al., 2025 )
65.39
HippoRAG ( Gutiérrez et al., 2025 )
72.10
AMA- Agent (AMA)
76.20
Appendix
Table 23: Out-of-benchmark generalization on LoCoMo ( Maharana et al., 2024 ) (Qwen3-32B). AMA-Agent outperforms the strongest baseline HippoRAG by 4.1 absolute points.
Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce EvoMemBench, a unified benchmark organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. execution-oriented). We compare 15 representative memory methods with strong long-context baselines under a standardized protocol. Results show that current memory systems are still far from a general solution: long-context baselines remain highly competitive, memory helps most when the current context is insufficient or tasks are difficult, and no single memory form works consistently across all settings. Retrieval-based methods remain strong for knowledge-intensive settings, whereas procedural and long-term memory methods are more effective for execution-oriented tasks when their stored experience matches the task structure. We hope EvoMemBench facilitates future research on more effective memory systems for LLM-based agents. Our code is available at https://github.com/DSAIL-Memory/EvoMemBench.
Yuyao Wang, Zhongjian Zhang, Mo Chi +7
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
Agentic memory systems enable large language model (LLM) agents to maintain state across long interactions, supporting long-horizon reasoning and personalization beyond fixed context windows. Despite rapid architectural development, the empirical foundations of these systems remain fragile: existing benchmarks are often underscaled, evaluation metrics are misaligned with semantic utility, performance varies significantly across backbone models, and system-level costs are frequently overlooked. This survey presents a structured analysis of agentic memory from both architectural and system perspectives. We first introduce a concise taxonomy of MAG systems based on four memory structures. Then, we analyze key pain points limiting current systems, including benchmark saturation effects, metric validity and judge sensitivity, backbone-dependent accuracy, and the latency and throughput overhead introduced by memory maintenance. By connecting the memory structure to empirical limitations, this survey clarifies why current agentic memory systems often underperform their theoretical promise and outlines directions for more reliable evaluation and scalable system design.
Dongming Jiang, Yi Li, Songtao Wei +8
University of Texas at Dallas · University of California, Davis · Texas A&M University
Memory emerges as the core module in the large language model (LLM)-based agents for long-horizon complex tasks (e.g., multi-turn dialogue, game playing, scientific discovery), where memory can enable knowledge accumulation, iterative reasoning and self-evolution. A number of memory methods have been proposed in the literature. However, these methods have not been systematically and comprehensively compared under the same experimental settings. In this paper, we first summarize a unified framework that covers existing representative agent memory methods from a high-level perspective. We then extensively compare representative agent memory methods on two long-term conversational benchmarks and an agentic memory benchmark, and examine the effectiveness of representative methods, providing a thorough analysis of those methods. As a byproduct of our experimental analysis, we also design a new memory method by exploiting modules in the existing methods, which outperforms the state-of-the-art methods. Finally, based on these findings, we offer promising future research opportunities. We believe that a deeper understanding of the behavior of existing methods can provide valuable new insights for future research.