Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory. MemoryArena is now released at https://memoryarena.github.io/.
Figures & tables
Memory-Agent-Env. Loops
Task Settings
Benchmark
Memory Eval.
Agentic Actions
Env. Feedback
Multi-Sess. Tasks
Interdep. ST
# T (# Q)
# Interdep. ST
#S
LOCOMO ( Maharana et al., 2024 )
✓
✗
✗
✓
✗
7512
1
N/A 1
LongMemEval ( Wu et al., 2025 )
✓
✗
✗
✓
✗
500
1
MemoryAgentBench ( Hu et al., 2025a )
✓
✗
✗
✓
✗
2k
1
MemoryBench ( Ai et al., 2025 )
✓
✗
✗
✓
✗
778
1
WebArena ( Zhou et al., )
✗
✓
✓
✗
✗
812
1
13.3
Table 1: We compare benchmarks along key dimensions: if the benchmark evaluates different memory mechanism, if it evaluates agent actions, and if it involves environment feedbacks in memory–agent–environment loops. We also compare their evaluation task settings and scales. (Notations: T. : tasks; ST. : subtasks; Env. : environment; Interdep. : interdependent; S. : Steps, Q : Queries). Green checkmarks indicate supported features; red crosses indicate unsupported features. Note 1: These benchmarks use long-context conversational QA tasks without agentic actions; thus, the number of action steps is Not Applicable (N/A). Note 2: Evo-Memory constructs a multi-session setting by executing independent tasks from existing single-session agent benchmarks sequentially. Because these tasks are directly reused , there is no explicit subtask-level dependency or cross-session causal structure enforced. So the number of tasks, interdependent subtasks, and per-task action steps cannot be meaningfully defined or aggregated. We marked them as N/A. Note 3: Computed from the official AgencyBench-v2 release.
Figure 1: MemoryArena Evaluates agents with Memory with multi-session tasks in a Memory-Agent-Environment Loop.
Figure 2: MemoryArena supports four distinct evaluation environments, where a memory-augmented task agent completes a sequence of interdependent subtasks. Each subtask session involves multiple agent actions.
#min ST (or Sess.)
#max ST (or Sess.)
# avg T. Trace L
# T (Groups of Subtasks)
Bundled Web Shopping
6
6
41.5k
150
Included domain
[ Grocery, Beauty, Electronics, Home Decor, Baking ]
Group Travel Planning
5
9
40.6k
270
Progressive Web Search
2
16
122.4k
221
Math Formal Reasoning
2
16
18.1k
40
Included Domains
[Pure math, Optimization, Learning theory]
Table 2: Benchmark Statistics in MemoryArena .
Formal Reasoning
Bundled web shopping
Group Travel Planing
Progressive Web Search
Math
Phys
Memory Type
SR
PS
SR
PS
sPS
SR
PS
SR
PS
SR
PS
All Task Avg SR
Task agent + Long Context
GPT-5.1-mini
0D
0.01
0.58
0.00
0.00
0.52
0.06
0 .05
0 .21
0 .30
0.50
0.59
0.16
GPT-4.1-mini
0D
0.00
0.43
0.00
0.00
0.19
0.02
0.03
0.13
0.27
0.30
0.45
0.09
Gemini-3-Flash
0D
0.12
0.76
0.00
0.01
0.62
0 .07
0.04
0.17
0 .30
0 .60
0 .61
0 .19
Table 3: Main results on task agent (gpt-5.1-mini) with long-context memory, memory agent, and RAG agent over four agentic environments MemoryArena . We bold the global best methods and u nderline the group best ones within each category. 0D : raw context without any processing; 1D : flat memory, 2D : structured memory. SR : Success Rate. PS : Progress Score (defined in Section 4.2 ). sPS : soft Process Score (we provided sPS here for more informative compression as PS is all near-zero in Group Travel Planning. See Section 4.2 for more details.)
Figure 3: Success Rate at subtask depth k . The decay trend indicates agents cannot sustain execution as dependencies span more sessions.
Bundled Web Shopping
Group Travel Plan
Progressive Web Search
Formal Reasoning Math
Formal Reasoning Phys.
Avg.
Long Context
GPT-5.1-mini
95
119
60
50
47
74.2
GPT-4.1-mini
31
63
22
21
31
33.6
Claude-Sonnet-4.5
56
52
180
83
38
81.8
Gemini-3-Flash
78
33
42
43
65
52.2
Memory Systems
Table 4: Latency of agents with different memory paradigms (sec.).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Data example for bundled web shopping task.
Figure 5: Data example for the Group Travel Planning task.
Figure 6: Data example for the Progressive Web Search task.
Figure 7: An example from the math formal reasoning task with iterative problem solving in MemoryArena .
Figure 8: Bundled Web Shopping Prompt Framework
Figure 9: Group Travel Planning Prompts
Figure 10: Prompts used in Progressive Web Search tasks
Figure 11: Prompts and workflow used in Sequential Formal Reasoning (Math as an example) tasks
Shopping
Travel
Search
Math
Phys
SR
PS
SR
PS
sPS
SR
PS
SR
PS
SR
PS
Claude-Sonnet-4.6
0.13
0.79
0.00
0.20
0.89
0.06
0.05
0.37
0.4
0.4
0.53
Letta
0
0.48
0.00
0.00
0.15
0.23
0.11
0.16
0.27
0.4
0.53
Mem0-g
0
0.49
0.00
0.00
0.12
0.21
0.10
0.17
0.3
0.15
0.41
Text-Embedding-3-Small
0.04
0.55
0.00
0.04
0.42
0.27
0.09
0.37
0.35
0.63
0.68
MemoRAG
0.01
0.54
0.00
0.04
0.47
0.35
0.22
0.30
0.34
0.5
0.60
Appendix
Table 5: MemoryArena results with Claude-Sonnet-4.6.
BWS
GTP
PWS
FR(M)
FR(P)
AVG
Long Context
GPT-5.1-mini
570
802
837
390
190
557.8
GPT-4.1-mini
186
425
196
154
123
216.8
Claude-Sonnet-4.5
336
350
450
635
157
385.6
Gemini-3-Flash
468
227
101
334
251
276.2
Memory Systems
Appendix
Table 6: Latency in memory systems (sec.).
L0
L1
L2
L3
L4
long-context
0.48
0.29
0.21
0.18
0.15
Appendix
Table 7: Pass rate drop with dependency distance (on Group Travel Planning tasks)
Shopping
Travel
Search
Math
original
0.01
0.00
0.06
0.26
Δ
+0.016
+0.050
+0.05
+0.16
Appendix
Table 8: Δ performance with oracle memory in long-context settings.
Figure 12: Comparison of exploration depth. GPT-5-mini exhibits ”satisficing” behavior, purchasing the first relevant result (Option 1) immediately. In contrast, Gemini/Claude demonstrates ”optimizing” behavior by backtracking and exploring intermediate options, ultimately selecting Option 5 which best fits the ”brightening” goal and budget constraints.
Figure 13: Impact of Retrieval Failure on Sequential Compatibility. The BM25 RAG model fails to retrieve the ”Compact” attribute from the Step 2 purchase history. Consequently, it violates the negative constraint (”Compact avoids Low Profile”), whereas the Long Context model correctly utilizes the history to select the ”Articulating” option.
Figure 14: Case study in Group travel planning: MemGPT achieves best precision in memory, however long-context cannot capture the correct details from the beginning and suffer from “lost in the middle”.
Figure 15: Group Travel Planning case study: a memory retrieval failure causes drift from the finalized seed plan (wrong date/origin in flight search) and has a downstream constraint violation when selecting dinner.
Figure 16: Progressive Web Search case study 1: comparision between different models in memory retrieval.
Figure 17: Progressive Web Search case study 2: comparison between different memory systems.
Figure 18: Case study 1: comparison between memory systems in Math Formal Reasoning.
Figure 19: Case study 2: comparision between memory systems in Math Formal Reasoning.