AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations

Cheng Jiayang; Dongyu Ru; Lin Qiu; Yiyang Li; Xuezhi Cao; Yangqiu Song; Xunliang Cai

cs.CLMar 2, 2026

AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations

Authors: Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, Xunliang Cai

Organizations: The Hong Kong University of Science and Technology · Meituan

Abstract

Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reasons. AMemGym not only enables effective selection among competing approaches but also can potentially drive the self-evolution of memory management strategies. By bridging structured state evolution with free-form interactions, our framework provides a scalable, diagnostically rich environment for advancing memory capabilities in conversational agents.

Figures & tables

Figure 1: On-policy v.s. off-policy evaluation for assistants’ memory.

Benchmark	Eval. Mode	Optim. Feedback	Automation Level	Context Length	Eval. Metrics
MSC ( Xu et al., 2022 )	Static	✗	Manual	1.2K	-
RealTalk ( Lee et al., 2025 )	Static	✗	Manual	17K	Emotional Intelligence, Persona Simulation, Memory Probing (F1, accuracy)
DialSim ( Kim et al., 2024 )	Static	✗	Manual	-	QA Accuracy
LoCoMo ( Maharana et al., 2024 )	Static	✗	Semi-Automated	9.2K	QA Accuracy, Summarization, Generation
PerLTQA ( Du et al., 2024 )	Static	✗	Semi-Automated	-	QA Accuracy
LongMemEval ( Wu et al., 2024 )	Static	✗	Semi-Automated	Configurable (115K, 1.5M)	Retrieval Recall, QA Accuracy

Table 1: A comparison of features across agent memory benchmarks.

Figure 2: An overview of the AMemGym framework.

Figure 3: An overview of diagnostic metrics: write, read, and utilization .

Memory Agents	On-policy $\uparrow$	Off-policy $\uparrow$	$\Delta$ Rank
AWE-(2,4,30)	.291	.253( .038 )	$\blacktriangledown$ 3
AWE-(2,8,30)	.278	.271( .007 )	–
AWE-(2,4,10)	.275	.273( .002 )	$\blacktriangle$ 2
AWE-(4,4,30)	.262	.229( .033 )	$\blacktriangledown$ 3
AWE-(2,0,30)	.261	.262( .001 )	$\blacktriangle$ 2
AWE-(2,4,50)	.251	.248( .003 )	$\blacktriangle$ 1

Table 2: The on-policy v.s. off-policy comparison on memory scores of various assistants. Results on different native LLMs are listed in a separate table below. Memory agents use the same LLM (gpt-4.1-mini) for generation.

Panel of Figure 5 — Figure 5: Evaluation on native LLMs. Overall scores and normalized memory scores are both demonstrated.

Figure 6: Memory scores of different memory agents. We omit the overall score comparison as they use the same LLM (gpt-4.1-mini) for generation.

Panel of Figure 7 — Figure 7: Diagnosis on various memory implementations.

Feedback	Memory $\uparrow$	Write $\downarrow$	Read $\downarrow$	Util. $\downarrow$
No Evolution	.172	.293	.242	.118
Question Only	.197	.291	.235	.110
Complete	.197	.263	.237	.136

Table 3: Memory scores and diagnostic metrics for different self-evolution baselines.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Base	Extra
Total Users	20	20
Total Periods	220	420
Total Sessions (Turns)	943	1,632
Evaluation Questions	200	200
Avg Turns per User	47.15	81.60

Appendix

Table 4: Structural statistics of the base and extended benchmark configurations

Figure 8: Annotation interface for state exposure.

Figure 9: Annotation interface for conversation states.

Figure 10: Annotation interface for ground-truth judgment reliability evaluation.

Figure 11: Comparison of memory performance and factual recall for evolution assistants under different environmental feedback conditions.

State Schema	P1	P2	P5	P10
volunteering personal mobility level [“highly mobile”, “occasional assistance needed”, “limited mobility”]	Implied: . “Maintain Up-to-Date Health, Wellness, and Dietary Profiles: … changes over time, including… medical considerations .”	Implied: . “Document Detailed Plans, Goals, and Intentions with Complete Logistics and Contingencies: Track upcoming events… including specific logistical details such as… accessibility considerations , and contingency plans.”	Explicit: . “Capture Specific Personal Preferences with Contextual and Situational Details: … and hobbies (preferred formats, skill levels, group sizes, engagement styles, and accessibility needs )…”	Explicit: . “Record Activity, Service, and Volunteering Preferences…: … accessibility features ), hobbies and teaching approaches (skill levels… accessibility aids )…”
mentoring delivery format [“oneon one”, “small group”, “workshop series”]	Implied: . “Save Professional, Mentorship, and Development Details: Remember …, preferred learning styles , and relevant networking or community involvement.”	Implied: . “Save Professional, Mentorship, and Development Details with Learning and Engagement Styles: Remember …, preferred learning styles , networking involvement, …”	Implied: . “Save Professional, Mentorship, and Development Details with Learning, Engagement, and Support Styles: Remember… and mentoring activity preferences .”	Explicit: . “Save Professional, … and Session Structures: …Capture detailed session structures , preferred icebreakers, …”
potluck available cooking time [“limited under 2 hours”, “flexible afternoon”, “full day prep”]	Implied: . “Document Detailed Plans, Goals, and Intentions with Logistics: Track upcoming events… including specific logistical details such as dates, times , locations, …”	Implied: . “Document Detailed Plans, Goals, and Intentions with Complete Logistics and Contingencies: Track upcoming events… including specific logistical details such as dates, times , locations, …”	Explicit: . “Capture Specific Personal Preferences with Contextual and Situational Details: …and products (including situational factors such as event type, timing , preparation ease , and cost sensitivity).”	Explicit: . “Capture Specific Personal Preferences with Context, …and products (situational factors such as event type, timing , preparation ease , cost sensitivity, durability, and user experience).”
soul food guest health goals [“general healthy eating”, “weight management”, “chronic condition management”]	Implied: . “Maintain Up-to-Date Health, …, wellness goals , and any adaptations or changes over time…”	Implied: . “Maintain Up-to-Date Health, …, wellness goals … and any adaptations or changes over time…”	Explicit: . “Capture Specific Personal Preferences with Contextual and Situational Details: Extract explicit likes, …, and health-conscious modifications )…”	Explicit: . “Maintain Up-to-Date Health, …, symptom management strategies , evolving health needs , and personalized wellness preferences …”
crimson tide game tech setup [“basic tv livestream”, “outdoor projector”, “no live viewing available”]	Absent	Absent	Implied: . “Record Activity, …, technology comfort and tools , volunteer safety checklists with tone and language preferences)…”	Implied: . “Record Activity, …, technology comfort and tools , volunteer safety checklists…)”

Appendix

Table 5: Running examples of prompt evolution traces on period 1 (P1), 2 (P2), 5 (P5), and 10 (P10).

Panel of Figure 12 — Figure 12: Memory evaluation results on the extra configuration.

Panel of Figure 13 — Figure 13: Memory evaluation results with deepseek-v3 as the user LLM.

Figure 14: Full figure for diagnosis on write strategies.

Open-source Memory System	Overall	Memory
Mem0 ( Chhikara et al., 2025 ) w/o Graph (AWE in the paper)	0.430	0.296
Mem0 ( Chhikara et al., 2025 ) w/ Graph	0.424	0.284
A-Mem ( Xu et al., 2025 )	0.378	0.220
Nemori ( Nan et al., 2025 )	0.385	0.231

Appendix

Table 6: Performance comparison of open-source memory systems

Model	Overall	Memory
gemini-2.5-flash ( Google, 2025b )	0.448	0.327
qwen3-235b-a22b-instruct-2507 ( Yang et al., 2025 )	0.331	0.148
deepseek-v3.1-terminus ( DeepSeek, 2025 )	0.327	0.151
kimi-k2-instruct ( Team et al., 2025 )	0.389	0.234
glm-4.6 ( Z.ai, 2025a )	0.404	0.257

Appendix

Table 7: Performance comparison of open-source models

Model	Overall Score	Memory Score
gpt-4o-mini	0.3178 (±0.0025)	0.1504 (±0.0044)
gpt-4.1-mini	0.3675 (±0.0023)	0.2030 (±0.0031)
gemini-2.5-flash-lite	0.3921 (±0.0062)	0.2583 (±0.0100)
gemini-2.5-flash	0.4465 (±0.0037)	0.3240 (±0.0056)

Appendix

Table 8: Performance stability across five independent runs.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 446 papers

MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios
Mar 23, 2026Yihang Ding, Wanke Xia, Yiting Zhao +5LLM Memory Memory Agent Benchmarks
Tsinghua University · Renmin University of China · CASIA
MemGym: a Long-Horizon Memory Environment for LLM Agents
May 20, 2026Wujiang Xu, Yu Wang, Kai Mei +8LLM Agent Memory LLM Agent Evaluation
1Rutgers University · 2Capital One · 3Princeton University +1
AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
Feb 26, 2026Yujie Zhao, Boqin Yuan, Junbo Huang +9Agent Memory Long-Horizon Agent Evaluation
UCSD · Recursive.

Mar 23, 2026J/K move · Enter open · S save

MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios

Yihang Ding, Wanke Xia, Yiting Zhao, Jinbo Su+4

Tsinghua University · Renmin University of China · CASIA

LLM Memory Memory Agent Benchmarks Temporal Reasoning in Language Models

Open paper AskPDF arXiv

Current evaluations of long-term memory in LLMs are fundamentally static. By fixating on simple retrieval and short-context inference, they neglect the multifaceted nature of complex memory systems, such as dynamic state tracking and hierarchical reasoning in continuous interactions. To overcome these limitations, we propose MemGround, a rigorous long-term memory benchmark natively grounded in rich, gamified interactive scenarios. To systematically assess these capabilities, MemGround introduces a three-tier hierarchical framework that evaluates Surface State Memory, Temporal Associative Memory, and Reasoning-Based Memory through specialized interactive tasks. Furthermore, to comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite comprising Question-Answer Score (QA Overall), Memory Fragments Unlocked (MFU), Memory Fragments with Correct Order (MFCO), and Exploration Trajectory Diagrams (ETD). Extensive experiments reveal that state-of-the-art LLMs and memory agents still struggle with sustained dynamic tracking, temporal event association, and complex reasoning derived from long-term accumulated evidence in interactive environments.

Abstract

Agent Memory Management

Explore similar work

MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios

MemGym: a Long-Horizon Memory Environment for LLM Agents

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications