MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

Weizhi Zhang; Xiaokai Wei; Wei-Chieh Huang; Zheng Hui; Chen Wang; Michelle Gong; Philip S. Yu

cs.CLMar 26, 2026

MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

Authors: Weizhi Zhang, Xiaokai Wei, Wei-Chieh Huang, Zheng Hui, Chen Wang, Michelle Gong, Philip S. Yu

Organizations: Roblox · University of Illinois Chicago · University of Cambridge

Abstract

Recent advancements in Large Language Models (LLMs) have expanded context windows to million-token scales, yet benchmarks for evaluating memory remain limited to short-session synthetic dialogues. We introduce MemoryCD, the first large-scale, user-centric, cross-domain memory benchmark derived from lifelong real-world behaviors in the Amazon Review dataset. Unlike existing memory datasets that rely on scripted personas to generate synthetic user data, MemoryCD tracks authentic user interactions across years and multiple domains. We construct a multi-faceted long-context memory evaluation pipeline of 14 state-of-the-art LLM base models with 6 memory method baselines on 4 distinct personalization tasks over 12 diverse domains to evaluate an agent's ability to simulate real user behaviors in both single and cross-domain settings. Our analysis reveals that existing memory methods are far from user satisfaction in various domains, offering the first testbed for cross-domain life-long personalization evaluation.

Figures & tables

Figure 1: Comparison memory benchmarks: MemoryCD (ours) captures cross-domain real-user activities over long time horizons. LaMP ( Salemi et al., 2024 ) focuses only on the short-term single-domain user behaviors; LoCoMo ( Maharana et al., 2024 ) represents non-authentic LLM-simulated user behaviors.

Figure 2: The MemoryCD benchmark spans 12 real-world domains and evaluates 6 SOTA memory methods . Different from other memory benchmarks targeting one specific memory stage (mostly retrieval), we design 4 basic tasks with 2 settings to provide end-to-end user satisfaction evaluation grounded on the lifelong real user behaviors (Table 2 ).

Data	Memory	Context	# of	Long	User	Cross-	Real
Source	Benchmarks	Length	Sessions	Timespan	Level	Domain	Feedback
Synthetic	LoCoMo ( Maharana et al., 2024 )	9K	35	✗	✓ ✗	✗	✗
	PrefEval ( Zhao et al., 2025 )	100K	–	✗	✓ ✗	✓ ✗	✗
	LongMemEval ( Wu et al., 2025a )	1.5M	500	✓ ✗	✓ ✗	✗	✗
	PersonaMem ( Jiang et al., 2025 )	1M	60	✓	✓ ✗	✓ ✗	✗
	MemBench ( Tan et al., 2025a )	100K	0.5K-8K	✓ ✗	✓ ✗	✗	✗

Table 1: Comparison of representative memory benchmarks. ✓ : fully covered; ✓ ✗ : partially covered; ✗ : not covered. Synthetic denotes simulated user behaviors. Real User denotes data constructed from real user interaction histories. # of Sessions is the number of sessions per individual users.

Aspect	★ Rating Prediction	Item Ranking	Review Summarization	Review Generation
Category	User decision-making (quantitative preference modeling)	User decision-making (relative preference modeling)	User text generation (attitude & sentiment reflection)	User text generation (preference & linguistic expression style)
Goal	Predict user’s rating (1–5) for a target item.	Rank candidate items by expected user interest.	Generate a concise title reflecting user tone and sentiment.	Generate full review text expressing user preferences and style.
Input	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$	User memory $\mathcal{M}_{u}$ ; candidate set $C$	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$ ; user review context $x_{u,i}^{*}$	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$
Output	Rating $\hat{r}_{u,i}\in\{1,\dots,5\}$	Ranking $\pi_{u}(C)$	Short title $x_{u,i}^{\prime}$	Full review $x_{u,i}^{*}$
Metrics	MAE, RMSE	NDCG@ $K$	ROUGE-L, BLEU-1	ROUGE-L, BLEU-1

Table 2: Summary of the four personalization tasks in MemoryCD .

Figure 3: Ratios of overlapping users across domains . Each cell shows the percentage of users in one domain that also appear in another, revealing feasible cross-domain personalization settings.

Dataset	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Books	GPT-5-Nano	0.543	0.872	0.135	0.164	0.122	0.055	0.110	0.145
	GPT-5-Mini	0.464	0.816	0.161	0.190	0.138	0.063	0.130	0.174
	GPT-5	0.330	0.624	0.146	0.177	0.137	0.064	0.132	0.195
	Claude-4 Opus	0.611	0.954	0.142	0.183	0.126	0.070	0.104	0.143
	Claude-4 Sonnet	0.670	1.000	0.109	0.163	0.145	0.089	0.110	0.151

Table 3: Performance comparison of frontier LLMs in single-domain with long-context memory prompting.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Long-Context	GPT-5	0.330	0.624	0.146	0.177	0.137	0.064	0.132	0.195
	Claude-4 Sonnet	0.670	1.000	0.109	0.163	0.145	0.089	0.110	0.151
	Gemini-2.5 Pro	0.401	0.762	0.206	0.229	0.208	0.128	0.162	0.207
Mem0	GPT-5	0.419	0.698	0.184	0.217	0.173	0.097	0.143	0.195
	Claude-4 Sonnet	0.491	0.772	0.172	0.207	0.202	0.126	0.141	0.182

Table 4: Comparison of memory methods on Books using representative LLMs.

Figure 4: Performance comparisons using different memory sources evaluated on the Home & Kitchen . Each radar plot corresponds to a representative frontier long-context LLM (GPT-5, Claude-4 Sonnet, Gemini-2.5 Pro). Curves correspond to different memory sources: no memory (Source N/A), single-domain memories ( Books , Electronics , Personal Care ), and aggregated cross-domain memory (All 3 Sources). Scales are unified for fair comparison (MAE and RMSE scales are reversed).

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Long-Context	GPT-5	0.354	0.675	0.179	0.299	0.174	0.070	0.104	0.111
	Claude-4 Sonnet	0.570	0.839	0.108	0.212	0.192	0.108	0.125	0.145
	Gemini-2.5 Pro	0.391	0.723	0.140	0.255	0.204	0.119	0.152	0.180
Mem0	GPT-5	0.367	0.661	0.146	0.284	0.189	0.094	0.112	0.122
	Claude-4 Sonnet	0.596	0.823	0.140	0.255	0.197	0.116	0.127	0.146

Table 5: Cross-domain evaluation of different memory methods on Home & Kitchen . Memory includes source domains of Books, Electronics, and Personal Care .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Context Length	Max # Sessions	Avg # Sessions	Min # Sessions	Filtering Thresholds	# of Users	# of Samples
Beauty	11,137	165	81	54	50	57	171
Arts	49,101	801	338	204	200	50	150
Automotive	27,104	512	258	203	200	64	192
Personal Care	202,096	2,973	839	504	500	128	384
Books	272,372	3,888	1,552	1,002	1,000	89	267
Electronics	184,084	2,602	781	502	500	94	282

Appendix

Table 6: Domain-level statistics for the proposed ultra long-context user-centric memory benchmark. Only users exceeding the minimum interaction threshold are retained; each user contributes three evaluation samples.

Task	Model	GPT-5	Claude	Gemini	Avg.	Final Rank	Our Rank
Review Summarization	GPT-5	5.9	5.3	5.1	5.43	3	3
	Claude 4 Sonnet	5.4	5.8	5.6	5.60	2	2
	Gemini-2.5 Pro	5.7	5.6	6.1	5.80	1	1
Review Generation	GPT-5	4.6	4.0	3.8	4.13	2	2
	Claude 4 Sonnet	4.0	4.3	4.0	4.10	3	3
	Gemini-2.5 Pro	4.5	4.3	4.8	4.53	1	1

Appendix

Table 7: LLM-as-a-judge evaluation with scores averaged across three judge models. Individual LLM judges exhibit self-preference bias, tending to favor outputs from their own model family. The final ranking is determined by the averaged scores, which align with our rank using ROUGE-L and BLEU-1 based evaluation in the Books domain.

Dataset	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Beauty	GPT-5	0.544	0.946	0.175	0.290	0.203	0.085	0.231	0.118
	Claude-4 Sonnet	0.544	0.889	0.246	0.323	0.195	0.117	0.231	0.118
	Gemini-2.5 Pro	0.719	1.221	0.228	0.349	0.230	0.129	0.284	0.159
Arts Crafts	GPT-5	0.167	0.440	0.367	0.531	0.209	0.092	0.246	0.131
	Claude-4 Sonnet	0.347	0.739	0.267	0.430	0.250	0.170	0.230	0.134

Appendix

Table 8: Overall performance of representative LLMs across four tasks in the rest eight single domains of MemoryCD with long-context memory prompting.

Figure 5: Performance comparisons of 14 frontier long-context LLMs on Books based on MemoryBank.

Figure 6: Performance comparisons of 14 frontier long-context LLMs on Books based on A-Mem.

Figure 7: Performance comparisons of 14 frontier long-context LLMs on Home & Kitchen based on MemoryBank.

Figure 8: Performance comparisons of 14 frontier long-context LLMs on Home & Kitchen based on A-Mem.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.289	0.602	0.443	0.604	0.182	0.082	0.123	0.145
	Claude-4 Sonnet	0.374	0.749	0.354	0.511	0.228	0.139	0.144	0.176
	Gemini-2.5 Pro	0.337	0.719	0.378	0.561	0.222	0.141	0.161	0.200
Mem0	GPT-5	0.354	0.735	0.646	0.744	0.209	0.115	0.133	0.153
	Claude-4 Sonnet	0.411	0.741	0.565	0.661	0.238	0.145	0.144	0.174

Appendix

Table 9: Comparison of memory methods on Electronics using representative LLMs.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.261	0.551	0.446	0.610	0.185	0.077	0.134	0.140
	Claude-4 Sonnet	0.330	0.630	0.276	0.400	0.189	0.117	0.145	0.158
	Gemini-2.5 Pro	0.303	0.679	0.315	0.444	0.222	0.133	0.178	0.207
Mem0	GPT-5	0.261	0.561	0.606	0.707	0.221	0.122	0.151	0.163
	Claude-4 Sonnet	0.339	0.623	0.521	0.613	0.227	0.152	0.152	0.167

Appendix

Table 10: Comparison of memory methods on Home & Kitchen using representative LLMs.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.279	0.597	0.315	0.435	0.161	0.053	0.131	0.139
	Claude-4 Sonnet	0.432	0.767	0.203	0.304	0.196	0.118	0.138	0.148
	Gemini-2.5 Pro	0.323	0.665	0.266	0.417	0.200	0.110	0.183	0.204
Mem0	GPT-5	0.331	0.635	0.578	0.685	0.175	0.082	0.150	0.164
	Claude-4 Sonnet	0.361	0.672	0.583	0.683	0.197	0.111	0.151	0.163

Appendix

Table 11: Comparison of memory methods on Personal Care using representative LLMs.

Explore similar work

Mar 24, 2026

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

Shuochen Liu, Junyi Zhu, Long Shu +11

University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong +2

Memory Agent Benchmarks Long-Term Conversational Memory Persona Consistency Personalized Language Models User Preference Modeling

Ask PDF

Jun 22, 2026

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/

Wenya Xie, Shengming Zhou, Zelin Li +9

University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2

Agent Memory LLM Agent Memory Persistent Memory for Language Models Long-Horizon Agent Evaluation Memory Agent Benchmarks User Modeling+1

Ask PDF

Mar 9, 2026

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment

As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation benchmark. Existing benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management. Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework.

Jianfei Xiao, Xiang Yu, Chengbing Wang +8

University of Science and Technology of China · National University of Singapore

LLM Personalization Personalized Language Model Alignment Persistent Memory for Language Models LLM Evaluation

Ask PDF