MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

Weizhi Zhang; Xiaokai Wei; Wei-Chieh Huang; Zheng Hui; Chen Wang; Michelle Gong; Philip S. Yu

cs.CLMar 26, 2026

MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

Authors: Weizhi Zhang, Xiaokai Wei, Wei-Chieh Huang, Zheng Hui, Chen Wang, Michelle Gong, Philip S. Yu

Organizations: Roblox · University of Illinois Chicago · University of Cambridge

Abstract

Recent advancements in Large Language Models (LLMs) have expanded context windows to million-token scales, yet benchmarks for evaluating memory remain limited to short-session synthetic dialogues. We introduce MemoryCD, the first large-scale, user-centric, cross-domain memory benchmark derived from lifelong real-world behaviors in the Amazon Review dataset. Unlike existing memory datasets that rely on scripted personas to generate synthetic user data, MemoryCD tracks authentic user interactions across years and multiple domains. We construct a multi-faceted long-context memory evaluation pipeline of 14 state-of-the-art LLM base models with 6 memory method baselines on 4 distinct personalization tasks over 12 diverse domains to evaluate an agent's ability to simulate real user behaviors in both single and cross-domain settings. Our analysis reveals that existing memory methods are far from user satisfaction in various domains, offering the first testbed for cross-domain life-long personalization evaluation.

Figures & tables

Figure 1: Comparison memory benchmarks: MemoryCD (ours) captures cross-domain real-user activities over long time horizons. LaMP ( Salemi et al., 2024 ) focuses only on the short-term single-domain user behaviors; LoCoMo ( Maharana et al., 2024 ) represents non-authentic LLM-simulated user behaviors.

Figure 2: The MemoryCD benchmark spans 12 real-world domains and evaluates 6 SOTA memory methods . Different from other memory benchmarks targeting one specific memory stage (mostly retrieval), we design 4 basic tasks with 2 settings to provide end-to-end user satisfaction evaluation grounded on the lifelong real user behaviors (Table 2 ).

Data	Memory	Context	# of	Long	User	Cross-	Real
Source	Benchmarks	Length	Sessions	Timespan	Level	Domain	Feedback
Synthetic	LoCoMo ( Maharana et al., 2024 )	9K	35	✗	✓ ✗	✗	✗
	PrefEval ( Zhao et al., 2025 )	100K	–	✗	✓ ✗	✓ ✗	✗
	LongMemEval ( Wu et al., 2025a )	1.5M	500	✓ ✗	✓ ✗	✗	✗
	PersonaMem ( Jiang et al., 2025 )	1M	60	✓	✓ ✗	✓ ✗	✗
	MemBench ( Tan et al., 2025a )	100K	0.5K-8K	✓ ✗	✓ ✗	✗	✗

Table 1: Comparison of representative memory benchmarks. ✓ : fully covered; ✓ ✗ : partially covered; ✗ : not covered. Synthetic denotes simulated user behaviors. Real User denotes data constructed from real user interaction histories. # of Sessions is the number of sessions per individual users.

Aspect	★ Rating Prediction	Item Ranking	Review Summarization	Review Generation
Category	User decision-making (quantitative preference modeling)	User decision-making (relative preference modeling)	User text generation (attitude & sentiment reflection)	User text generation (preference & linguistic expression style)
Goal	Predict user’s rating (1–5) for a target item.	Rank candidate items by expected user interest.	Generate a concise title reflecting user tone and sentiment.	Generate full review text expressing user preferences and style.
Input	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$	User memory $\mathcal{M}_{u}$ ; candidate set $C$	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$ ; user review context $x_{u,i}^{*}$	User memory $\mathcal{M}_{u}$ ; item metadata $m_{i}$
Output	Rating $\hat{r}_{u,i}\in\{1,\dots,5\}$	Ranking $\pi_{u}(C)$	Short title $x_{u,i}^{\prime}$	Full review $x_{u,i}^{*}$
Metrics	MAE, RMSE	NDCG@ $K$	ROUGE-L, BLEU-1	ROUGE-L, BLEU-1

Table 2: Summary of the four personalization tasks in MemoryCD .

Figure 3: Ratios of overlapping users across domains . Each cell shows the percentage of users in one domain that also appear in another, revealing feasible cross-domain personalization settings.

Dataset	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Books	GPT-5-Nano	0.543	0.872	0.135	0.164	0.122	0.055	0.110	0.145
	GPT-5-Mini	0.464	0.816	0.161	0.190	0.138	0.063	0.130	0.174
	GPT-5	0.330	0.624	0.146	0.177	0.137	0.064	0.132	0.195
	Claude-4 Opus	0.611	0.954	0.142	0.183	0.126	0.070	0.104	0.143
	Claude-4 Sonnet	0.670	1.000	0.109	0.163	0.145	0.089	0.110	0.151

Table 3: Performance comparison of frontier LLMs in single-domain with long-context memory prompting.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Long-Context	GPT-5	0.330	0.624	0.146	0.177	0.137	0.064	0.132	0.195
	Claude-4 Sonnet	0.670	1.000	0.109	0.163	0.145	0.089	0.110	0.151
	Gemini-2.5 Pro	0.401	0.762	0.206	0.229	0.208	0.128	0.162	0.207
Mem0	GPT-5	0.419	0.698	0.184	0.217	0.173	0.097	0.143	0.195
	Claude-4 Sonnet	0.491	0.772	0.172	0.207	0.202	0.126	0.141	0.182

Table 4: Comparison of memory methods on Books using representative LLMs.

Figure 4: Performance comparisons using different memory sources evaluated on the Home & Kitchen . Each radar plot corresponds to a representative frontier long-context LLM (GPT-5, Claude-4 Sonnet, Gemini-2.5 Pro). Curves correspond to different memory sources: no memory (Source N/A), single-domain memories ( Books , Electronics , Personal Care ), and aggregated cross-domain memory (All 3 Sources). Scales are unified for fair comparison (MAE and RMSE scales are reversed).

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE	BLEU	ROUGE	BLEU
Long-Context	GPT-5	0.354	0.675	0.179	0.299	0.174	0.070	0.104	0.111
	Claude-4 Sonnet	0.570	0.839	0.108	0.212	0.192	0.108	0.125	0.145
	Gemini-2.5 Pro	0.391	0.723	0.140	0.255	0.204	0.119	0.152	0.180
Mem0	GPT-5	0.367	0.661	0.146	0.284	0.189	0.094	0.112	0.122
	Claude-4 Sonnet	0.596	0.823	0.140	0.255	0.197	0.116	0.127	0.146

Table 5: Cross-domain evaluation of different memory methods on Home & Kitchen . Memory includes source domains of Books, Electronics, and Personal Care .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Context Length	Max # Sessions	Avg # Sessions	Min # Sessions	Filtering Thresholds	# of Users	# of Samples
Beauty	11,137	165	81	54	50	57	171
Arts	49,101	801	338	204	200	50	150
Automotive	27,104	512	258	203	200	64	192
Personal Care	202,096	2,973	839	504	500	128	384
Books	272,372	3,888	1,552	1,002	1,000	89	267
Electronics	184,084	2,602	781	502	500	94	282

Appendix

Table 6: Domain-level statistics for the proposed ultra long-context user-centric memory benchmark. Only users exceeding the minimum interaction threshold are retained; each user contributes three evaluation samples.

Task	Model	GPT-5	Claude	Gemini	Avg.	Final Rank	Our Rank
Review Summarization	GPT-5	5.9	5.3	5.1	5.43	3	3
	Claude 4 Sonnet	5.4	5.8	5.6	5.60	2	2
	Gemini-2.5 Pro	5.7	5.6	6.1	5.80	1	1
Review Generation	GPT-5	4.6	4.0	3.8	4.13	2	2
	Claude 4 Sonnet	4.0	4.3	4.0	4.10	3	3
	Gemini-2.5 Pro	4.5	4.3	4.8	4.53	1	1

Appendix

Table 7: LLM-as-a-judge evaluation with scores averaged across three judge models. Individual LLM judges exhibit self-preference bias, tending to favor outputs from their own model family. The final ranking is determined by the averaged scores, which align with our rank using ROUGE-L and BLEU-1 based evaluation in the Books domain.

Dataset	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Beauty	GPT-5	0.544	0.946	0.175	0.290	0.203	0.085	0.231	0.118
	Claude-4 Sonnet	0.544	0.889	0.246	0.323	0.195	0.117	0.231	0.118
	Gemini-2.5 Pro	0.719	1.221	0.228	0.349	0.230	0.129	0.284	0.159
Arts Crafts	GPT-5	0.167	0.440	0.367	0.531	0.209	0.092	0.246	0.131
	Claude-4 Sonnet	0.347	0.739	0.267	0.430	0.250	0.170	0.230	0.134

Appendix

Table 8: Overall performance of representative LLMs across four tasks in the rest eight single domains of MemoryCD with long-context memory prompting.

Figure 5: Performance comparisons of 14 frontier long-context LLMs on Books based on MemoryBank.

Figure 6: Performance comparisons of 14 frontier long-context LLMs on Books based on A-Mem.

Figure 7: Performance comparisons of 14 frontier long-context LLMs on Home & Kitchen based on MemoryBank.

Figure 8: Performance comparisons of 14 frontier long-context LLMs on Home & Kitchen based on A-Mem.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.289	0.602	0.443	0.604	0.182	0.082	0.123	0.145
	Claude-4 Sonnet	0.374	0.749	0.354	0.511	0.228	0.139	0.144	0.176
	Gemini-2.5 Pro	0.337	0.719	0.378	0.561	0.222	0.141	0.161	0.200
Mem0	GPT-5	0.354	0.735	0.646	0.744	0.209	0.115	0.133	0.153
	Claude-4 Sonnet	0.411	0.741	0.565	0.661	0.238	0.145	0.144	0.174

Appendix

Table 9: Comparison of memory methods on Electronics using representative LLMs.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.261	0.551	0.446	0.610	0.185	0.077	0.134	0.140
	Claude-4 Sonnet	0.330	0.630	0.276	0.400	0.189	0.117	0.145	0.158
	Gemini-2.5 Pro	0.303	0.679	0.315	0.444	0.222	0.133	0.178	0.207
Mem0	GPT-5	0.261	0.561	0.606	0.707	0.221	0.122	0.151	0.163
	Claude-4 Sonnet	0.339	0.623	0.521	0.613	0.227	0.152	0.152	0.167

Appendix

Table 10: Comparison of memory methods on Home & Kitchen using representative LLMs.

Method	Model	Rating $\downarrow$		Ranking $\uparrow$		Summarization $\uparrow$		Generation $\uparrow$
		MAE	RMSE	N@1	N@3	ROUGE-L	BLEU-1	ROUGE-L	BLEU-1
Long-Context	GPT-5	0.279	0.597	0.315	0.435	0.161	0.053	0.131	0.139
	Claude-4 Sonnet	0.432	0.767	0.203	0.304	0.196	0.118	0.138	0.148
	Gemini-2.5 Pro	0.323	0.665	0.266	0.417	0.200	0.110	0.183	0.204
Mem0	GPT-5	0.331	0.635	0.578	0.685	0.175	0.082	0.150	0.164
	Claude-4 Sonnet	0.361	0.672	0.583	0.683	0.197	0.111	0.151	0.163

Appendix

Table 11: Comparison of memory methods on Personal Care using representative LLMs.

Explore similar work

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
Mar 24, 2026Shuochen Liu, Junyi Zhu, Long Shu +11Memory Agent Benchmarks Long-Term Conversational Memory
University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong +2
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Mar 9, 2026Jianfei Xiao, Xiang Yu, Chengbing Wang +8LLM Personalization Personalized Language Model Alignment
University of Science and Technology of China · National University of Singapore

Mar 24, 2026J/K move · Enter open · S save

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin+10

University of Science and Technology of China · ESAT-PSI, KU Leuven · City University of Hong Kong+2

Memory Agent Benchmarks Long-Term Conversational Memory Persona Consistency Personalized Language Models User Preference Modeling

Open paper AskPDF arXiv

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

Abstract

Explore similar work

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment