DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Yifei Tao; Xinyu Zhong; Henry Hengyuan Zhao; Fanyi Wang; Tengda Guo; Wentao Qiu; Ying Wang; Liujian Tang

cs.AIOct 2, 2026

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Authors: Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang

Organizations: Nanyang Technological University · Stepfun · National University of Singapore · University of Hong Kong

Abstract

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.

Figures & tables

Figure 1: DyadMem is a dual-domain, full-pipeline memory benchmark. Across multi-session user–agent interactions, Capture extracts current-session memories, Update reconciles them with the prior bank, Recall selects valid terminal-bank entries, and QA uses the selected evidence. The evolving bank contains shared episodic memory, user-side memory, and User-conditioned Relational Agent Memory (URAM); unlike general agent memory that transfers across users, URAM records how this particular agent should work with this particular user.

Benchmark	User-side memory	User-conditioned agent memory	Per-session gold memory	Gold lifecycle annotations	Query-to-bank supports
LoCoMo [ 23 ]	$\checkmark$	$\times$	$\times$	$\times$	$\triangle$
LongMemEval [ 41 ]	$\checkmark$	$\times$	$\times$	$\times$	$\triangle$
MultiSessionCollab [ 25 ]	$\triangle$	$\triangle$	$\times$	$\times$	$\times$
AMemGym [ 5 ]	$\checkmark$	$\times$	$\triangle$	$\times$	$\times$
HaluMem [ 3 ]	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\triangle$
MemOps [ 14 ]	$\checkmark$	$\times$	$\triangle$	$\checkmark$	$\triangle$

Table 1: Comparison with related works.

\checkmark

denotes direct support,

\triangle

related but non-equivalent supervision, and

\times

no corresponding supervision. Per-session gold memory denotes the entries to extract from that session. Gold lifecycle annotations denote add, revise, and invalidate operations on the prior bank and do not repeat the per-session extraction set. Query-to-bank supports are gold links from each QA query to entries that remain valid in the terminal memory bank and are used as the recall target.

Figure 2: Dataset composition and scale of DyadMem across five major domains and 25 subdomains. Our benchmark shows balanced data composition and large-scale sessions and QAs.

	Capture			Update			Recall				QA
Model	F1 $\uparrow$	P $\uparrow$	R $\uparrow$	op-F1 $\uparrow$	Add $\uparrow$	Del $\uparrow$	@ $n$ $\uparrow$	@ $n$ +4 $\uparrow$	All $\uparrow$	P $\uparrow$	Grounded $\uparrow$	OA $\downarrow$	Ung. $\downarrow$	Gold-Mem. $\uparrow$	Full-Pipe. $\uparrow$
Open-weight Models
Kimi-K3	28.14	58.27	18.55	31.69	22.47	40.92	62.09	70.14	69.16	55.29	69.26	9.21	7.94	94.99	74.85
GLM-5.2	26.73	56.73	17.49	21.06	18.27	23.85	56.71	61.66	59.99	59.15	67.50	11.44	5.51	94.30	71.40
DeepSeek-V4-Pro	30.70	63.23	20.27	22.48	16.80	28.16	59.32	69.09	70.42	41.29	60.90	13.00	8.45	92.54	66.16
MiniMax-M2.5	22.22	49.85	14.30	23.43	14.66	32.19	53.79	62.45	62.70	52.37	56.50	14.52	13.51	93.88	65.30

Table 2: Main results on DyadMem. Capture

=

F1/P/R; gold-conditioned standalone Update

=

op-F1/Add/Del; Recall

=

@n

n

+4/@All/P; QA

=

Grounded/OA/Ung./Gold-Memory/Full-Pipeline. OA

=

over-abstention, Ung.

=

ungrounded-correct. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

	Update			QA
Model	$\Delta$ (op-F1) $\uparrow$	$\Delta$ (Add) $\uparrow$	$\Delta$ (Del) $\uparrow$	$\Delta$ (QA) $\uparrow$
Open-weight Models
DeepSeek-V4-Pro	$+$ 3.37	$+$ 5.71	$+$ 2.84	$+$ 9.77
Qwen3.5-9B	$+$ 4.98	$+$ 3.33	$+$ 4.49	$+$ 7.21
Qwen3.5-27B	$+$ 4.37	$+$ 5.01	$+$ 1.77	$+$ 5.94
MiniMax-M2.5	$+$ 1.89	$+$ 1.56	$+$ 2.25	$+$ 5.85

Table 3: Controlled URAM comparison on 500 paired episodes. All entries are percentage-point differences.

\Delta=

User-conditioned Relational Agent Memory

-

Agent-only. Controlled URAM Update

=

op-F1 / Add F1 / Del F1. QA

=

Full-Pipeline accuracy with Recall restricted to URAM on 1,735 URAM-only questions. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

	User-centric				Agent-centric
Model	Cap. $\uparrow$	Upd. $\uparrow$	Rec. $\uparrow$	QA $\uparrow$	Cap. $\uparrow$	Upd. $\uparrow$	Rec. $\uparrow$	QA $\uparrow$
Lower bounds & references
No Memory	—	—	—	—	—	—	—	—
Open-weight Models
Kimi-K3	26.12	31.68	61.87	69.07	30.18	31.70	62.31	80.59
GLM-5.2	24.56	21.20	56.01	66.03	28.92	20.93	57.41	76.73

Table 4: User- and agent-centric results on DyadMem (3,065 episodes, 61,210 QA instances). User-centric

n{=}30{,}468

; agent-centric

n{=}30{,}742

. Capture

=

F1; gold-conditioned standalone Update

=

op-F1; Recall

=

@n

; QA

=

Full-Pipeline. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Role	English	Degree
1	Career & Learning	Reads English	B.S. in computer science
2	Family & Living	With a translation	M.S. in software engineering
3	Interpersonal & Communication	With a translation	M.S. in artificial intelligence
4	Personal Preferences & Resources	Reads English	M.S. in data science
5	Physical Health	With a translation	M.S. in computer science
6	Second-pass sample	Reads English	B.S. in software engineering

Appendix

Table 6: The nine quality-control annotators. All worked full time for two months.

Status	Files	Share
Pending	116	14.9%
In review	0	0%
Needs fix	13	1.7%
Accepted	633	81.8%
Rejected	14	1.8%
Total	776	100%

Appendix

Table 7: Status of the 776 files in the batch shown in Figure 4 .

Figure 4: The quality-control page for one accepted Physical Health episode. The center pane shows the dialogue and the session-level Capture and Update labels. The right pane shows a question, its answer, and the gold recall memories. Status for the batch is in Table 7 .

Model	Unsafe Del. Rate (%) $\downarrow$
Llama-3.1-8B-Instruct	98.51
Qwen3.5-4B	95.32
Qwen3-30B-A3B	94.94
Qwen3.5-35B-A3B	94.72
Qwen3-8B	92.49
Qwen3-4B	91.90

Appendix

Table 8: Unsafe deletion rate on the full DyadMem benchmark (3,065 episodes). Unsafe deletion rate

=

incorrect deletes / (correct

+

incorrect deletes); lower is better.

Figure 5: Unsafe deletion rate of open-source and proprietary models.

	User-conditioned Relational Agent Memory	Agent-only
Source session	$D_{t}$ (User $+$ Assistant)	$D_{t}$ (User $+$ Assistant)
Captured types	user_profile , fact , URAM	General Agent Memory
Excluded types	episodic	episodic , user_profile , fact
Session 1, Capture	$C_{1}=\operatorname{Capture}(D_{1})$	$C_{1}^{\text{Agent-only}}=\operatorname{Capture}(D_{1})$
Memory bank	$B_{1}=C_{1}$	$B^{\prime}_{1}=C_{1}^{\text{Agent-only}}$
Session 2, Capture	$C_{2}=\operatorname{Capture}(D_{2})$	$C_{2}^{\text{Agent-only}}=\operatorname{Capture}(D_{2})$

Appendix

Table 9: Controlled comparison. Both variants read the same user–assistant session. They differ in the types captured and then evolved. Update takes the variant’s previous bank and its current capture. Pipeline QA reads the terminal bank.

	Recall@ $n$ $\uparrow$					MRR $\uparrow$					nDCG $\uparrow$					QA $\uparrow$
Model	dense	base	large	v2	model	dense	base	large	v2	model	dense	base	large	v2	model	dense	base	large	v2	model
Qwen3-4B	39.32	29.97	36.59	41.25	32.11	74.25	60.93	72.07	77.47	70.56	62.05	54.65	60.35	63.78	42.61	47.72	47.74	47.75	48.47	43.57
Qwen3-8B	39.32	29.97	36.59	41.25	37.98	74.25	60.93	72.07	77.48	73.51	62.05	54.65	60.35	63.79	50.19	50.86	50.64	51.14	52.02	47.51
Qwen3.5-4B	39.32	29.97	36.59	41.25	51.08	74.25	60.93	72.07	77.48	83.29	62.05	54.65	60.35	63.79	63.68	26.83	26.77	26.96	27.38	26.26
Qwen3.8-27B	39.32	29.96	36.59	41.25	60.34	74.25	60.93	72.07	77.48	91.72	62.05	54.65	60.35	63.79	71.59	52.07	51.97	52.01	53.30	53.81
Llama-3.1-8B-Instruct	39.32	29.97	36.59	41.25	38.69	74.26	60.93	72.07	77.48	78.94	62.06	54.65	60.35	63.79	49.25	24.53	23.14	24.22	25.32	23.84

Appendix

Table 10: Retrieval and QA on five models. Dense is BGE-M3. Base, large, and v2 are bge-reranker-base , bge-reranker-large , and bge-reranker-v2-m3 . Model is the tested model’s own selection. Bold and underlined values mark the best and second-best track within each model and metric.

	GPT-5-mini		Gemini-2.5-Flash		Claude-Haiku-4.5
Model	Full-Pipeline QA $\uparrow$	Rank $\downarrow$	Full-Pipeline QA $\uparrow$	Rank $\downarrow$	Full-Pipeline QA $\uparrow$	Rank $\downarrow$
Kimi-K3	74.92	1	63.37	1	63.05	1
GLM-5.2	70.62	2	58.71	2	61.68	2
GPT-5.6-Sol	70.28	3	57.55	4	56.96	4
Doubao-Seed-2.1-Pro	67.98	4	58.09	3	57.46	3
DeepSeek-V4-Pro	66.12	5	53.79	5	54.08	5

Appendix

Table 11: Full-Pipeline QA on DyadMem-Simple (300 episodes, 5,935 QA instances) under three judges. Rank is 1-based among the 20 models and is computed independently per judge. Rows are ordered by GPT-5-mini rank.

Figure 6: Headmap of three error categories.

Figure 7: Capability of 6 memory types across 20 LLMs.

Figure 8: Performance visualization on 25 domains.

Model	Release Time	Source
Closed-source Models
GPT-4o-mini	2024-07-18	https://openai.com/index/gpt-4o-mini/
GPT-5-mini	2025-08-07	https://openai.com/index/gpt-5-mini/
GPT-5.6 Sol	2026-07-09	https://openai.com/index/gpt-5-6/
Gemini-2.5-Flash	2025-06-17	https://deepmind.google/models/gemini/flash/
Gemini-3.1-Pro-Preview	2026-02-19	https://deepmind.google/models/gemini/pro/

Appendix

Table 12: The release time and model source of LLMs used in our benchmark.

Model	Backend	Reasoning Mode	Temperature
GPT-4o-mini	OpenAI-compatible	non-reasoning	0
GPT-5-mini	OpenAI-compatible	Disabled	0
GPT-5.6-sol	OpenAI-compatible	Disabled	0
Gemini-2.5-Flash	Gemini Native API	Disabled	0
Gemini-3.1-Pro-Preview	Gemini Native API	Disabled	0
Claude-Haiku-4.5	Anthropic Native API	Disabled	0

Appendix

Table 13: Model Backend and Inference Configuration

Name	Model License	Code License
GPT-4o-mini	Proprietary	Proprietary
GPT-5-mini	Proprietary	Proprietary
GPT-5.6 Sol	Proprietary	Proprietary
Gemini-2.5-Flash	Proprietary	Proprietary
Gemini-3.1-Pro-Preview	Proprietary	Proprietary
Claude-Haiku-4.5	Proprietary	Proprietary

Appendix

Table 14: Summary of licenses of models evaluated in our benchmark.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 190 papers

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Agent Memory Persistent Memory for Language Models
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Apr 21, 2026Md Nayem Uddin, Kumar Shubham, Eduardo Blanco +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Arizona State University · Genies · University of Arizona
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0

Jun 19, 2026J/K move · Enter open · S save

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management Personal AI Agents Memory Management for LLM Agents

Open paper AskPDF arXiv

Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user's current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user's memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available: https://github.com/galaxyChen/AdaMem

Abstract

Agent Memory

Explore similar work

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

DolphinBench: Mapping the Pareto Frontier of Agent Memory