DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Yifei Tao; Xinyu Zhong; Henry Hengyuan Zhao; Fanyi Wang; Tengda Guo; Wentao Qiu; Ying Wang; Liujian Tang

cs.AIOct 2, 2026

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Authors: Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang

Organizations: Nanyang Technological University · Stepfun · National University of Singapore · University of Hong Kong

Abstract

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.

Figures & tables

Figure 1: DyadMem is a dual-domain, full-pipeline memory benchmark. Across multi-session user–agent interactions, Capture extracts current-session memories, Update reconciles them with the prior bank, Recall selects valid terminal-bank entries, and QA uses the selected evidence. The evolving bank contains shared episodic memory, user-side memory, and User-conditioned Relational Agent Memory (URAM); unlike general agent memory that transfers across users, URAM records how this particular agent should work with this particular user.

Benchmark	User-side memory	User-conditioned agent memory	Per-session gold memory	Gold lifecycle annotations	Query-to-bank supports
LoCoMo [ 23 ]	$\checkmark$	$\times$	$\times$	$\times$	$\triangle$
LongMemEval [ 41 ]	$\checkmark$	$\times$	$\times$	$\times$	$\triangle$
MultiSessionCollab [ 25 ]	$\triangle$	$\triangle$	$\times$	$\times$	$\times$
AMemGym [ 5 ]	$\checkmark$	$\times$	$\triangle$	$\times$	$\times$
HaluMem [ 3 ]	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\triangle$
MemOps [ 14 ]	$\checkmark$	$\times$	$\triangle$	$\checkmark$	$\triangle$

Table 1: Comparison with related works.

\checkmark

denotes direct support,

\triangle

related but non-equivalent supervision, and

\times

no corresponding supervision. Per-session gold memory denotes the entries to extract from that session. Gold lifecycle annotations denote add, revise, and invalidate operations on the prior bank and do not repeat the per-session extraction set. Query-to-bank supports are gold links from each QA query to entries that remain valid in the terminal memory bank and are used as the recall target.

Figure 2: Dataset composition and scale of DyadMem across five major domains and 25 subdomains. Our benchmark shows balanced data composition and large-scale sessions and QAs.

	Capture			Update			Recall				QA
Model	F1 $\uparrow$	P $\uparrow$	R $\uparrow$	op-F1 $\uparrow$	Add $\uparrow$	Del $\uparrow$	@ $n$ $\uparrow$	@ $n$ +4 $\uparrow$	All $\uparrow$	P $\uparrow$	Grounded $\uparrow$	OA $\downarrow$	Ung. $\downarrow$	Gold-Mem. $\uparrow$	Full-Pipe. $\uparrow$
Open-weight Models
Kimi-K3	28.14	58.27	18.55	31.69	22.47	40.92	62.09	70.14	69.16	55.29	69.26	9.21	7.94	94.99	74.85
GLM-5.2	26.73	56.73	17.49	21.06	18.27	23.85	56.71	61.66	59.99	59.15	67.50	11.44	5.51	94.30	71.40
DeepSeek-V4-Pro	30.70	63.23	20.27	22.48	16.80	28.16	59.32	69.09	70.42	41.29	60.90	13.00	8.45	92.54	66.16
MiniMax-M2.5	22.22	49.85	14.30	23.43	14.66	32.19	53.79	62.45	62.70	52.37	56.50	14.52	13.51	93.88	65.30

Table 2: Main results on DyadMem. Capture

=

F1/P/R; gold-conditioned standalone Update

=

op-F1/Add/Del; Recall

=

@n

n

+4/@All/P; QA

=

Grounded/OA/Ung./Gold-Memory/Full-Pipeline. OA

=

over-abstention, Ung.

=

ungrounded-correct. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

	Update			QA
Model	$\Delta$ (op-F1) $\uparrow$	$\Delta$ (Add) $\uparrow$	$\Delta$ (Del) $\uparrow$	$\Delta$ (QA) $\uparrow$
Open-weight Models
DeepSeek-V4-Pro	$+$ 3.37	$+$ 5.71	$+$ 2.84	$+$ 9.77
Qwen3.5-9B	$+$ 4.98	$+$ 3.33	$+$ 4.49	$+$ 7.21
Qwen3.5-27B	$+$ 4.37	$+$ 5.01	$+$ 1.77	$+$ 5.94
MiniMax-M2.5	$+$ 1.89	$+$ 1.56	$+$ 2.25	$+$ 5.85

Table 3: Controlled URAM comparison on 500 paired episodes. All entries are percentage-point differences.

\Delta=

User-conditioned Relational Agent Memory

-

Agent-only. Controlled URAM Update

=

op-F1 / Add F1 / Del F1. QA

=

Full-Pipeline accuracy with Recall restricted to URAM on 1,735 URAM-only questions. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

	User-centric				Agent-centric
Model	Cap. $\uparrow$	Upd. $\uparrow$	Rec. $\uparrow$	QA $\uparrow$	Cap. $\uparrow$	Upd. $\uparrow$	Rec. $\uparrow$	QA $\uparrow$
Lower bounds & references
No Memory	—	—	—	—	—	—	—	—
Open-weight Models
Kimi-K3	26.12	31.68	61.87	69.07	30.18	31.70	62.31	80.59
GLM-5.2	24.56	21.20	56.01	66.03	28.92	20.93	57.41	76.73

Table 4: User- and agent-centric results on DyadMem (3,065 episodes, 61,210 QA instances). User-centric

n{=}30{,}468

; agent-centric

n{=}30{,}742

. Capture

=

F1; gold-conditioned standalone Update

=

op-F1; Recall

=

@n

; QA

=

Full-Pipeline. rose , sky , and sage mark the top three; Bold and underlined values mark the best and second-best.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Role	English	Degree
1	Career & Learning	Reads English	B.S. in computer science
2	Family & Living	With a translation	M.S. in software engineering
3	Interpersonal & Communication	With a translation	M.S. in artificial intelligence
4	Personal Preferences & Resources	Reads English	M.S. in data science
5	Physical Health	With a translation	M.S. in computer science
6	Second-pass sample	Reads English	B.S. in software engineering

Appendix

Table 6: The nine quality-control annotators. All worked full time for two months.

Status	Files	Share
Pending	116	14.9%
In review	0	0%
Needs fix	13	1.7%
Accepted	633	81.8%
Rejected	14	1.8%
Total	776	100%

Appendix

Table 7: Status of the 776 files in the batch shown in Figure 4 .

Figure 4: The quality-control page for one accepted Physical Health episode. The center pane shows the dialogue and the session-level Capture and Update labels. The right pane shows a question, its answer, and the gold recall memories. Status for the batch is in Table 7 .

Model	Unsafe Del. Rate (%) $\downarrow$
Llama-3.1-8B-Instruct	98.51
Qwen3.5-4B	95.32
Qwen3-30B-A3B	94.94
Qwen3.5-35B-A3B	94.72
Qwen3-8B	92.49
Qwen3-4B	91.90

Appendix

Table 8: Unsafe deletion rate on the full DyadMem benchmark (3,065 episodes). Unsafe deletion rate

=

incorrect deletes / (correct

+

incorrect deletes); lower is better.

Figure 5: Unsafe deletion rate of open-source and proprietary models.

	User-conditioned Relational Agent Memory	Agent-only
Source session	$D_{t}$ (User $+$ Assistant)	$D_{t}$ (User $+$ Assistant)
Captured types	user_profile , fact , URAM	General Agent Memory
Excluded types	episodic	episodic , user_profile , fact
Session 1, Capture	$C_{1}=\operatorname{Capture}(D_{1})$	$C_{1}^{\text{Agent-only}}=\operatorname{Capture}(D_{1})$
Memory bank	$B_{1}=C_{1}$	$B^{\prime}_{1}=C_{1}^{\text{Agent-only}}$
Session 2, Capture	$C_{2}=\operatorname{Capture}(D_{2})$	$C_{2}^{\text{Agent-only}}=\operatorname{Capture}(D_{2})$

Appendix

Table 9: Controlled comparison. Both variants read the same user–assistant session. They differ in the types captured and then evolved. Update takes the variant’s previous bank and its current capture. Pipeline QA reads the terminal bank.

	Recall@ $n$ $\uparrow$					MRR $\uparrow$					nDCG $\uparrow$					QA $\uparrow$
Model	dense	base	large	v2	model	dense	base	large	v2	model	dense	base	large	v2	model	dense	base	large	v2	model
Qwen3-4B	39.32	29.97	36.59	41.25	32.11	74.25	60.93	72.07	77.47	70.56	62.05	54.65	60.35	63.78	42.61	47.72	47.74	47.75	48.47	43.57
Qwen3-8B	39.32	29.97	36.59	41.25	37.98	74.25	60.93	72.07	77.48	73.51	62.05	54.65	60.35	63.79	50.19	50.86	50.64	51.14	52.02	47.51
Qwen3.5-4B	39.32	29.97	36.59	41.25	51.08	74.25	60.93	72.07	77.48	83.29	62.05	54.65	60.35	63.79	63.68	26.83	26.77	26.96	27.38	26.26
Qwen3.8-27B	39.32	29.96	36.59	41.25	60.34	74.25	60.93	72.07	77.48	91.72	62.05	54.65	60.35	63.79	71.59	52.07	51.97	52.01	53.30	53.81
Llama-3.1-8B-Instruct	39.32	29.97	36.59	41.25	38.69	74.26	60.93	72.07	77.48	78.94	62.06	54.65	60.35	63.79	49.25	24.53	23.14	24.22	25.32	23.84

Appendix

Table 10: Retrieval and QA on five models. Dense is BGE-M3. Base, large, and v2 are bge-reranker-base , bge-reranker-large , and bge-reranker-v2-m3 . Model is the tested model’s own selection. Bold and underlined values mark the best and second-best track within each model and metric.

	GPT-5-mini		Gemini-2.5-Flash		Claude-Haiku-4.5
Model	Full-Pipeline QA $\uparrow$	Rank $\downarrow$	Full-Pipeline QA $\uparrow$	Rank $\downarrow$	Full-Pipeline QA $\uparrow$	Rank $\downarrow$
Kimi-K3	74.92	1	63.37	1	63.05	1
GLM-5.2	70.62	2	58.71	2	61.68	2
GPT-5.6-Sol	70.28	3	57.55	4	56.96	4
Doubao-Seed-2.1-Pro	67.98	4	58.09	3	57.46	3
DeepSeek-V4-Pro	66.12	5	53.79	5	54.08	5

Appendix

Table 11: Full-Pipeline QA on DyadMem-Simple (300 episodes, 5,935 QA instances) under three judges. Rank is 1-based among the 20 models and is computed independently per judge. Rows are ordered by GPT-5-mini rank.

Figure 6: Headmap of three error categories.

Figure 7: Capability of 6 memory types across 20 LLMs.

Figure 8: Performance visualization on 25 domains.

Model	Release Time	Source
Closed-source Models
GPT-4o-mini	2024-07-18	https://openai.com/index/gpt-4o-mini/
GPT-5-mini	2025-08-07	https://openai.com/index/gpt-5-mini/
GPT-5.6 Sol	2026-07-09	https://openai.com/index/gpt-5-6/
Gemini-2.5-Flash	2025-06-17	https://deepmind.google/models/gemini/flash/
Gemini-3.1-Pro-Preview	2026-02-19	https://deepmind.google/models/gemini/pro/

Appendix

Table 12: The release time and model source of LLMs used in our benchmark.

Model	Backend	Reasoning Mode	Temperature
GPT-4o-mini	OpenAI-compatible	non-reasoning	0
GPT-5-mini	OpenAI-compatible	Disabled	0
GPT-5.6-sol	OpenAI-compatible	Disabled	0
Gemini-2.5-Flash	Gemini Native API	Disabled	0
Gemini-3.1-Pro-Preview	Gemini Native API	Disabled	0
Claude-Haiku-4.5	Anthropic Native API	Disabled	0

Appendix

Table 13: Model Backend and Inference Configuration

Name	Model License	Code License
GPT-4o-mini	Proprietary	Proprietary
GPT-5-mini	Proprietary	Proprietary
GPT-5.6 Sol	Proprietary	Proprietary
Gemini-2.5-Flash	Proprietary	Proprietary
Gemini-3.1-Pro-Preview	Proprietary	Proprietary
Claude-Haiku-4.5	Proprietary	Proprietary

Appendix

Table 14: Summary of licenses of models evaluated in our benchmark.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 190 papers

Jun 19, 2026

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user's current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user's memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available: https://github.com/galaxyChen/AdaMem

Xingyu Chen, Rui Wang, Zhaopeng Tu +1

Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management Personal AI Agents Memory Management for LLM Agents

Ask PDF

Apr 21, 2026

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.

Md Nayem Uddin, Kumar Shubham, Eduardo Blanco +2

Arizona State University · Genies · University of Arizona

Long-Horizon Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks Long-Term Conversational Memory Agent Memory Management

Ask PDF

Sep 21, 2026

DolphinBench: Mapping the Pareto Frontier of Agent Memory

Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.

Soumil Rathi, Deshraj Yadav, Taranjeet Singh

Mem0

Agent Memory Persistent Memory for Language Models Long-Horizon Agent Evaluation Memory Agent Benchmarks

Ask PDF

Abstract

Agent Memory

Explore similar work

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

DolphinBench: Mapping the Pareto Frontier of Agent Memory