HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

Junming Zhang; Rubin Wei; Xu Cheng; Hao Sun; Jiarui Wang; Ruoxi Cheng; Xiquan Wang; Peisong Wu

cs.AIOct 17, 2026

HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

Authors: Junming Zhang, Rubin Wei, Xu Cheng, Hao Sun, Jiarui Wang, Ruoxi Cheng, Xiquan Wang, Peisong Wu,

Organizations: LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University

Abstract

In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.

Figures & tables

Benchmark	Long Hist.	Weak Evid.	Action	Context Gate	Exception	Temp. Drift	Withhold
LoCoMo [ Maharana et al., 2024 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\circ$	$\boldsymbol{\times}$
LongMemEval [ Wu et al., 2024 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$
MemoryAgentBench [ Hu et al., 2026a ]	$\boldsymbol{\checkmark}$	$\circ$	$\circ$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$
MemoryArena [ He et al., 2026 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\circ$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$
Mem2ActBench [ Shen et al., 2026 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\circ$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$
PERMA [ Liu et al., 2026 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\circ$	$\circ$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$

Table 1: Capability coverage across representative long-term-memory and personalization benchmarks.

\boldsymbol{\checkmark}

\circ

, and

\boldsymbol{\times}

denote dedicated, partial, and absent benchmark-level coverage, respectively. Columns cover longitudinal history, weak-evidence induction, memory-guided action, contextual gating, exceptions, temporal drift, and withholding unsupported personalization.

Figure 1: HABIT-Bench framework. (A) contrasts explicit-memory retrieval with latent habit induction. (B) combines task-oriented prompt seeds with latent habit graphs to construct real-prompt-seeded synthetic lifelines. (C) evaluates memory systems through a shared answerer and diagnostic choice interface.

Domain	Users	Sessions	Probes	Source dataset
Food	30	3900	630	CookDial [ Jiang et al., 2023 ]
Finance	36	19440	1368	MultiDoGO [ Peskov et al., 2019 ]
Software	18	9720	680	MultiDoGO [ Peskov et al., 2019 ]
Travel	30	4092	650	Taskmaster-2 [ Byrne et al., 2019 ]

Table 2: Dataset statistics and source datasets. Sources provide dialogue and task seeds, while HABIT-Bench constructs the longitudinal histories and probe labels.

Method	Qwen3-235B-A22B					GLM-5.3-Flash
Method	Food	Finance	Software	Travel	Macro	Food	Finance	Software	Travel	Macro
Controls
No Memory	22.38	23.54	24.85	28.31	24.77	23.81	22.95	22.50	32.46	25.43
Full Memory	48.25	20.54	21.76	33.38	30.99	45.56	33.77	37.50	51.08	41.98
Retrieval baselines
Recency-5	24.13	22.51	24.26	31.23	25.53	18.10	23.46	23.53	39.69	26.20

Table 3: Main results with Qwen3-235B-A22B and GLM-5.3-Flash: exact-choice accuracy (%). Macro is the unweighted four-domain mean. Controls measure performance without memory or with compressed history. Retrieval baselines select sessions, while memory methods construct and maintain memory representations. Letta and MIRIX are shown separately as multi-store memory methods. Gray averages weight methods equally. All averages excludes the No Memory and Full Memory controls. TH-RAG denotes Temporal Hybrid RAG.

Method	Qwen3-8B				Qwen3-235B-A22B
Method	Food	Finance	Software	Travel	Food	Finance	Software	Travel
No Memory	20.79	23.32	27.21	26.46	22.38	23.54	24.85	28.31
Oracle Evidence	53.33	29.82	31.91	42.15	58.25	42.91	39.26	55.54
Oracle Habit State	92.54	93.71	94.56	98.31	98.41	98.54	99.26	99.23

Table 4: No Memory and private-information diagnostics: probe-level exact-choice accuracy (%) on the same four datasets. Oracles are non-deployable. Reader protocols differ across models, so this is not a scale-only ablation.

Figure 2: Qwen model-size comparison. Each group shows four-domain macro accuracy for five model settings. Full Memory is a control, and the dashed line marks 25% uniform random choice. Both memory-stage and answer models change, and the 235B setting uses a revised reader protocol, so the comparison does not isolate parameter scale.

Panel of Figure 3 — Figure 3: Sensitivity and cost analysis with Qwen3-8B. (a) Four-domain macro accuracy under retrieval top- $k$ and ingestion-chunk ablations, with 95% within-domain user-bootstrap intervals. (b) Cumulative build/query latency and LLM token usage on the same 32-user cohort. Build and query times are reported in hours and minutes, respectively. Token counts use a logarithmic scale.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Human-validation protocol. A and B independently choose an action and apply the quality rubric. C adjudicates answer or quality disagreements without gold access. Only after the merged record is frozen does the data manager compare it with gold and assign a release disposition. Content corrections receive new, versioned inputs and targeted blind review. The diagram describes the protocol, while the tables report the completed stages and coverage in each domain.

Criterion	Food	Finance	Software	Travel
Recorded quality judgments
Unique answerability	200/200	336/340	311/311	618/618
Evidence sufficiency	200/200	340/340	311/311	618/618
Scope correctness	200/200	336/340	311/311	618/618
Boundary / exception	200/200	340/340	311/311	495/495
Choice balance	200/200	340/340	311/311	618/618

Appendix

Table 5: Human-validation quality judgments across four domains. Counts give positive judgments over applicable reviewed items. Lower is better for the modification flag. Finance retains four initial Modify judgments for output-placement omissions corrected before final evaluation. Completed naturalization and content validation are included in the quality outcomes.

Domain	$N$	$n$	Agreement (%)	$\kappa$	A–gold	B–gold	Blanks A/B
Food	200	200	99.50	0.9933	200/200	199/200	0/0
Finance	340	336	100.00	1.0000	340/340	336/336	0/4
Software	311	309	99.68	0.9957	310/311	309/309	0/2
Travel	618	545	92.84	0.9044	515/577	501/564	41/54

Appendix

Table 6: Independent A/B answer agreement across four domains.

N

: jointly annotated items.

n

: jointly valid choices. A–gold and B–gold give matches over valid choices. Agreement and Cohen’s

\kappa

are measured before adjudication, and gold agreement is evaluated before label corrections.

Coverage	Probes
Distinct reviewed probes	618
Outside the human-review sample	32
Total probes	650
Content-revised probes passing targeted validation	79/79

Appendix

Table 7: Human-review coverage and targeted content validation in Travel. The coverage rows partition the dataset. The content-validation set is a subset of the reviewed probes.

Implementation class	Methods
official-code adapter	A-MEM, Letta, LightMem, Mem0, MemOS, MemRL, MIRIX
official-source adapted	SeCom
HABIT-native retrieval	Recency-5, Recency-10, BM25 RAG, Dense RAG, Temporal Hybrid RAG
evaluator control	Full History, Full Memory, No Memory
private diagnostic oracle	Oracle Evidence, Oracle Habit State

Appendix

Table 8: Implementation provenance of evaluated systems. Adapters share HABIT-Bench chronology, retrieval, and answer interfaces; they are not upstream-native end-to-end evaluations.

Method	F/S temporal	F/S provenance	F/S reference	Travel induction	Travel exception
Full Memory	22.8	20.2	21.2	29.6	39.4
A-MEM	24.8	23.0	23.1	35.2	36.4
Letta	23.5	22.0	23.1	37.0	45.5
LightMem	24.4	22.8	22.5	37.0	27.3
Mem0	25.1	24.0	22.8	35.2	42.4
MemOS	24.6	22.8	22.2	35.2	27.3

Appendix

Table 9: Qwen3-8B capability slices: probe-level exact-choice accuracy (%) in Finance, Software, and Travel. Finance/Software columns pool the two domains. The single-probe Travel boundary slice is omitted from the columns (no method answers it correctly).

Method	Food	Finance	Software	Travel
Controls
Full Memory	41.1 [37.6, 44.8]	20.8 [18.1, 23.6]	23.2 [20.1, 26.1]	28.1 [24.0, 32.7]
Retrieval baselines
Recency-5	21.3 [17.8, 25.1]	23.0 [20.5, 25.7]	25.9 [23.1, 28.7]	26.5 [22.1, 31.0]
Recency-10	21.9 [19.2, 24.8]	22.8 [20.2, 25.5]	26.3 [23.5, 29.3]	30.2 [25.5, 34.9]
BM25 RAG	42.7 [39.5, 45.7]	23.0 [20.7, 25.4]	24.4 [21.2, 27.6]	33.6 [29.0, 38.2]

Appendix

Table 10: Supplementary Qwen3-8B comparison. Values are user-macro accuracy (%) with 95% user-cluster intervals in gray. Full Memory is a control. Shaded averages weight methods equally within each group or across all methods and exclude controls. Means are computed before display rounding. Intervals are not averaged.

Method	Food	Finance	Software	Travel
No Memory	20.8 [17.1, 24.6]	23.3 [20.5, 26.3]	27.2 [23.9, 30.3]	26.9 [23.4, 30.4]
Oracle Evidence	53.3 [50.3, 56.5]	29.8 [27.4, 32.3]	31.9 [29.1, 34.7]	42.2 [37.8, 46.4]
Oracle Habit State	92.5 [90.6, 94.4]	93.7 [92.4, 94.9]	94.6 [92.2, 96.6]	98.4 [97.6, 99.2]

Appendix

Table 11: Qwen3-8B controls and private-information diagnostics under user-macro aggregation, with 95% user-cluster bootstrap intervals in gray. These intervals describe user resampling, not repeated-run variability.

Method	4B	8B	14B	32B	235B-A22B
Full Memory	26.29	28.08	29.40	29.88	30.99
A-MEM	29.41	30.34	30.63	31.46	33.06
Mem0	25.14	26.58	27.80	28.52	28.51
MemOS	27.40	29.49	30.01	29.90	29.60
MemRL	29.31	29.23	30.21	31.25	31.85
LightMem	26.33	27.16	28.14	25.04	27.01

Appendix

Table 12: Qwen model comparison using the four-domain mean of probe-level accuracy (%). All settings use the same four datasets. The 235B column uses Qwen3-235B-A22B and an updated reader protocol. Both memory-stage and answer models change, so this is not an answer-only or protocol-controlled scaling ablation. Full Memory is a control.

Figure 5: Memory-method accuracy with GLM-5.3-Flash across four domains. Bars show probe-level accuracy, and the dashed line marks 25% uniform random choice. Full results, including controls and retrieval baselines, appear in Table 3 .

Method	Macro Acc.	ER †	Hit@5	NDCG@5	MRR@5	CG@5	Joint A+E	Memory tokens
Full Memory	29.88	36.83	—	—	—	—	22.69	33,131.1
Source-attributed top-five memory methods
A-MEM	31.46	13.89	41.44	16.21	26.22	8.20	16.20	2,933.7
Mem0	28.52	11.17	39.30	13.63	24.58	7.00	12.92	107.2
MemOS	29.90	11.85	38.40	14.50	25.22	7.27	14.42	436.6
MemRL	31.25	12.67	37.77	14.99	24.68	5.49	14.36	1,735.1

Appendix

Table 13: Detailed Qwen3-32B results. Acc. is four-domain macro accuracy. Evidence metrics use the eligible probes for each statistic. All scores are percentages except memory tokens. ER: evidence recall; CG: Clean Grounded; Joint A+E: correct answer with an evidence hit. † Full Memory uses context-wide evidence coverage, not top-five retrieval. Dashes indicate inapplicable metrics.

Figure 6: Food case: recovering a default and its serving-context boundary from a CookDial-seeded lifeline [ Jiang et al., 2023 ] . History is condensed; the query and four choices are verbatim. The final panel is evaluator-only explanation.

Figure 7: Finance case: two workflows resolved at a historical target time in a MultiDoGO-seeded lifeline [ Peskov et al., 2019 ] . History, query, and choices are condensed, preserving option order and the gold label. The final panel is evaluator-only explanation.

Figure 8: Software case: recovering two user-confirmed workflows while rejecting an assistant-originated alternative in a MultiDoGO-seeded lifeline [ Peskov et al., 2019 ] . History, query, and choices are condensed. The final panel is evaluator-only explanation.

Figure 9: Travel case: combining a room-selection exception with a recreation default in a Taskmaster-2-seeded lifeline [ Byrne et al., 2019 ] . History, query, and choices are condensed. The final panel is evaluator-only explanation.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 368 papers

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2
AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Agent Memory Persistent Memory for Language Models
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent
DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users
Oct 2, 2026Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao +5Agent Memory Memory Agent Benchmarks
Nanyang Technological University · Stepfun · National University of Singapore +1

Jun 22, 2026J/K move · Enter open · S save

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

Wenya Xie, Shengming Zhou, Zelin Li, Pouya Parsa+8

University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University+2

Agent Memory LLM Agent Memory Persistent Memory for Language Models Long-Horizon Agent Evaluation Memory Agent Benchmarks User Modeling+1

Open paper AskPDF arXiv

LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/

Abstract

LLM Agent Memory

Explore similar work

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users