Personalized LLM agents must act on a user's current preferences, which change across interactions and are rarely stated in full. Existing memory systems store and retrieve content from the interaction history and leave the agent to work out which preferences still hold and where they apply. Such content does not indicate whether a preference is still current or where it applies, so agents often act on outdated or misapplied preferences. We instead treat memory as the assistant's belief about the user. Following Bourdieu's account of habitus as durable but revisable dispositions, we model the user as a latent preference state that every interaction, including the current request, both reveals and updates. The belief summarizes what the history supports about this state, and memory consists of the individual beliefs that bear on the agent's current decision, each with its evidence, scope, and status. We implement this view in HabitusMem, a trained writer that states these beliefs as text records for a frozen agent. Across two agent families, HabitusMem achieves the best results among memory systems on both PersonaMem-v2 and VitaBench 2.0. Controlled analyses show that learned request conditioning drives the answering gain, while auxiliary training tasks improve tool use.
Figures & tables
Figure 1: Content-based memory method versus belief memory on a personalized order. The user has repeatedly rejected Penfolds wine and now asks for a bottle of wine without naming a brand. Content-based memory methods pass on fragments such as past purchases and a negative review, and agents using them order the wrong wine or none at all. HabitusMem infers that the user prefers mid-priced layered reds and rejects Penfolds, records the evidence, scope, and status of each belief, and leads the agent to the same order as the ground-truth profile.
Figure 2: Overview of HabitusMem. The writer takes the retained raw interactions and the current user query as input and infers belief records, each with its evidence, scope, and status. It then projects only the beliefs that bear on the user query to the agent, which answers or acts on the user query. Here, the bland diet and the late-shift hospital address shape the dinner order, whereas the partner-specific vegetarian preference and the one-off oat-milk instruction are kept but not projected. A single writer learns these steps from four tasks: direct writing, state update, request projection, and action projection.
Memory
Know-
Pro-
Pro-
Crea-
Trans-
Personal-
Trouble
Chat
Social
Overall
ledge
Email
Writing
tive
lation
Email
Qwen3.5-397B-A17B
Base LLM
0.644
0.408
0.381
0.420
0.438
0.433
0.393
0.396
0.386
0.446
Full context
0.519
0.594
0.573
0.543
0.624
0.585
0.599
0.548
0.565
0.569
RAG
0.436
0.481
0.453
0.462
0.486
0.470
0.437
0.450
0.474
0.460
Rewrite
0.519
0.503
0.466
0.466
0.479
0.469
0.509
0.440
0.485
0.484
Table 1: Personalized QA accuracy on PersonaMem-v2 by task scenario. Bold and underlined values mark the best and second-best compact memory systems in each column.
Memory
Qwen3.5-397B-A17B Agent
DeepSeek-V4-Flash-0731 Agent
Avg@4
Pass@4
Pass4
Avg@4
Pass@4
Pass4
Base LLM
0.0895
0.2010
0.0182
0.0833
0.1946
0.0130
Full context
0.3382
0.5888
0.1297
0.3119
0.5409
0.1102
Ground-truth profile
0.3330
0.5888
0.1193
0.3839
0.6316
0.1569
RAG
0.2857
0.5391
0.0864
0.2251
0.4705
0.0562
Rewrite
0.3139
0.6109
0.0700
0.3123
0.5863
0.0869
Table 2: Personalized tool-use results on VitaBench 2.0. Bold and underlined values mark the best and second-best compact memory systems in each column.
Figure 3: Component analysis on PersonaMem-v2. (a–b) Inference interventions with a fixed writer and leave-one-out retraining. (c–d) Training lineage, matched controls, and reference conditions. Bars are grouped by agent, the rightmost bar in each group is HabitusMem, and hatched bars are reference conditions. Tables 4 and 5 list exact values.
Qwen3.5-397B-A17B Agent
DeepSeek-V4-Flash-0731 Agent
Writer
Avg@4
Pass@4
Pass4
Avg@4
Pass@4
Pass4
Base, static
0.2510
0.4620
0.0840
0.2700
0.5040
0.0840
Base + request
0.3410
0.5380
0.1620
0.2897
0.5320
0.0866
Base + unrelated
0.1940
0.3700
0.0610
0.2100
0.4250
0.0600
Trained, static
0.2640
0.4530
0.1040
0.2480
0.4660
0.0890
Parent writer
0.3012
0.5214
0.1245
0.3243
0.5499
0.1310
Table 3: Training lineage and matched-control results on VitaBench 2.0.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen3.5-397B-A17B
DeepSeek-V4-Flash-0731
Setting
Accuracy
Generation
Accuracy
Generation
HabitusMem (default)
0.5372
0.3257
0.4842
0.3824
Inference intervention (writer fixed)
Without request conditioning
0.4004
0.1345
0.3394
0.1995
With an unrelated request
0.4019
0.1247
0.3376
0.1859
Training component removed (retrained)
Appendix
Table 4: Request and training ablations on PersonaMem-v2.
Qwen3.5-397B-A17B
DeepSeek-V4-Flash-0731
Writer
Accuracy
Generation
Accuracy
Generation
Training lineage
Base, static
0.383
0.112
0.329
0.181
Base + request
0.393
0.146
0.345
0.211
Base + unrelated
0.401
0.133
0.347
0.189
Trained, static
0.406
0.135
0.335
0.200
Appendix
Table 5: Training lineage and matched answering controls on PersonaMem-v2.
Figure 4: Sensitivity to training-mixture weights on the full VitaBench 2.0 benchmark. Weights follow direct writing, state update, request projection, and action projection. Scores are percentages.
Figure 5: Inference sensitivity on PersonaMem-v2. (a–b) Personalized QA accuracy and (c–d) generation score. Each cluster varies one setting while the others stay at their defaults, and darker bars mark the default configuration.
Factor
Condition
Full, Qwen
Full, DeepSeek
Plain, Qwen
Attribution
Self
10/10
10/10
10/10
Gift / prior interest
0/10
1/10
0/10
Scope
Unconditional, partner
10/10
10/10
10/10
Unconditional, solo
10/10
10/10
10/10
Conditional, partner
10/10
10/10
10/10
Conditional, solo
10/10
2/10
7/10
Appendix
Table 6: All controlled probe results. Entries are correct agent answers out of ten. The gift cell also includes an earlier self-statement of interest; it does not isolate writer attribution. Manual inspection finds the gift relationship in all ten memories from both writers.
Figure 6: A PersonaMem-v2 example in which the agent answers a recommendation question after the user has retracted a preference.
Figure 7: A VitaBench 2.0 example in which the agent completes a paid delivery order from an underspecified request.
Large language model (LLM) agents increasingly use external memory systems to support personalization by drawing on long and evolving interaction histories, in which user preferences may be distributed across time, change with context, and conflict with earlier evidence. However, existing systems face three limitations: fixed-turn, fixed-token, or session-based boundaries can mix unrelated dialogue or split an event from its causes, decisions, and outcomes; storing multiple pieces of user information from the same interaction as a single memory binds together items that serve different functions and should be independently retrievable; and treating the current task as a single top-k retrieval query can return fragments that are individually relevant but fail to jointly capture preference evolution, temporal validity, and contextual applicability. We introduce QUMem, a structured memory framework for query-conditioned user-state inference. QUMem first segments interaction histories into variable-length episodes according to semantic continuity, then decomposes each episode into independently retrievable factual, preference, and transferable insight memories while preserving temporal positions and source evidence. At inference time, three sequential agents identify task-specific information needs, plan multi-query retrieval over the typed memory stores, and jointly infer a temporally and contextually valid user state for downstream response generation. QUMem achieves state-of-the-art performance on both PersonaMem and KnowU-Bench, demonstrating the effectiveness of query-conditioned user-state inference for long-term personalization.
Heng Wang, Yifei Li, Lingling Zhang +4
School of Computer Science and Technology, Xi’an Jiaotong University · MOE KLNN Lab, Xi’an Jiaotong University
As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.
In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.
Junming Zhang, Rubin Wei, Xu Cheng +7
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University