Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agents

Jiajun Chen; Xinyue Chen; Yicheng Jiang; Hua Shen

cs.AIOct 13, 2026

Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agents

Authors: Jiajun Chen, Xinyue Chen, Yicheng Jiang, Hua Shen

Organizations: Carnegie Mellon University · New York University · NYU Shanghai

Abstract

Personal-agent memory has largely focused on learning what users know, prefer, and want. However, knowing the user does not necessarily mean knowing what the user and the agent have established together. An assistant's proposal should not become an authorized instruction merely because it is remembered. We introduce Grounding Awareness, the ability to distinguish accepted, unacknowledged, revised, and scoped arrangements when selecting actions. We develop a controlled benchmark to test whether agents can tell the difference between proposals users have accepted and those they have not, and whether they apply accepted arrangements only where appropriate. Across multiple agents and memory systems, we find that strong memory performance does not ensure Grounding Awareness. More surprisingly, providing more complete user information can increase execution of unacknowledged proposals, while persistent memory can cause agents to treat these proposals as if they were user instructions. We trace these failures through memory writing, retrieval, and action selection. Finally, we introduce Grounding Layer, a lightweight, training-free extension that explicitly tracks grounding status and checks consequential actions, substantially improving agent behavior without modifying models or memory systems.

Figures & tables

Figure 1: Knowing a proposal is not knowing it was established. An assistant proposes the same arrangement in two cases, but the user accepts it only in Case A. A memory pipeline extracts and stores the arrangement without preserving this distinction, producing the same persistent instruction in both cases. When a matching request arrives, direct execution is appropriate only in Case A; in Case B, the assistant should seek confirmation.

Subtype	Count	Example Task
What was said
Facts and events	20	Recalling facts or events across sessions ( Maharana et al., 2024 ) .
Preferences	6	Following a preference stated earlier ( Zhao et al., 2025 ) .
Instructions	3	Applying earlier instructions to later tasks ( Rakotonirina et al., 2025 ) .
Temporal updates	11	Answering with the latest value after an update ( Wu et al., 2025 ) .
Information provenance

Table 1: Taxonomy of 34 benchmarks by evaluation information. Counts overlap across categories (total: 50). No prior benchmark evaluates cross-session grounding (shaded). Detailed Analysis is in Appendix A .

Figure 2: Performance of the same Qwen-based memory pipelines on existing evolving-user benchmarks (left) and our two Grounding Awareness dimensions (right).

Sensitivity $\uparrow$
Memory	GPT	Qwen	Gemini	DS
Stateless	0.04	0.10	0.00	0.00
Raw History	0.09	0.14	0.03	0.07
Mem0	0.09	0.13	0.03	0.07
A-Mem	0.03	0.09	0.00	0.10
MemOS	0.09	0.11	0.03	0.23

Table 2: Grounding Awareness across four agent backbones and memory conditions. Sensitivity and Contextual Selectivity use matched G/I and S-in/S-out pairs, respectively. Red and Blue boxes highlight the highest Sensitivity and Contextual Selectivity within each backbone, respectively. DS denotes DeepSeek.

Figure 3: Effects of adding proposal content, grounding status, and action policy on Sensitivity and Contextual Selectivity across four agent backbones. Grounding status generally improves both dimensions, whereas additional action guidance yields mixed effects.

Figure 4: Failure attribution across memory systems. Each row shows the distribution of incorrect trajectories across writing, retrieval, and action-selection failures. Action errors are distinguished by whether grounding evidence was recoverable.

Input	No Reason.	Reason.	$\boldsymbol{\Delta}$
Raw History	0.06	0.16	+0.10
Content Oracle	0.04	0.07	+0.03
Status Oracle	0.43	0.73	+0.30

Table 3: Effect of DeepSeek reasoning on Sensitivity under different grounding information. All scores use the same evaluator.

Qwen
Agent	Sens.	Sel.	FCG $\boldsymbol{\Delta}$	Fact
Hermes	0.21	0.23	+0.14	0.97
OpenJarvis	0.13	0.15	+0.11	0.98
DeepSeek
Hermes	0.17	0.07	+0.06	0.86
OpenJarvis	0.06	0.00	+0.04	1.00

Table 4: Grounding Awareness in full personal agents. FCG

\Delta

measures the increase in false common ground with memory enabled. Fact denotes the fraction of stored unacknowledged proposals recorded as confirmed information or user facts.

	Sens. $\uparrow$		UE $\downarrow$
Agent	Base	GL	Base	GL
DeepSeek	0.17	0.36	0.30	0.09
Qwen	0.21	0.51	0.44	0.08

Table 5: Grounding Layer with LangMem. Unauthorized execution (UE) is measured at the action level. Full results appear in Appendix H.3 .

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Memory	GPT	Qwen	Gemini	DS
Raw History	+0.23	+0.44	+0.41	+0.33
Mem0	+0.29	+0.19	+0.29	+0.09
A-Mem	+0.34	+0.34	+0.36	+0.17
MemOS	+0.38	+0.38	+0.36	+0.36
LangMem	+0.24	+0.14	+0.26	+0.29
Content Oracle	+0.44	+0.51	+0.44	+0.54

Appendix

Table 6: Exposure effect: difference in execution probability between ignored (I) and absent (N) proposals. Positive values denote greater execution after an unacknowledged proposal.

Memory	Model	Base	+Content	+Status
Mem0	Qwen	0.26	0.53	0.29
Mem0	Gemini	0.74	0.87	0.80
A-Mem	Qwen	0.36	0.52	0.31
A-Mem	Gemini	0.75	0.84	0.81
MemOS	Qwen	0.34	0.46	0.30
MemOS	Gemini	0.84	0.85	0.83

Appendix

Table 7: False common ground under content and grounding-status supplementation (lower is better). These cell-level results cover Qwen and Gemini.

Memory	Recoverable
	Stored	Retrieved
Full History	0.81	0.81
Mem0	0.24	0.14
A-Mem	0.94	0.00
MemOS	0.63	0.56
LangMem	0.14	0.14

Appendix

Table 8: Recoverability of grounding distinctions in stored and retrieved information. Results are measured on matched G/I pairs.

Comparison	$\boldsymbol{\Delta}$	95% CI
Status–Content (Qwen)	+0.25	[0.13, 0.39]
Status–Content (DS)	+0.34	[0.21, 0.47]

Appendix

Table 9: Sensitivity gains from adding grounding status to content-only oracle memories. Confidence intervals use paired bootstrap resampling.

Memory	Revision Accuracy $\uparrow$
Full History	0.736
Mem0	0.395
A-Mem	0.433
MemOS	0.523
LangMem	0.591

Appendix

Table 10: Appropriate-action accuracy on revision (R) instances, averaged across four agent backbones (640 evaluations per memory condition).

Figure 5: Benchmark construction. Blue stages use LLM generation and automatic verification; amber stages involve human review; white stages use deterministic checks. Drafting agents iteratively revise scenarios against a validator before human review and history assembly.

Scenario type	Low	High	Total
Preference	45	35	80
Delegation	18	22	40
Convention	27	13	40
Total	90	70	160

Appendix

Table 11: Test-split composition by scenario type and action stakes. High-stakes events involve costly or irreversible actions.

Dataset property	Count / value
Test personas	10
Scenarios per persona	16
Conditions per scenario	5
Histories (persona $\times$ version)	50
Sessions per history	18
Base test instances	800

Appendix

Table 12: Benchmark scale. The full evaluation comprises 800 base instances and 274 additional in-scope probes per setting.

Study	Measure	Agree.	$\boldsymbol{\kappa}$
V0	Action gold	0.88	–
V1	Response labels	0.94	0.91
V2b	Content	0.82	0.71
	Current value	0.86	0.77
	Status	0.88	0.80
	Scope	0.88	0.80

Appendix

Table 13: Human validation of action gold (V0), dialogue manipulations (V1), memory auditing (V2b), and behavioral scoring (V2c). V2b compares human annotations with GPT-5-mini; GPT and DS denote GPT-5-mini and DeepSeek-Flash.

Condition	Off	On	$\Delta$
Raw History	0.06	0.16	+0.10
Mem0	0.07	0.10	+0.03
A-Mem	0.13	0.04	-0.09
MemOS	0.21	0.13	-0.08
LangMem	0.17	0.23	+0.06
Content Oracle	0.04	0.07	+0.03

Appendix

Table 14: DeepSeek reasoning ablation on Sensitivity. Off and On indicate reasoning disabled and enabled.

\Delta

is the difference between On and Off.

Backbone	Configuration	Sens.	Sel.	Overgen.	Overcons.	$\boldsymbol{\Delta\mathrm{Exec}_{I-N}}$
Qwen	Hermes full	0.21	0.23	0.28	0.37	+0.24
	Hermes no memory	0.09	0.10	0.17	0.53	-0.07
	Hermes no search	0.29	0.33	0.27	0.30	+0.26
	OpenJarvis memory	0.13	0.15	0.23	0.48	+0.03
	OpenJarvis no memory	0.06	0.10	0.10	0.67	-0.01
DeepSeek	Hermes full	0.17	0.07	0.07	0.85	-0.03

Appendix

Table 15: Full-agent results. Sensitivity uses G/I pairs; Contextual Selectivity uses S-in/S-out pairs. Overgen. and Overcons. denote overgeneralization and over-conservatism.

\Delta\mathrm{Exec}_{I-N}

measures proposal exposure.

Backbone	Agent	$\boldsymbol{\Delta\mathrm{FCG}}$	$\boldsymbol{\Delta\mathrm{RC}}$
Qwen	Hermes	+0.14	-0.24
Qwen	OpenJarvis	+0.11	-0.12
DeepSeek	Hermes	+0.06	-0.23
DeepSeek	OpenJarvis	+0.04	-0.06

Appendix

Table 16: Paired effect of enabling persistent memory on false common ground (FCG) and redundant confirmation (RC). Positive

\Delta\mathrm{FCG}

indicates more execution of unacknowledged proposals.

Backbone	Agent	G kept	I kept	I $\rightarrow$ Fact	R stale	R updated	S kept	S scope kept
Qwen	Hermes	0.89	0.65	0.97	0.27	0.69	0.79	0.73
Qwen	OpenJarvis	0.41	0.41	0.98	0.00	0.78	0.41	0.11
DeepSeek	Hermes	0.62	0.22	0.86	0.47	0.38	0.32	0.59
DeepSeek	OpenJarvis	0.28	0.29	1.00	0.01	0.79	0.31	0.04

Appendix

Table 17: Audit of native persistent memory. I

\rightarrow

Fact is measured among stored unacknowledged proposals; S scope kept is measured among stored scoped proposals. Other columns report retention or updating rates for the indicated conditions.

Backbone	Hermes	Search	Skills	Memory
Qwen	Full	0.20	0.64	0.15
Qwen	No memory	0.27	0.01	0.22
Qwen	No search	0.00	0.74	0.17
DeepSeek	Full	0.06	0.25	0.00
DeepSeek	No memory	0.10	0.00	0.01

Appendix

Table 18: Fraction of probes invoking each Hermes tool category. No search is evaluated only with Qwen on a subset of test instances.

Backbone	Configuration	Overall	N	G	I	R	S
Qwen	Hermes full	0.65	0.72	0.61	0.67	0.56	0.68
Qwen	Hermes no memory	0.53	0.73	0.21	0.68	0.34	0.66
Qwen	Hermes no search	0.64	0.66	0.62	0.60	0.56	0.72
Qwen	OpenJarvis memory	0.60	0.78	0.29	0.66	0.63	0.66
Qwen	OpenJarvis no memory	0.55	0.78	0.16	0.76	0.26	0.78
DeepSeek	Hermes full	0.67	0.94	0.29	0.86	0.34	0.89

Appendix

Table 19: Appropriate-action rates by grounding condition. N: no proposal; G: accepted proposal; I: unacknowledged proposal; R: revised proposal; S: scoped acceptance. Hermes no search covers a subset of the test set.

Backbone	Agent	Original	Permissive
Qwen	Hermes	0.21	0.37
Qwen	OpenJarvis	0.13	0.13
DeepSeek	Hermes	0.17	0.20
DeepSeek	OpenJarvis	0.06	0.06

Appendix

Table 20: Sensitivity under the original and permissive action-scoring criteria, with persistent memory enabled.

DeepSeek-Flash
Memory	Base	+GL
Raw History	0.06	0.56
Mem0	0.07	0.37
A-Mem	0.13	0.43
MemOS	0.21	0.44
LangMem	0.17	0.36

Appendix

Table 21: Sensitivity (paired G/I correctness). All values use the GL evaluation runs.

DeepSeek-Flash
	Accuracy $\uparrow$		UE $\downarrow$
Memory	Base	+GL	Base	+GL
Raw History	0.66	0.84	0.51	0.13
Mem0	0.65	0.85	0.22	0.09
A-Mem	0.62	0.85	0.38	0.11
MemOS	0.70	0.85	0.33	0.08

Appendix

Table 22: Appropriate-action accuracy and combined unauthorized execution across the 800 individual test trajectories per configuration. UE measures execution under unacknowledged or out-of-scope conditions.

	Sens. $\uparrow$		Sel. $\uparrow$
Agent	Dialogue	GL	Dialogue	GL
DeepSeek	0.27	0.44	0.37	0.57
Qwen	0.36	0.41	0.60	0.57

Appendix

Table 23: GL versus Dialogue Check with MemOS on discriminating pairs. Sensitivity and Selectivity measure paired correctness across acceptance and scope conditions, respectively.

Removed component	DeepSeek	Qwen
Both checks	$-0.07$	$-0.12$
Ledger gate	$-0.04$	$-0.03$
Provenance check	$-0.01$	$-0.08$

Appendix

Table 24: Action-accuracy changes relative to full GL on LangMem. The ledger-gate-only and provenance-check-only rows refer to removal of the named check.

User response	Count	Rate
Explicit acceptance	29	0.073
Scoped acceptance	8	0.020
Implicit acceptance	163	0.413
Revision	85	0.215
Rejection	14	0.035
No uptake	96	0.243

Appendix

Table 25: User responses to 395 verified assistant proposals in WildChat. Rates are computed over proposals rather than conversation turns.

Memory	Explicit	Implicit	No uptake	Scoped
Mem0	0.83	0.22	0.10	0.42
MemOS	0.57	0.05	0.02	0.21
LangMem	0.84	0.79	0.77	0.90
Macro Avg.	0.75	0.35	0.30	0.51

Appendix

Table 26: Fraction of retained memories encoding proposed arrangements as general user preferences. Macro Avg. is the unweighted mean across the three memory systems.

Agent memory	Explicit	Implicit	No uptake
Raw History	0.60	0.50	0.47
Mem0	0.20	0.10	0.10
MemOS	0.53	0.30	0.35
LangMem	0.40	0.33	0.30
Macro Avg.	0.43	0.31	0.31

Appendix

Table 27: Direct execution rates across acceptance conditions, using DeepSeek-Flash as the agent backbone. Macro Avg. is the unweighted mean across four memory conditions.

Evaluation setting	Accuracy
Immediate task only	0.74
Unambiguous cases only	0.81
Including ambiguous cases	0.66–0.74

Appendix

Table 28: Appropriate-action accuracy under different interpretations of implicit acceptance, pooled across four agent-memory conditions.

Personal AI Agents

Personal AI agents use persistent user context and digital tools to carry out personalized, proactive tasks across apps, devices, and services.

See all 64 papers

User as Code: Executable Memory for Personalized Agents
Jun 15, 2026Bojie LiPersistent Memory for Language Models Knowledge Representation
Pine AI
Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Jul 31, 2026Zhaoxin Feng, Jianfei Ma, Emmanuele ChersoniLLM Personalization LLM Agent Memory
The Hong Kong Polytechnic University
Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Oct 21, 2026Jialin Li, Zhenhao Chen, Hanjun Luo +1Memory Agent Benchmarks Personal AI Agents
New York University Abu Dhabi, Abu Dhabi, UAE

Jun 15, 2026J/K move · Enter open · S save

User as Code: Executable Memory for Personalized Agents

Bojie Li

Pine AI

Persistent Memory for Language Models Knowledge Representation User Modeling Long-Term Conversational Memory Agent Memory Management Personal AI Agents

Open paper AskPDF arXiv

A personalized AI agent needs a user memory: a persistent model of who the user is, built across many conversations and consulted on each new one. Today this memory is almost always stored as unstructured text, a knowledge graph, or a flat store of facts, and consulted by retrieval -- fetching the entries most similar to the current request. Such "bag-of-facts" memory recalls individual facts well, but because storing a fact and acting on it are separate steps, it struggles to resolve contradictions, aggregate over many records, or enforce rules. We argue that user memory should instead be executable. We introduce User as Code (UaC), a paradigm in which an agent's model of a user is a living software project: typed Python objects hold the user's state and ordinary Python functions encode the rules that govern it, so representing and reasoning about the user happen in one medium an interpreter can run. The enabling mechanism is a two-phase pipeline: an append-only log that never discards a fact, periodically checkpointed into typed code. This changes what memory can do. On standard long-term conversation benchmarks, UaC matches both a full-context upper bound and the strongest prior memory systems on recall (78.8% on LOCOMO). Its advantage emerges where representation matters most. On aggregate questions over a user's history -- "how many international trips did I take last year?" -- retrieval-based memory collapses (6-43%) while UaC stays near-perfect (99%), because the answer is a one-line computation over typed state rather than a search over text. And because its rules execute deterministically whenever the state changes, UaC can surface unsolicited, safety-critical alerts -- such as a newly prescribed drug that conflicts with an allergy recorded months earlier -- a capability query-driven memory cannot provide.

Abstract

Personal AI Agents

Explore similar work

User as Code: Executable Memory for Personalized Agents

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences