cs.CROct 20, 2026

Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory

Authors: Masahiro Kaneko, Hiromi Arai, Timothy Baldwin

Organizations: MBZUAI · Third Intelligence · RIKEN AIP

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents increasingly use persistent memory to personalize responses, and personal attributes in this memory pose a privacy risk if exposed. We ask how consulting memory during dialogue changes what users go on to say, and thus what the memory comes to reveal. In multi-session simulations, a simulated user with 24 personal attributes talks with a support agent that does or does not consult memory, and in both settings attributes are inferred from the full accumulated record. Across four support models, memory raises response quality but also empirical leakage, the share of attribute values correctly inferred from the record, by a margin that widens over sessions. The cause is a loop in ordinary, non-adversarial assistance. Memory-conditioned responses carry earlier values forward, often as presuppositions, and users affirm them or leave them uncorrected, so that cues and the agent's own guesses become recorded values. The loop reinforces itself, and removing the agent's mentions of an attribute lowers its later disclosure about twice as much as single-turn effects predict. Users cannot leave it on their own, since correcting a value discloses the attribute, declining to discuss it shifts later predictions toward sensitive values, and incorrect values left in place become entrenched. The loop is best weakened at the agent's response. Instructing the agent not to state or presuppose attributes brings leakage to about the no-memory level while keeping most of the quality gains, whereas filtering or anonymizing what users say or what is stored gives up much of them.

Figures & tables

Figure 1: An example of the loop from our simulations. The user’s mention of changing shifts (1) is recorded and presupposed by the agent in a later session (2), and the user’s affirmation with a new detail (3) is recorded again (4).
Figure 2: Memory raises empirical leakage, and filtering lowers it in both settings. Dashed lines show the majority-category reference.
Figure 3: Memory raises helpfulness and, more strongly, coherence, and stricter filtering gives these gains up. Direct and Strict include memory.
MeasureMemoryNo memoryDifference
Share of user turns with a value-level disclosure (%)
Session 1 (placebo)14.313.90.4
Opening turn21.220.50.7
Later turns18.512.16.4 †
Share of responses (%)
Mention from an earlier session only38.10.038.1 †
Table 1: Memory changes disclosure only after the agent responds, through values carried forward from earlier sessions. Unfiltered runs.
MeasureGroupUnfilteredDirect
First value-level disclosuresUntouched58n/a
Explicit removed50n/a
All removed37n/a
First cue-level disclosuresUntouched7140
Explicit removed6435
All removed4926
Table 2: Removing the agent’s mentions reduces later disclosure beyond what single-turn effects predict. Counts are per 1,000 attributes over sessions 41 to 50.
AttributeValue setRefusal rate differenceChange
Health insurancenot covered14 †33 †
Citizenshipnot a citizen19 †30 †
Disabilitywith a disability12 †21 †
Employment statusunemployed9 †15 †
Incomelowest two bands7 †11 †
Table 3: Users decline to discuss sensitive values more often, and a refusal shifts the next prediction toward them. Direct-filter memory runs, in percentage points, in set minus not in set ( Table 18 ).
Figure 4: Only the response-side instruction lowers leakage without giving up most of the quality gains. Averages over the two open-weight support models ( Table 19 ).
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

AttributeInference candidate space∣Vainf∣|\mathcal{V}^{\mathrm{inf}}_{a}|
Adapted from Staab et al. (2024)
Age ( AGEP )5-year bands, highest open-ended12
Sex ( SEX )male, female2
Location ( STATE , PUMA )PUMA of residence234
Place of birth ( POBP )state if domestic, country if foreign87
Income ( PINCP )$10,000 bands, highest open-ended16
Appendix
Table 4: The 24 attributes and their inference candidate spaces. ∣Vainf∣|\mathcal{V}^{\mathrm{inf}}_{a}| excludes “undetermined”, and for location, place of birth, and occupation it counts values occurring among the 300 profiles.
MeasureItemsAgreement
Judges, agreement with adjudicated labels (%)
Direct judge20092
Inferable judge20088
Evaluator
Helpfulness, between annotators ( α\alpha )3000.69
Helpfulness, evaluator vs. annotators (Spearman)3000.67
Appendix
Table 5: Human validation of the LLM components.
ParticipantStartGonG^{\mathrm{on}}GoffG^{\mathrm{off}}Correct at 30Not recalled
P1on3031
P2off3141
P3on4262
P4off3030
P5on4131
P6off4141
Appendix
Table 6: Every participant had more attributes first inferred while memory was on.
 MentionsAffirmationCorrectionRefusalNo correction
Participants, memory on214115480
Participants, memory off54106480
Simulated, memory on16222789
Appendix
Table 7: Participants affirmed mentions more often than simulated users despite the instruction. Reactions in percent.
  LunaTerraQwen3.8-27BV4-Flash
Turns per sessionMemory4.64.35.24.9
No memory5.14.95.75.5
User words per sessionMemory212198236224
No memory241229268257
Appendix
Table 8: Memory sessions are shorter. Dialogue volume per session without filtering, averaged over profiles and the 80 sessions. Words are whitespace-delimited tokens.
  MemoryNo memory
ModelMetricUnfilt.DirectStrictUnfilt.DirectStrict
LunaAbstention591371315
Precision595654555251
Leakage565147514543
TerraAbstention47116913
Precision686257615753
Appendix
Table 9: Memory’s gain in leakage is mostly in precision, not in fewer abstentions. All values in percent.
 Support model (self)Gemini 3.1 ProKimi K3
Support modelMemoryNo memoryDifferenceMemoryNo memoryDifferenceMemoryNo memoryDifference
Luna5651+5 †6457+7 †6255+7 †
Terra6557+8 †7061+9 †6962+7 †
Qwen3.8-27B5650+6 †6456+8 †6253+9 †
V4-Flash6055+5 †6760+7 †6659+7 †
Mean59.353.3+6.0 †66.358.5+7.8 †64.857.3+7.5 †
Appendix
Table 10: A stronger inference model recovers more attributes in both settings, and the gap between memory and no memory remains. Leakage (%) at the final checkpoint of unfiltered runs.
Support modelConditionSelfGemini 3.1 ProKimi K3
Qwen3.8-27BNo memory50 †56 †53 †
Memory566462
Response-side instruction51 †56 †58
V4-FlashNo memory55 †60 †59 †
Memory606766
Response-side instruction55 †60 †60 †
Appendix
Table 11: Against stronger inference models, the response-side instruction mostly keeps leakage (%) near the no-memory level. Daggers are relative to memory for the same inference model.
Figure 5: Empirical leakage with Mistral Small 4 on the user side. Dashed lines show the majority-category reference.
  Qwen3.8-27BV4-Flash
FormatConditionLeakageHelp.Coh.MentionsLeakageHelp.Coh.Mentions
Append-onlyNo memory51 †3.0 †2.9 †27 †58 †3.3 †3.3 †25 †
Memory624.24.449664.54.746
Response-side instruction52 †4.1 †4.3 †19 †57 †4.4 †4.6 †17 †
Update-based (Mem0)No memory48 †3.0 †2.9 †27 †53 †3.3 †3.3 †25 †
Memory584.14.244634.44.541
Appendix
Table 12: Memory raises leakage in every format, and the response-side instruction brings it to about the no-memory level. Leakage and mentions in percent. Daggers are relative to memory in the same format.
Figure 6: Memory adds disclosures mainly as affirmations at the value level, and its mentions are mostly correct.
 Response versionDirectInferable
With mentionsOriginal20.349.8
Without memory15.9 †44.2 †
All removed16.8 †45.3 †
All (reweighted)Original18.647.5
Without memory15.6 †43.6 †
All removed16.7 †45.2 †
Appendix
Table 13: Removing only the mentions removes most of a memory-conditioned response’s effect on the next user turn. Percentage of next user turns with a disclosure, with daggers relative to the original.
  UnfilteredDirect filter
LinkFormMemoryNo memoryMemoryNo memory
User turnValue6.04.30.30.2
Range or cue6.46.27.46.9
RecordValue8554
Cue910109
Uncorrected agent value2142
Appendix
Table 14: Every link occurs in explicit and implicit forms, and under the direct filter the implicit forms dominate. Units are given in Appendix J.3 .
Figure 7: Without filtering, cycles often close through explicit links, whereas under the direct filter they mostly close through implicit ones. Closed cycles per profile in parentheses.
MeasureGroup or windowUnfilteredDirect filter
References or questionsUntouched1.20.7
Explicit removed0.00.0
All removed0.00.0
PresuppositionsUntouched0.90.8
Explicit removed0.80.8
All removed0.00.0
Appendix
Table 15: The edits remove what they target, and the excess over the single-turn prediction grows over the ten sessions. Predicted reductions and spillover are per 1,000 attributes, and the other counts are per attribute.
 Response versionFirst val.First cueInferableUncorrectedHelpfulness
UnfilteredOriginaln/an/a49.865n/a
Explicit removed−- 4.4 †−- 4.8 †47.2 †71 †0.00
Presuppositions removed−- 4.1 †−- 5.2 †47.0 †60 †−- 0.01
All removed−- 5.1 †−- 5.9 †45.3 †n/a−- 0.02
Direct filterOriginaln/an/a43.197n/a
Explicit removed−- 0.2−- 5.4 †41.3 †98−- 0.01
Appendix
Table 16: Single-turn effects of each removal, which give the predictions of Table 2 . First val. and first cue are per 1,000 removed pairs, and inferable and uncorrected are percentages.
 UnfilteredDirect filterDifference
Incorrect value carried forward341 †
corrected313−- 28 †
refused198 †
of which recorded uncorrectedn/a71n/a
of which predicted at the next checkpointn/a55n/a
neither corrected nor refused688820 †
Appendix
Table 17: Incorrect values carried forward are often recorded and predicted again, more so under the direct filter. Percentages, with rows defined in Appendix K.2 .
  Refusal rateCorrect, in setCorrect, not in set 
AttributeValue setIn setNot in setDifferenceNo refusalRefusalNo refusalRefusalChange
Health insurancenot covered21714 †2741907133 †
Citizenshipnot a citizen24519 †2339948030 †
Disabilitywith a disability18612 †2938867421 †
Employment statusunemployed1569 †2431918315 †
Incomelowest two bands1257 †3337756811 †
Appendix
Table 18: Users decline to discuss sensitive values more often, and a refusal shifts the next prediction toward those values. Direct-filter memory runs, in percent. Change is the shift in correct predictions after a refusal, in set minus not in set.
ConditionLeakageHelpfulnessCoherenceMentions
No memory533.13.231
Memory584.24.554
User side
Direct filter (idealized)53 †3.7 †3.9 †44 †
Strict filter (idealized)47 †3.0 †3.2 †32 †
User-side anonymization49 †3.6 †3.3 †35 †
Appendix
Table 19: Only the response-side instruction lowers leakage without giving up most of the quality gains. Averages over the two open-weight support models, with leakage and mentions in percent and daggers relative to memory.
 Qwen3.8-27BV4-Flash
ConditionLeakageHelp.Coh.MentionsLeakageHelp.Coh.Mentions
No memory502.93.030553.23.332
Memory564.14.552604.34.556
Direct filter (idealized)513.53.742543.94.046
Strict filter (idealized)452.62.831483.43.533
User-side anonymization473.43.234513.83.436
Appendix
Table 20: The ordering of the mitigations holds for both support models. Columns as in Table 19 .
More in this topic

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 330 papers

Explore similar work

CardsList
Sep 28, 2026

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0% privacy classification accuracy, improves disclosure-permission judgment from 22% to 68%, and reduces privacy leakage by 75.6%, while maintaining retrieval performance and cross-benchmark generalization.

Fengzhou Sun, Yuan Zhang, Xintong Yu +1

State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

LLM Agent MemoryPrivacy Leakage in Language ModelsLong-Term Conversational Memory
AskPDF
Aug 8, 2026

Mitigating Over-Personalization in LLMs via Structured Memory

Conversational assistants increasingly rely on persistent long-term memory to personalize responses across sessions. However, when stored user information is reintroduced into the model context, it can also influence responses in inappropriate or unrelated settings. We study two such failure modes in memory-augmented LLMs: cross-domain leakage, where memories from one life domain affect responses in another, and memory-induced sycophancy, where stored user beliefs make models more likely to agree with the user rather than respond truthfully. We apply a simple inference-time modification to how memories are presented to the model, without changing the model or the memory contents. Across seven models on PersistBench, we compare the commonly used all-in context format, where memories are injected as an unstructured list, with structured formats that partition memories by domain. This simple modification consistently reduces cross-domain leakage while preserving utility, with our strongest method reducing leakage by 8.8%8.8\% on average relative to the baseline.

Hakeem Hannoon, Andrew Zhao, Mihir Narayan +2

University of Saskatchewan · University of California–Irvine · University of Wisconsin–Madison +2

LLM SycophancyPersistent Memory for Language ModelsLLM MemoryPrivacy Leakage in Language ModelsLong-Term Conversational Memory
AskPDF
Jul 31, 2026

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni

The Hong Kong Polytechnic University

LLM PersonalizationLLM Agent MemoryLLM Agent EvaluationUser Preference Modeling
AskPDF