Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory

Masahiro Kaneko; Hiromi Arai; Timothy Baldwin

cs.CROct 20, 2026

Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory

Authors: Masahiro Kaneko, Hiromi Arai, Timothy Baldwin

Organizations: MBZUAI · Third Intelligence · RIKEN AIP

Abstract

Large language model (LLM) agents increasingly use persistent memory to personalize responses, and personal attributes in this memory pose a privacy risk if exposed. We ask how consulting memory during dialogue changes what users go on to say, and thus what the memory comes to reveal. In multi-session simulations, a simulated user with 24 personal attributes talks with a support agent that does or does not consult memory, and in both settings attributes are inferred from the full accumulated record. Across four support models, memory raises response quality but also empirical leakage, the share of attribute values correctly inferred from the record, by a margin that widens over sessions. The cause is a loop in ordinary, non-adversarial assistance. Memory-conditioned responses carry earlier values forward, often as presuppositions, and users affirm them or leave them uncorrected, so that cues and the agent's own guesses become recorded values. The loop reinforces itself, and removing the agent's mentions of an attribute lowers its later disclosure about twice as much as single-turn effects predict. Users cannot leave it on their own, since correcting a value discloses the attribute, declining to discuss it shifts later predictions toward sensitive values, and incorrect values left in place become entrenched. The loop is best weakened at the agent's response. Instructing the agent not to state or presuppose attributes brings leakage to about the no-memory level while keeping most of the quality gains, whereas filtering or anonymizing what users say or what is stored gives up much of them.

Figures & tables

Figure 1: An example of the loop from our simulations. The user’s mention of changing shifts (1) is recorded and presupposed by the agent in a later session (2), and the user’s affirmation with a new detail (3) is recorded again (4).

Panel of Figure 2 — Figure 2: Memory raises empirical leakage, and filtering lowers it in both settings. Dashed lines show the majority-category reference.

Figure 3: Memory raises helpfulness and, more strongly, coherence, and stricter filtering gives these gains up. Direct and Strict include memory.

Share of user turns with a value-level disclosure (%)
Measure	Memory	No memory	Difference
Session 1 (placebo)	14.3	13.9	0.4
Opening turn	21.2	20.5	0.7
Later turns	18.5	12.1	6.4 †
Share of responses (%)
Mention from an earlier session only	38.1	0.0	38.1 †

Table 1: Memory changes disclosure only after the agent responds, through values carried forward from earlier sessions. Unfiltered runs.

Measure	Group	Unfiltered	Direct
First value-level disclosures	Untouched	58	n/a
	Explicit removed	50	n/a
	All removed	37	n/a
First cue-level disclosures	Untouched	71	40
	Explicit removed	64	35
	All removed	49	26

Table 2: Removing the agent’s mentions reduces later disclosure beyond what single-turn effects predict. Counts are per 1,000 attributes over sessions 41 to 50.

Attribute	Value set	Refusal rate difference	Change
Health insurance	not covered	14 †	33 †
Citizenship	not a citizen	19 †	30 †
Disability	with a disability	12 †	21 †
Employment status	unemployed	9 †	15 †
Income	lowest two bands	7 †	11 †

Table 3: Users decline to discuss sensitive values more often, and a refusal shifts the next prediction toward them. Direct-filter memory runs, in percentage points, in set minus not in set ( Table 18 ).

Panel of Figure 4 — Figure 4: Only the response-side instruction lowers leakage without giving up most of the quality gains. Averages over the two open-weight support models ( Table 19 ).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	Inference candidate space	$\|\mathcal{V}^{\mathrm{inf}}_{a}\|$
Adapted from Staab et al. (2024)
Age ( AGEP )	5-year bands, highest open-ended	12
Sex ( SEX )	male, female	2
Location ( STATE , PUMA )	PUMA of residence	234
Place of birth ( POBP )	state if domestic, country if foreign	87
Income ( PINCP )	$10,000 bands, highest open-ended	16

Appendix

Table 4: The 24 attributes and their inference candidate spaces.

|\mathcal{V}^{\mathrm{inf}}_{a}|

excludes “undetermined”, and for location, place of birth, and occupation it counts values occurring among the 300 profiles.

Judges, agreement with adjudicated labels (%)
Measure	Items	Agreement
Direct judge	200	92
Inferable judge	200	88
Evaluator
Helpfulness, between annotators ( $\alpha$ )	300	0.69
Helpfulness, evaluator vs. annotators (Spearman)	300	0.67

Appendix

Table 5: Human validation of the LLM components.

Participant	Start	$G^{\mathrm{on}}$	$G^{\mathrm{off}}$	Correct at 30	Not recalled
P1	on	3	0	3	1
P2	off	3	1	4	1
P3	on	4	2	6	2
P4	off	3	0	3	0
P5	on	4	1	3	1
P6	off	4	1	4	1

Appendix

Table 6: Every participant had more attributes first inferred while memory was on.

	Mentions	Affirmation	Correction	Refusal	No correction
Participants, memory on	214	11	5	4	80
Participants, memory off	54	10	6	4	80
Simulated, memory on	162	2	2	7	89

Appendix

Table 7: Participants affirmed mentions more often than simulated users despite the instruction. Reactions in percent.

		Luna	Terra	Qwen3.8-27B	V4-Flash
Turns per session	Memory	4.6	4.3	5.2	4.9
Turns per session	No memory	5.1	4.9	5.7	5.5
User words per session	Memory	212	198	236	224
User words per session	No memory	241	229	268	257

Appendix

Table 8: Memory sessions are shorter. Dialogue volume per session without filtering, averaged over profiles and the 80 sessions. Words are whitespace-delimited tokens.

		Memory			No memory
Model	Metric	Unfilt.	Direct	Strict	Unfilt.	Direct	Strict
Luna	Abstention	5	9	13	7	13	15
	Precision	59	56	54	55	52	51
	Leakage	56	51	47	51	45	43
Terra	Abstention	4	7	11	6	9	13
	Precision	68	62	57	61	57	53

Appendix

Table 9: Memory’s gain in leakage is mostly in precision, not in fewer abstentions. All values in percent.

	Support model (self)			Gemini 3.1 Pro			Kimi K3
Support model	Memory	No memory	Difference	Memory	No memory	Difference	Memory	No memory	Difference
Luna	56	51	+5 †	64	57	+7 †	62	55	+7 †
Terra	65	57	+8 †	70	61	+9 †	69	62	+7 †
Qwen3.8-27B	56	50	+6 †	64	56	+8 †	62	53	+9 †
V4-Flash	60	55	+5 †	67	60	+7 †	66	59	+7 †
Mean	59.3	53.3	+6.0 †	66.3	58.5	+7.8 †	64.8	57.3	+7.5 †

Appendix

Table 10: A stronger inference model recovers more attributes in both settings, and the gap between memory and no memory remains. Leakage (%) at the final checkpoint of unfiltered runs.

Support model	Condition	Self	Gemini 3.1 Pro	Kimi K3
Qwen3.8-27B	No memory	50 †	56 †	53 †
	Memory	56	64	62
	Response-side instruction	51 †	56 †	58
V4-Flash	No memory	55 †	60 †	59 †
	Memory	60	67	66
	Response-side instruction	55 †	60 †	60 †

Appendix

Table 11: Against stronger inference models, the response-side instruction mostly keeps leakage (%) near the no-memory level. Daggers are relative to memory for the same inference model.

Panel of Figure 5 — Figure 5: Empirical leakage with Mistral Small 4 on the user side. Dashed lines show the majority-category reference.

		Qwen3.8-27B				V4-Flash
Format	Condition	Leakage	Help.	Coh.	Mentions	Leakage	Help.	Coh.	Mentions
Append-only	No memory	51 †	3.0 †	2.9 †	27 †	58 †	3.3 †	3.3 †	25 †
	Memory	62	4.2	4.4	49	66	4.5	4.7	46
	Response-side instruction	52 †	4.1 †	4.3 †	19 †	57 †	4.4 †	4.6 †	17 †
Update-based (Mem0)	No memory	48 †	3.0 †	2.9 †	27 †	53 †	3.3 †	3.3 †	25 †
	Memory	58	4.1	4.2	44	63	4.4	4.5	41

Appendix

Table 12: Memory raises leakage in every format, and the response-side instruction brings it to about the no-memory level. Leakage and mentions in percent. Daggers are relative to memory in the same format.

Panel of Figure 6 — Figure 6: Memory adds disclosures mainly as affirmations at the value level, and its mentions are mostly correct.

	Response version	Direct	Inferable
With mentions	Original	20.3	49.8
	Without memory	15.9 †	44.2 †
	All removed	16.8 †	45.3 †
All (reweighted)	Original	18.6	47.5
	Without memory	15.6 †	43.6 †
	All removed	16.7 †	45.2 †

Appendix

Table 13: Removing only the mentions removes most of a memory-conditioned response’s effect on the next user turn. Percentage of next user turns with a disclosure, with daggers relative to the original.

		Unfiltered		Direct filter
Link	Form	Memory	No memory	Memory	No memory
User turn	Value	6.0	4.3	0.3	0.2
User turn	Range or cue	6.4	6.2	7.4	6.9
Record	Value	8	5	5	4
	Cue	9	10	10	9
	Uncorrected agent value	2	1	4	2

Appendix

Table 14: Every link occurs in explicit and implicit forms, and under the direct filter the implicit forms dominate. Units are given in Appendix J.3 .

Panel of Figure 7 — Figure 7: Without filtering, cycles often close through explicit links, whereas under the direct filter they mostly close through implicit ones. Closed cycles per profile in parentheses.

Measure	Group or window	Unfiltered	Direct filter
References or questions	Untouched	1.2	0.7
	Explicit removed	0.0	0.0
	All removed	0.0	0.0
Presuppositions	Untouched	0.9	0.8
	Explicit removed	0.8	0.8
	All removed	0.0	0.0

Appendix

Table 15: The edits remove what they target, and the excess over the single-turn prediction grows over the ten sessions. Predicted reductions and spillover are per 1,000 attributes, and the other counts are per attribute.

	Response version	First val.	First cue	Inferable	Uncorrected	Helpfulness
Unfiltered	Original	n/a	n/a	49.8	65	n/a
	Explicit removed	$-$ 4.4 †	$-$ 4.8 †	47.2 †	71 †	0.00
	Presuppositions removed	$-$ 4.1 †	$-$ 5.2 †	47.0 †	60 †	$-$ 0.01
	All removed	$-$ 5.1 †	$-$ 5.9 †	45.3 †	n/a	$-$ 0.02
Direct filter	Original	n/a	n/a	43.1	97	n/a
	Explicit removed	$-$ 0.2	$-$ 5.4 †	41.3 †	98	$-$ 0.01

Appendix

Table 16: Single-turn effects of each removal, which give the predictions of Table 2 . First val. and first cue are per 1,000 removed pairs, and inferable and uncorrected are percentages.

	Unfiltered	Direct filter	Difference
Incorrect value carried forward	3	4	1 †
corrected	31	3	$-$ 28 †
refused	1	9	8 †
of which recorded uncorrected	n/a	71	n/a
of which predicted at the next checkpoint	n/a	55	n/a
neither corrected nor refused	68	88	20 †

Appendix

Table 17: Incorrect values carried forward are often recorded and predicted again, more so under the direct filter. Percentages, with rows defined in Appendix K.2 .

Attribute	Value set	In set	Not in set	Difference	No refusal	Refusal	No refusal	Refusal	Change
		Refusal rate			Correct, in set		Correct, not in set
Health insurance	not covered	21	7	14 †	27	41	90	71	33 †
Citizenship	not a citizen	24	5	19 †	23	39	94	80	30 †
Disability	with a disability	18	6	12 †	29	38	86	74	21 †
Employment status	unemployed	15	6	9 †	24	31	91	83	15 †
Income	lowest two bands	12	5	7 †	33	37	75	68	11 †

Appendix

Table 18: Users decline to discuss sensitive values more often, and a refusal shifts the next prediction toward those values. Direct-filter memory runs, in percent. Change is the shift in correct predictions after a refusal, in set minus not in set.

User side
Condition	Leakage	Helpfulness	Coherence	Mentions
No memory	53	3.1	3.2	31
Memory	58	4.2	4.5	54
Direct filter (idealized)	53 †	3.7 †	3.9 †	44 †
Strict filter (idealized)	47 †	3.0 †	3.2 †	32 †
User-side anonymization	49 †	3.6 †	3.3 †	35 †

Appendix

Table 19: Only the response-side instruction lowers leakage without giving up most of the quality gains. Averages over the two open-weight support models, with leakage and mentions in percent and daggers relative to memory.

Condition	Leakage	Help.	Coh.	Mentions	Leakage	Help.	Coh.	Mentions
	Qwen3.8-27B				V4-Flash
No memory	50	2.9	3.0	30	55	3.2	3.3	32
Memory	56	4.1	4.5	52	60	4.3	4.5	56
Direct filter (idealized)	51	3.5	3.7	42	54	3.9	4.0	46
Strict filter (idealized)	45	2.6	2.8	31	48	3.4	3.5	33
User-side anonymization	47	3.4	3.2	34	51	3.8	3.4	36

Appendix

Table 20: The ordering of the mitigations holds for both support models. Columns as in Table 19 .

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 331 papers

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents
Sep 28, 2026Fengzhou Sun, Yuan Zhang, Xintong Yu +1LLM Agent Memory Privacy Leakage in Language Models
State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China
Mitigating Over-Personalization in LLMs via Structured Memory
Aug 8, 2026Hakeem Hannoon, Andrew Zhao, Mihir Narayan +2LLM Sycophancy Persistent Memory for Language Models
University of Saskatchewan · University of California–Irvine · University of Wisconsin–Madison +2
Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Jul 31, 2026Zhaoxin Feng, Jianfei Ma, Emmanuele ChersoniLLM Personalization LLM Agent Memory
The Hong Kong Polytechnic University

Sep 28, 2026J/K move · Enter open · S save

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan

State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

LLM Agent Memory Privacy Leakage in Language Models Long-Term Conversational Memory

Open paper AskPDF arXiv

Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0% privacy classification accuracy, improves disclosure-permission judgment from 22% to 68%, and reduces privacy leakage by 75.6%, while maintaining retrieval performance and cross-benchmark generalization.

Abstract

LLM Agent Security

Explore similar work

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Mitigating Over-Personalization in LLMs via Structured Memory

Know It, Act on It: Investigating Memory Utilization in LLM Personalization