Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

Soumya Sharma; Ayana Hussain; Heber Hwang Arcolezi; Ulrich Aïvodji; Spandana Gella; Adriana Romero-Soriano; Golnoosh Farnadi

cs.CLOct 7, 2026

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

Authors: Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi, Ulrich Aïvodji, Spandana Gella, Adriana Romero-Soriano, Golnoosh Farnadi

Organizations: Mila – Quebec AI Institute · McGill University · Simon Fraser University · ÉTS ·

Abstract

As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue has become critical. Privacy leakage in such settings requires reasoning about whether information flows respect the contextual norms under which they originated. We extend CIMemories to CIMEMORIES DIALOGUE, a benchmark of 4,410 two-agent dialogue instances grounded in Contextual Integrity theory. Using this benchmark, we systematically evaluate 7 open-weight models from the Qwen3 and Llama 3.1 families across conversation lengths of 1 and 6 turns under three prompting conditions. We find that task capability and privacy leakage are fundamentally coupled: more capable models are systematically leakier, with no model in our evaluation able to decouple the two. Thinking models consistently leak more than their non-thinking counterparts, a pattern we attribute to the extended reasoning surface created by chain-of-thought. Leakage compounds with conversation length, rising up to 3x from 1 to 6 turns. Our results suggest that the coupling between task capability and privacy leakage is not an artifact of misconfiguration but a structural feature of current agentic systems.

Figures & tables

Panel of Figure 1 — Figure 1: Example of Necessary and Private items shared in the conversation. This example is picked from Qwen3-4B-Thinking x Llama-3.1-8B PC 0 simulation.

Benchmark	CI	Mem	Dial	# Datapoints	# Turns	Users	Max Turns
ConfAIde Mireshghallah et al. (2024)	✓	✗	✗	$\sim$ 270 (T3) / 50 (T4) †	1	Single	-
PrivacyLens Shao et al. (2024)	✓	✗	✗	493 seeds/vign./traj.	1	Single	-
AirGapAgent Bagdasarian et al. (2024)	✓	✓	✗	16,640 (20 prof. $\times$ 26 fld $\times$ 8 tasks)	1	Single	-
CIMemories Mireshghallah et al. (2025)	✓	✓	✗	$\sim$ 450 (10 prof. $\times$ 45 ctx)	1	Single	-
TAMAS Kavathekar et al. (2025)	✗	✗	✓	400 (300 adv. + 100 benign)	Multi	Multi-agent	n/r
AgentLeak Yagoubi et al. (2026)	✗	✗	✓	1,000 (4,979 traces)	Multi	Multi-agent	n/r

Table 1: Comparison of existing privacy benchmarks for LLMs. CI -grounded evaluation, persistent Mem ory, multi-turn Dial ogue, plus dataset scale (# datapoints), interaction length (# turns / max turns), and single- vs. multi-user setting. † ConfAIde spans 4 tiers; the CI-relevant Tiers 3–4 contain

\sim

270 and

\sim

50 items respectively. n/r shows not reported in paper.

Panel of Figure 2 — Figure 2: Two-agent dialogue setting: conversation prompt, prompt conditions, and how the conversation progresses.

Figure 3: Task Completeness vs Full Privacy Leakage rate across open-source language model families. Each ellipse represents one sender (agent-1) model averaged across prompt conditions; points within each ellipse correspond to the five receiver (agent-2) assistants.

Figure 4: Average leakage rate for different sender models across all receiver models, evaluated at 6 turns. Each line corresponds to a sender-side prompting condition: Control (PC0), privacy-aware (PC1), and task-maximizing (PC2). The bar shows both partial and full leakage rates.

Figure 5: Average leakage rate for different sender models across all receiver models, evaluated at different rounds of conversation.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Cumulative leakage rate for different sender models across all receiver models, evaluated at different rounds of conversation for all 3 prompt conditions.

Panel of Figure 7 — Figure 7: Task template prompt

Panel of Figure 8 — Figure 8: PC1 Prompt

Panel of Figure 9 — Figure 9: PC2 Prompt

Panel of Figure 10 — Figure 10: REVEAL prompt used in Stage 1 of evaluation

Figure 11: Sender model vs Receiver model Full Completeness Scores averaged across prompt conditions

Figure 12: Sender model vs Receiver model Full Leakage Rates averaged across prompt conditions

Figure 13: Sender model vs Receiver model Full Completeness Scores For Privacy Aware prompt condition

Figure 14: Sender model vs Receiver model Full Leakage Rates for Privacy Aware prompt condition

Model	PC	T1	T6	Rel. (T1→T6)
Q-4B	PC0	15.8	49.5	$3.14\times$
	PC1	18.9	28.0	$1.48\times$
	PC2	21.1	45.3	$2.15\times$
Q-8B	PC0	23.9	23.6	$0.99\times$
	PC1	20.5	19.8	$0.97\times$
	PC2	27.4	25.5	$0.93\times$

Appendix

Table 2: Leakage rate (

\%

) across conversation lengths and prompt conditions (PC).

Figure 15: Average leakage rate for different sender models across all receiver models, evaluated at different turns of conversation.

Figure 16: Average completion rates for different sender models across all receiver models, evaluated for different prompt conditions.

Panel of Figure 17 — Figure 17: Example of Necessary and Private items shared in the conversation. Douglas Perry applies for a personal loan to cover adoption expenses; across two rounds he discloses five necessary items and leaks one irrelevant personal detail (sobriety status). Example taken from Qwen3-30B-A3B x Qwen3-4B-Thinking experiment

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 272 papers

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaAI Agent Safety Benchmarks Multi-Agent LLM Systems
Foundation AI · USA · Corvic AI
PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
May 29, 2026Mingxuan Zhang, Jiahui Han, Dadi Guo +5Data Leakage Privacy Auditing
Shanghai Artificial Intelligence Laboratory · Southeast University
CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents
Sep 18, 2026Tao Huang, Guosen Wu, Guolong Zheng +5Privacy Leakage in Language Models LLM Agent Security
Minjiang University, Fuzhou, Fujian 350108, China · Renmin University of China, Beijing 100872, China · RMIT University, Melbourne, VIC 3000, Australia

May 26, 2026J/K move · Enter open · S save

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems

Aman Priyanshu, Supriti Vijay, Esha Pahwa

Foundation AI · USA · Corvic AI

AI Agent Safety Benchmarks Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Security Privacy Leakage in Language Models Data Leakage+2

Open paper AskPDF arXiv

LLM safety evaluations predominantly test models in isolation, yet deployed AI agents increasingly operate within persistent social environments alongside other agents. We introduce a Moltbook-style simulation platform where thousands of LLM agents interact across communities over a simulated month, and use it to evaluate privacy as a downstream safety concern under varying degrees of social pressure. We find that shifting from single turn to multi turn social evaluation amplifies privacy violations (CIMemories 19.95% to Ours 45.30% across OpenAI models), that leakage is socially contagious, with agents 8 times more likely to disclose sensitive information after observing a peer do so, and that explicit privacy instructions reduce but do not eliminate this effect, leaving leakage rates above 37.8% even with safeguards. Our findings suggest that static chat based safety benchmarks systematically underestimate risks in agentic deployment, and that social context alone is sufficient to elicit sensitive disclosures that single turn evaluations would never surface.

Abstract

Privacy Leakage in Language Models

Explore similar work

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents