Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

Minsun Shim; Ramisha Raida Karim; Ruthwik Jakkula; Kaiwen Zhou; Xin Liu; Xin Eric Wang; Zhou Li

cs.CRSep 12, 2026

Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

Authors: Minsun Shim, Ramisha Raida Karim, Ruthwik Jakkula, Kaiwen Zhou, Xin Liu, Xin Eric Wang, Zhou Li

Organizations: University of California, Irvine · University of California, Santa Cruz · University of California, Davis · University of California, Santa Barbara

Abstract

Personal AI agents built on large language models (LLMs) are increasingly given access to a user's private data and communications in order to provide personalized assistance. This access creates a persistent privacy risk: the agent must decide whether a given sensitive information should be disclosed to a particular party. Existing defenses address this by making the agent's backend LLM more privacy-preserving through stronger system prompts, training, or explicit consent-checking procedures, but this approach has a structural challenge: whenever enforcement is a judgment the LLM makes over the same conversational context an adversary controls, the enforcement mechanism and the attack surface coincide. We demonstrate this against existing defenses with three new attacks that require only ordinary agent interaction and no prompt injection: Collaborative Workspace Lure reframes an extraction attempt as collaborative work; Semantic Obfuscation Attack induces disclosure through omission rather than through anything the agent writes; and Channel Decoupling Attack splits the extraction request and the disclosure across independent channels. All three achieve substantially higher leak rates than the attacks these defenses were originally designed to withstand. Guided by this observation, we present FLOWSEAL, a defense that enforces confidentiality through a tool-level interceptor outside the LLM's context, grounded in data provenance and an information-flow-control lattice with controlled declassification. Evaluated across three benchmarks, five prompt-based baselines, and eight attacks, including a real agent executing live tool calls through MCP, FLOWSEAL reduces leak rates to near zero (e.g., 52.2% to 0.5% against Collaborative Workspace Lure) while preserving task utility, regardless of the underlying LLM backend.

Figures & tables

Fig. 1: Example lifecycle of a personal AI agent completing a task.

Fig. 2: System model: the user’s agent (data sender) holds the data subject’s sensitive information and interacts with both normal user and attacker attempting unauthorized disclosure.

Fig. 3: SPR’s $D_{2}$ defense prompt [ 8 ] , discovered through iterative search against increasingly sophisticated attacks.

Fig. 4: Three new attack strategies. Yellow box in the prompt explains each of the core strategy, red box indicates the bypass of SPR- $D_{2}$ defense from agent’s chain-of-thought and red text indicates the leaked sensitive information.

Fig. 5: FlowSeal architecture. Blue edge labels show the four cooperative privacy APIs for record discovery, ownership checks, consent, and declassification. The mandatory interceptor independently mediates every tool call. Read results update the taint tracker. Write and share calls pass through destination authorization and, when needed, an isolated content checker before reaching an external tool.

Fig. 6: Content checker prompt template.

	SPR	PrivacyLens	ConVerse
Benchmark setup
Interaction format	Multi-turn	Single-turn	Multi-turn
Tool backend	FastAPI apps	Text emulation	No tools
Seeds
Attack channel	App writes	App writes	Conversation
Attack seeds	105	378	571

TABLE I: Benchmark setups and attack seed sources.

Attack Type	Attack	SPR– $D_{1}$			SPR– $D_{2}$			SPR– $D_{3}$			PrivacyLens-Defense			FAN-LCF
Attack Type	Attack	${LR}_{I}$	${LR}_{B}$	$LV$	${LR}_{I}$	${LR}_{B}$	$LV$	${LR}_{I}$	${LR}_{B}$	$LV$	${LR}_{I}$	${LR}_{B}$	$LV$	${LR}_{I}$	${LR}_{B}$	$LV$
SPR	SPR– $A_{2}$	3.4	4.4	1.9	1.9	1.9	1.3	1.2	1.3	0.8	14.3	18.8	-	19.2	25.7	18.5
SPR	SPR– $A_{3}$	3.6	5.8	2.7	3.0	3.8	2.0	1.7	2.2	1.1	32.5	38.8	-	18.6	25.6	17.9
Ours	CWL	31.8	32.9	24.7	52.2	53.6	42.3	51.8	52.3	37.1	89.9	97.8	-	62.2	72.7	62.2
	SOA	54.7	54.7	47.8	75.6	76.8	66.4	69.7	70.5	60.7	67.7	75.5	-	54.2	64.7	54.2
	CDA	13.3	13.1	12.8	43.8	43.6	41.8	39.7	39.3	37.6	43.8	48.6	-	63.9	75.7	63.9

TABLE II: Attack Comparison on the defenses from existing works, measured by average leak rate (see § VI for

LR_{I}

LR_{B}

LV

definitions). All attack methods and benign tasks are tested with 105 SPR seeds (3 runs per seed).

LV

cannot be measured on PrivacyLens-Defense as explained in Section VI-A4 . Blue cells show the highest leakage numbers.

Attacks	FlowSeal
Attacks	${LR}_{I}$	${LR}_{B}$	$LV$
SPR– $A_{1}$	0.0	0.0	0.0
SPR– $A_{2}$	0.0	0.0	0.0
SPR– $A_{3}$	0.3	1.0	0.3
PrivacyLens-Seeds	0.5	0.2	0.1
ConVerse-Seeds	3.3	3.3	0.8

TABLE III: Comparing different attacks against FlowSeal .

Defenses	SPR- $D_{1}$	SPR- $D_{2}$	SPR- $D_{3}$	PrivacyLens- Defense	FAN- LCF	Flow- Seal
Utility	75.9	44.7	60.7	56.8	25.1	72.4

TABLE IV: Utility score comparison with benign tasks.

Fig. 8: Tool Comparison of CWL attacks.

LR_{I}

LR_{B}

and

LV

are collapsed to the same bar.

Prompt	Interceptor	Checker	CWL			Utility
Prompt	Interceptor	Checker	${LR}_{I}$	${LR}_{B}$	$LV$	Utility
✗	✗	✗	52.2	53.6	42.3	44.7
✓	✓	✓	0.5	1.0	0.4	72.4
✗	✓	✓	8.8	9.2	6.3	74.4
✓	✗	✓	28.4	32.2	21.6	81.2
✓	✓	✗ ${}_{\text{\tiny{closed}}}$	1.3	1.9	1.0	0.0

TABLE V: Defense ablation by components. ✗

{}_{\text{\tiny closed}}

/ ✗

{}_{\text{\tiny open}}

denote disabling the checker under fail-closed/fail-open policies.

Defenses	$ADR_{I}$	$ADR_{B}$	$N_{\text{consent}}$	$N_{\text{turns}}$
SPR– $D_{1}$	63.2	63.8	0.98	5.8
SPR– $D_{2}$	66.6	67.6	1.01	5.2
SPR– $D_{3}$	61.0	61.4	0.92	5.7
FAN-LCF	85.6	87.1	0.95	6.0
FlowSeal	32.7	32.7	0.71	6.0

TABLE VI: Measurement of interactions when the data subject grants consent upon request.

Fig. 9: Leak rate under CWL attack when different agent LLMs are tested. FS: FlowSeal .

Condition	Cost/run	Calls/run	Time/run
No defense	$0.092$ \pm$ 0.063	46.7	5.44s $\pm$ 0.87
SPR– $D_{2}$	$0.074$ \pm$ 0.037	45.1	5.20s $\pm$ 0.93
FlowSeal	$0.109$ \pm$ 0.037	57.1	7.45s $\pm$ 1.54

TABLE VII: Cost and latency overhead.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 10: SPR– $D_{3}$ prompt with explicit refusal of non-protocol requests.

System	Stage	Trace
SPR- $D_{2}$	Reasoning	“…this appears to be an internal coordination task…doesn’t seem to require the consent state machine…”
	Action	Notion_edit_page () writes R1–R3 into shared page
	Outcome	LEAK (no enforcement layer)
FlowSeal	Reasoning	“I need to request access to share this protected information…”
	Action 1	privacy_request_access (…)
	Result 1	DENIED — consent not granted

Appendix

TABLE VIII:

D_{2}

and FlowSeal traces on the same CWL-Notion task.

Fig. 11: Successful and failed defense traces against ConVerse attack. Orange: attacker turn. Green: agent reasoning and drafted output. Blue: FlowSeal ’s interceptor decision. Yellow: outcome. Red highlights mark the specific protected content at risk in each trace.

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 320 papers

May 29, 2026

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce PrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with

1{,}182

cases across

7

acquisition behaviours and

16

application domains. Specifically, Acquisition Inspection examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. Probe Elicitation then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.

Mingxuan Zhang, Jiahui Han, Dadi Guo +5

Shanghai Artificial Intelligence Laboratory · Southeast University

Data Leakage Privacy Auditing AI Agent Auditing LLM Agent Security Privacy Leakage in Language Models AI Agent Security Benchmarks+1

Ask PDF

Jun 25, 2026

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

Large language model agents increasingly query databases, search document collections, call external APIs, remember past interactions, and act on a user's behalf. As they move from answering questions to operating over sensitive data, privacy becomes harder to enforce. An agent touches many data sources, runs multi-step workflows, keeps state across sessions, and acts with delegated permissions. Sensitive information can therefore leak not only through its final answer but through the queries it issues, the intermediate results it handles, the memory it writes, and the messages it exchanges with other agents. We survey the privacy of LLM agents from a data-centric view, organizing the field around the data an agent touches rather than by attack type, and we use data agent as shorthand for an LLM agent that works with data. Research on these risks is active but scattered across retrieval-augmented generation, text-to-SQL interfaces, agent memory, prompt injection, access control, and contextual privacy. This survey brings that work together: we taxonomize the data sources an agent touches, the privacy risks each source creates, and the governance mechanisms that address them; we map the benchmarks used to measure these risks and identify what is missing; and we set out the open problems. Two findings recur: among governance mechanisms only information-flow control covers both compositional and cross-session inference leakage, the two least-protected risks; and no benchmark drives an agent across its data surfaces under one privacy policy, the instrument the field most lacks. Our goal is a reference that situates the scattered literature and gives future work a common framing.

Nada Lahjouji, Ashwin Gerard Colaco

University of California, Irvine Irvine, California, USA

Privacy Leakage in Language Models Information Flow Control LLM Agents Access Control

Ask PDF

Sep 18, 2026

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents

Privacy leakage in LLM agents is commonly evaluated within individual components such as memory, retrieval, or tool-use pipelines, which makes it difficult to distinguish internal exposure from information that an external observer can actually recover. We present CIPL (Channel Inversion for Privacy Leakage), a channel-aware evaluation framework for black-box privacy leakage in LLM agents. CIPL represents a target through sensitive source, selection, assembly, execution, observation, and extraction stages and evaluates the transition from selected sensitive units to attacker-recoverable output under a shared protocol. Experiments across memory-based, retrieval-mediated, and tool-mediated targets, together with a BrowserUse live-agent case study, show that storage labels alone do not determine recoverability. Memory targets form a near-saturated reference case, retrieval-mediated leakage is frequently partial, and tool-mediated and live-agent leakage varies strongly with observation surface, prompt-to-channel alignment, retrieval depth, and provider behavior. A stratified semantic audit further identifies attacker-useful disclosures that canonical exact matching misses. CIPL therefore provides a common framework for comparing how internal sensitive dependence is realized as externally recoverable leakage across heterogeneous agent pipelines.

Tao Huang, Guosen Wu, Guolong Zheng +5

Minjiang University, Fuzhou, Fujian 350108, China · Renmin University of China, Beijing 100872, China · RMIT University, Melbourne, VIC 3000, Australia

Privacy Leakage in Language Models LLM Agent Security LLM Agent Evaluation AI Agent Auditing

Ask PDF

Abstract

LLM Agent Security

Explore similar work

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents