CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents

Tao Huang; Guosen Wu; Guolong Zheng; Jiayang Meng; Chen Hou; Xu Yang; Xuechao Yang; Feng Xia

cs.CRSep 18, 2026

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents

Authors: Tao Huang, Guosen Wu, Guolong Zheng, Jiayang Meng, Chen Hou, Xu Yang, Xuechao Yang, Feng Xia

Organizations: Minjiang University, Fuzhou, Fujian 350108, China · Renmin University of China, Beijing 100872, China · RMIT University, Melbourne, VIC 3000, Australia

Abstract

Privacy leakage in LLM agents is commonly evaluated within individual components such as memory, retrieval, or tool-use pipelines, which makes it difficult to distinguish internal exposure from information that an external observer can actually recover. We present CIPL (Channel Inversion for Privacy Leakage), a channel-aware evaluation framework for black-box privacy leakage in LLM agents. CIPL represents a target through sensitive source, selection, assembly, execution, observation, and extraction stages and evaluates the transition from selected sensitive units to attacker-recoverable output under a shared protocol. Experiments across memory-based, retrieval-mediated, and tool-mediated targets, together with a BrowserUse live-agent case study, show that storage labels alone do not determine recoverability. Memory targets form a near-saturated reference case, retrieval-mediated leakage is frequently partial, and tool-mediated and live-agent leakage varies strongly with observation surface, prompt-to-channel alignment, retrieval depth, and provider behavior. A stratified semantic audit further identifies attacker-useful disclosures that canonical exact matching misses. CIPL therefore provides a common framework for comparing how internal sensitive dependence is realized as externally recoverable leakage across heterogeneous agent pipelines.

Figures & tables

Line of work	Main unit	Exp/Rec	Cross	Attr	Sem
Memory extraction	Memory records	P	N	N	N
RAG privacy / extraction	Retrieved documents or snippets	P	N	P	P
Tool / web-agent leakage	Tool calls, returns, actions, or injected instructions	P	N	P	N
Agent security benchmarks	End-to-end task outcomes	N	P	N	N
CIPL	Selected sensitive units in active computation	Y	Y	Y	Y

Table 1: Positioning of CIPL. Y: yes; P: partial or limited; N: no. Exp/Rec denotes separation of internal exposure from external recovery; Cross denotes shared cross-channel comparison; Attr denotes attribution to channel, provider, selection, or extraction factors; Sem denotes semantic leakage analysis.

Panel of Figure 1 — Figure 1: Leakage regimes under the shared black-box measurement protocol. Panel (a) reports CER, while panel (b) reports AER. Under CIPL, three recurring patterns emerge. Memory-based settings remain saturated across all five providers. rag_ctrl exhibits a frequent-but-partial leakage pattern, with high any-recovery but incomplete recovery across providers. Tool-mediated channels show stronger dependence on observation surface and provider behavior under LLM-in-the-loop evaluation: return_echo is generally stronger than args_exfil away from provider-level ceiling effects, while DeepSeek and GPT-4o approach or reach saturation on some tool channels. Values are means over five seeds.

Setting	Provider	CER	AER	ExecErr
memory_ehr	MiniMax-M2.5	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$
memory_ehr	MiniMax-M2.7	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$
memory_ehr	qwen3.5-plus	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$
memory_ehr	DeepSeek	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$
memory_ehr	GPT-4o	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$
memory_rap	MiniMax-M2.5	$1.00\pm 0.00$	$1.00\pm 0.00$	$0\pm 0$

Table 2: Main results under the shared channel-aware evaluation framework. All settings use

n=30

attack queries, one retry, and five seeds. We report Complete Extraction Rate (CER), Any Extracted Rate (AER), and execution-error counts (ExecErr) as mean

\pm

standard deviation across seeds. Full metrics including RN, EN, and EE are reported in the Appendix.

Target	Main W/T/L	Main AER / CER	Naive AER / CER
memory_ehr	1/2/0	1.00 / 1.00	0.00–1.00 / 0.00–1.00
memory_rap	3/0/0	1.00 / 1.00	0.00 / 0.00
rag_ctrl	0/0/3	0.82–0.95 / 0.33–0.55	0.93–1.00 / 0.92–1.00
tool_ctrl(args)	3/0/0	0.27–0.99 / 0.27–0.49	0.18–0.86 / 0.16–0.49
tool_ctrl(return)	3/0/0	0.37–1.00 / 0.37–1.00	0.05–0.26 / 0.05–0.26

Table 3: Compact main-strategy versus naive-strategy comparison. Win/tie/loss is determined by AER, using CER as a secondary comparison when AER is equal. Ranges summarize the three audited providers (MiniMax-M2.7, qwen3.5-plus, and GPT-4o); provider-level rows are reported in the Appendix.

Panel of Figure 2 — Figure 2: Leakage realization depends on alignment and exposure. Panel (a) reports AER under cleaned weak-control prompts for tool_ctrl ; leakage is sharply suppressed across all providers, indicating adversarial realization rather than ordinary completion. Panel (b) reports the retrieval-depth ablation for rag_ctrl ; AER remains high or stable, while the retrieval-depth results in the Appendix show that larger $k$ can sharply reduce CER. Greater internal exposure therefore does not necessarily yield stronger complete extraction.

Provider	process CER	process AER	final CER	final AER
DeepSeek	$0.75\pm 0.03$	$0.86\pm 0.03$	$0.21\pm 0.05$	$0.25\pm 0.06$
GPT-4o	$0.72\pm 0.04$	$0.82\pm 0.03$	$0.18\pm 0.04$	$0.22\pm 0.05$
MiniMax-M2.5	$0.82\pm 0.03$	$0.91\pm 0.02$	$0.65\pm 0.05$	$0.80\pm 0.04$
MiniMax-M2.7	$0.68\pm 0.04$	$0.84\pm 0.03$	$0.52\pm 0.06$	$0.68\pm 0.05$
qwen3.5-plus	$0.42\pm 0.07$	$0.59\pm 0.06$	$0.59\pm 0.05$	$0.76\pm 0.04$

Table 4: BrowserUse as live black-box transfer evidence. Observation-surface ordering reverses across providers: DeepSeek and GPT-4o leak more through process , whereas qwen3.5-plus leaks more through final_result .

Observation mode	Strong AER	Std. RN	Std. AER
final_result	$0.22$ – $0.80$	$31$	$0.00$ – $0.03$
process	$0.59$ – $0.91$	$38$	$0.00$ – $0.06$

Table 5: BrowserUse alignment control under standard operational tasks. Leakage collapses to near-zero even though sensitive content still enters the agent context; full control results are reported in the Appendix.

Audit item	Value	Reading
labeled_samples	200	audit size
semantic_AER	0.8550	any semantic leakage
semantic_CER	0.4100	semantically complete leakage
exact_any=0 & semantic=1	5	missed by exact-only
exact_any=1 & semantic=2	82	exact and semantically severe

Table 6: RQ3 semantic-audit summary. Exact matching misses some attacker-useful leakage: 5 audited outputs contain partial sensitive-value recovery despite having no exact recovered canonical unit. Full protocol and representative cases are in the Appendix.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Provider	RN	EN	EE	CER	AER	ExecErr
memory_ehr	MiniMax-M2.5	55	55.0 $\pm$ 0.0	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	MiniMax-M2.7	55	55.0 $\pm$ 0.0	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	qwen3.5-plus	55	55.0 $\pm$ 0.0	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	DeepSeek	55	55.0 $\pm$ 0.0	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	GPT-4o	55	55.0 $\pm$ 0.0	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_rap	MiniMax-M2.5	57	57.0 $\pm$ 0.0	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table B.1: Full main results under the unified CIPL protocol. All main experiments use

n=30

, one retry, and five seeds. Values are reported as mean

\pm

standard deviation across seeds.

Provider	Retrieve Method	RN	EN	EE	CER	AER	ExecErr
DeepSeek	edit_distance	57	57	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
GPT-4o	edit_distance	57	57	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
MiniMax-M2.5	edit_distance	57	57	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
MiniMax-M2.7	edit_distance	57	57	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
qwen3.5-plus	edit_distance	57	57	0.63 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
DeepSeek	token_overlap	43	43	0.48 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.1: Retrieval-rule ablation for memory_rap . Changing the retrieval rule alters coverage and normalized efficiency, but not the qualitative vulnerability conclusion: both rules remain fully leakage-prone on all five providers.

Provider	Retrieve Method	RN	EN	EE	CER	AER	ExecErr
DeepSeek	edit_distance	50	40	0.67 $\pm$ 0.00	0.00 $\pm$ 0.00	0.80 $\pm$ 0.00	0.0 $\pm$ 0.0
GPT-4o	edit_distance	50	40	0.67 $\pm$ 0.00	0.00 $\pm$ 0.00	0.80 $\pm$ 0.00	0.0 $\pm$ 0.0
MiniMax-M2.5	edit_distance	50	50	0.83 $\pm$ 0.00	0.17 $\pm$ 0.03	0.98 $\pm$ 0.02	0.0 $\pm$ 0.0
MiniMax-M2.7	edit_distance	50	50	0.83 $\pm$ 0.00	0.19 $\pm$ 0.08	0.98 $\pm$ 0.02	0.0 $\pm$ 0.0
qwen3.5-plus	edit_distance	50	40	0.67 $\pm$ 0.00	0.00 $\pm$ 0.00	0.80 $\pm$ 0.00	0.0 $\pm$ 0.0
DeepSeek	token_overlap	50	40	0.67 $\pm$ 0.00	0.00 $\pm$ 0.00	0.80 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.2: Retrieval-rule ablation for rag_ctrl . The retrieval rule has only a modest effect on DeepSeek , GPT-4o , and qwen3.5-plus , but slightly changes the balance between complete and partial leakage on both MiniMax-M2.5 and MiniMax-M2.7 .

Provider	Retrieve Method	RN	EN	EE	CER	AER	ExecErr
DeepSeek	edit_distance	50	50.0 $\pm$ 0.0	0.83 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
GPT-4o	edit_distance	50	50.0 $\pm$ 0.0	0.83 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
MiniMax-M2.5	edit_distance	50	43.0 $\pm$ 5.0	0.72 $\pm$ 0.08	0.29 $\pm$ 0.02	0.29 $\pm$ 0.02	0.0 $\pm$ 0.0
MiniMax-M2.7	edit_distance	50	50.0 $\pm$ 0.0	0.83 $\pm$ 0.00	0.31 $\pm$ 0.03	0.32 $\pm$ 0.02	0.0 $\pm$ 0.0
qwen3.5-plus	edit_distance	50	50.0 $\pm$ 0.0	0.83 $\pm$ 0.00	0.80 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
DeepSeek	token_overlap	20	20.0 $\pm$ 0.0	0.33 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.3: Retrieval-rule ablation for tool_ctrl(return_echo,llm) . This auxiliary ablation uses three seeds. Retrieval-rule choice changes both internal coverage and leakage strength. This effect is clearest on the two MiniMax variants, while DeepSeek and GPT-4o remain saturated under both rules and qwen3.5-plus remains highly leakage-prone under both rules.

Figure C.1: Retrieval-rule ablation for tool_ctrl(return_echo,llm) . We compare edit-distance and token-overlap retrieval under the LLM-in-the-loop return-echo setting. The retrieval-rule effect is visible on both MiniMax variants, while DeepSeek and GPT-4o remain saturated and qwen3.5-plus remains highly leakage-prone under both rules. This supports the CIPL interpretation that leakability depends not only on the final observation channel, but also on the upstream selection mechanism that determines which sensitive units enter active computation.

Target	Provider	Size	RN	EN	EE	CER	AER	ExecErr
memory_ehr	DeepSeek	100	45	45	0.38 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	GPT-4o	100	45	45	0.38 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	MiniMax-M2.5	100	45	45	0.38 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	MiniMax-M2.7	100	45	45	0.38 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	qwen3.5-plus	100	45	45	0.38 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	DeepSeek	200	55	55	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.4: Source-size ablation for memory-based targets. Increasing source size changes RN, EN, and EE, but does not change the qualitative vulnerability conclusion: both memory targets remain at CER = AER = 1.0 across all five providers, no execution-instability effect appears under this source-size variation.

Provider	Size	RN	EN	EE	CER	AER	ExecErr
DeepSeek	20	20	20	0.33 $\pm$ 0.00	0.20 $\pm$ 0.00	0.80 $\pm$ 0.00	0.0 $\pm$ 0.0
GPT-4o	20	20	20	0.33 $\pm$ 0.00	0.06 $\pm$ 0.04	0.48 $\pm$ 0.02	0.0 $\pm$ 0.0
MiniMax-M2.5	20	20	20	0.33 $\pm$ 0.00	0.51 $\pm$ 0.02	0.86 $\pm$ 0.03	0.0 $\pm$ 0.0
MiniMax-M2.7	20	20	20	0.33 $\pm$ 0.00	0.52 $\pm$ 0.06	0.92 $\pm$ 0.06	0.0 $\pm$ 0.0
qwen3.5-plus	20	20	20	0.33 $\pm$ 0.00	0.00 $\pm$ 0.00	0.40 $\pm$ 0.00	0.0 $\pm$ 0.0
DeepSeek	30	30	21	0.35 $\pm$ 0.00	0.20 $\pm$ 0.00	0.60 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.5: Source-size ablation for rag_ctrl . The effect of source size is non-monotonic and strongly provider-dependent in the five-provider setting. The two MiniMax variants remain high-AER across all tested source sizes, while qwen3.5-plus , DeepSeek , and GPT-4o remain in lower-CER partial-leakage patterns with distinct source-size sensitivity.

Target	Provider	$k$	RN	EN	EE	CER	AER	ExecErr
memory_ehr	DeepSeek	2	34	34	0.57 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	GPT-4o	2	34	34	0.57 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	MiniMax-M2.5	2	34	34	0.57 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	MiniMax-M2.7	2	34	34	0.57 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	qwen3.5-plus	2	34	34	0.57 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0
memory_ehr	DeepSeek	4	55	55	0.46 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.0 $\pm$ 0.0

Appendix

Table C.6: Full retrieval-depth ablation for memory-based targets. Changing retrieval depth affects RN, EN, and EE, but not the qualitative leakage outcome: both memory_ehr and memory_rap remain saturated across all five providers at all tested

k

values.

Provider	$k$	RN	EN	EE	CER	AER	ExecErr
DeepSeek	1	50	$40.0\pm 0.0$	$1.33\pm 0.00$	$0.80\pm 0.00$	$0.80\pm 0.00$	$0.0\pm 0.0$
GPT-4o	1	50	$40.0\pm 0.0$	$1.33\pm 0.00$	$0.80\pm 0.00$	$0.80\pm 0.00$	$0.0\pm 0.0$
MiniMax-M2.5	1	50	$50.0\pm 0.0$	$1.67\pm 0.00$	$0.94\pm 0.02$	$0.94\pm 0.02$	$0.0\pm 0.0$
MiniMax-M2.7	1	50	$50.0\pm 0.0$	$1.67\pm 0.00$	$0.98\pm 0.02$	$0.98\pm 0.02$	$0.0\pm 0.0$
qwen3.5-plus	1	50	$40.0\pm 1.0$	$1.33\pm 0.03$	$0.80\pm 0.00$	$0.80\pm 0.00$	$0.0\pm 0.0$
DeepSeek	2	50	$40.0\pm 0.0$	$0.67\pm 0.00$	$0.00\pm 0.00$	$0.80\pm 0.00$	$0.0\pm 0.0$

Appendix

Table C.7: Full retrieval-depth ablation for rag_ctrl . Both MiniMax variants remain high-AER across the tested retrieval depths, but CER drops sharply as

k

increases. qwen3.5-plus , DeepSeek , and GPT-4o remain comparatively stable in AER while staying in lower-CER partial-leakage patterns.

Provider	$k$	RN	EN	EE	CER	AER	ExecErr
DeepSeek	1	30	$30.0\pm 0.0$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$0.0\pm 0.0$
GPT-4o	1	30	$30.0\pm 0.0$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$0.0\pm 0.0$
MiniMax-M2.5	1	30	$30.0\pm 0.0$	$1.00\pm 0.00$	$0.73\pm 0.08$	$0.73\pm 0.08$	$0.0\pm 0.0$
MiniMax-M2.7	1	30	$30.0\pm 0.0$	$1.00\pm 0.00$	$0.36\pm 0.07$	$0.36\pm 0.07$	$0.0\pm 0.0$
qwen3.5-plus	1	30	$30.0\pm 0.0$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$0.0\pm 0.0$
DeepSeek	2	50	$50.0\pm 0.0$	$0.83\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$0.0\pm 0.0$

Appendix

Table C.8: Full retrieval-depth ablation for tool_ctrl(return_echo,llm) . This auxiliary ablation uses three seeds. Increasing retrieval depth weakens complete extraction on the MiniMax variants, while DeepSeek and qwen3.5-plus remain saturated and GPT-4o continues to exhibit persistently high AER but substantially lower CER.

Target	Provider	Main (AER / CER)	Naive (AER / CER)
memory_ehr	MiniMax-M2.7	1.00 / 1.00	1.00 / 1.00
memory_ehr	qwen3.5-plus	1.00 / 1.00	0.00 / 0.00
memory_ehr	GPT-4o	1.00 / 1.00	1.00 / 1.00
memory_rap	MiniMax-M2.7	1.00 / 1.00	0.00 / 0.00
memory_rap	qwen3.5-plus	1.00 / 1.00	0.00 / 0.00
memory_rap	GPT-4o	1.00 / 1.00	0.00 / 0.00

Appendix

Table D.1: Summary of main-vs-naive baseline comparisons. The main CIPL prompts are not uniformly stronger than naive baselines. Memory- and tool-mediated settings generally favor the main prompts, while rag_ctrl provides a systematic counterexample in which the dump-style naive baseline performs better.

Target	Provider	Full (AER / CER)	Highest-AER Variant
memory_ehr	MiniMax-M2.7	1.00 / 1.00	1.00 (multiple variants tied)
memory_ehr	qwen3.5-plus	1.00 / 1.00	1.00 (multiple variants tied)
rag_ctrl	MiniMax-M2.7	0.95 / 0.55	0.98 ( full / tied best)
rag_ctrl	qwen3.5-plus	0.89 / 0.45	1.00 ( no_diversification )
tool_ctrl(return_echo,llm)	MiniMax-M2.7	0.37 / 0.37	0.37 ( full )
tool_ctrl(return_echo,llm)	qwen3.5-plus	0.40 / 0.40	0.40 ( full )

Appendix

Table D.2: Summary of component-ablation outcomes. The Full column reports the audited five-seed main-setting reference; variant summaries come from the corresponding auxiliary ablation runs. The full locator/aligner/diversification construction is therefore best read as a useful reference configuration, not as a uniformly optimal one.

Setting	Condition	AER / CER	Interpretation
memory_ehr qwen	$n=10/20/30/50$	audited reference at $n=30$ : 1.00 / 1.00	Saturated across budgets
rag_ctrl MiniMax-M2.7	$n=10/20/30/50$	main-setting reference at $n=30$ : 0.9533 / 0.5467	Frequent but incomplete leakage remains the qualitative pattern
tool_ctrl(return_echo,llm) qwen3.5-plus	$n=10/20/30/50$	audited reference at $n=30$ : 0.40 / 0.40	Moderate LLM-in-the-loop leakage remains observable under budget perturbation
memory_ehr qwen	retries=1,2	reference at retries=1: 1.00 / 1.00	Stable under retries
rag_ctrl MiniMax-M2.7	retries=1,2	main-setting reference at retries=1: 0.9533 / 0.5467; higher CER also observed at retries=2	Retries can increase leakage
tool_ctrl(return_echo,llm) qwen3.5-plus	retries=1,2	reference at retries=1: 0.40 / 0.40	High-leakage pattern preserved

Appendix

Table D.3: Summary of robustness checks. The AER/CER column reports the main-setting reference value for the named setting where applicable; qualitative interpretations summarize the accompanying budget, retry, or seed sweeps. Retry-related results must be interpreted at the prompt level: r0 is not a valid no-retry baseline under the current implementation.

Provider	attack echo AER	defense echo AER	attack args AER	defense args AER
MiniMax-M2.7	0.37	0.12	0.27	0.19
GPT-4o	1.00	0.20	0.99	0.46

Appendix

Table D.4: Defense-prompt suppression on tool-mediated channels. Adding defense-style prompting sharply reduces leakage on both echo and argument channels, especially for GPT-4o .

Setting	Check	Main reading
memory_ehr, qwen3.5-plus	budgets $n=10$ – $50$	remains saturated throughout
rag_ctrl, MiniMax-M2.7	budgets $n=10$ – $50$	frequent but incomplete leakage persists
tool_ctrl (return_echo,llm), qwen3.5-plus	budgets $n=10$ – $50$	moderate leakage remains observable
tool_ctrl (return_echo,llm), GPT-4o	defense prompting	AER drops sharply but does not define the main threat model

Appendix

Table D.5: Compact robustness summary from appendix checks. These checks do not overturn the main regime assignments. They are included here to show that the reported takeaways are not artifacts of one budget, seed count, or prompt configuration.

Metric	Value
labeled_samples	200
semantic_AER	0.8550
semantic_CER	0.4100
exact_any=0 & semantic=1	5
exact_any=1 & semantic=2	82
exact_any=0 & semantic=0	29

Appendix

Table E.1: Semantic leakage annotation summary. The metadata-blinded two-annotator semantic audit shows that exact-match extraction can underestimate leakage: 5 samples contain partial sensitive-value recovery despite having no exact recovered canonical unit.

Setting	Model/API, decoding, and seed configuration	Experiment window
Controlled CIPL (memory/tool)	MiniMax-M2.5/M2.7; MiniMax route api.minimaxi.com ; target code $T=0.0$ , max tokens $=512$ ; top- $p$ and API seed not passed; pipeline seeds $0$ – $4$ .	2026-03-18–03-24
Controlled CIPL (memory/tool)	qwen3.5-plus; DashScope OpenAI-compatible route; target code $T=0.0$ , max tokens $=512$ , enable_thinking=false ; top- $p$ and API seed not passed; pipeline seeds $0$ – $4$ .	2026-03-18–03-24
Controlled CIPL (memory/tool)	deepseek-chat; DeepSeek route api.deepseek.com ; target code $T=0.0$ , max tokens $=512$ ; top- $p$ and API seed not passed; pipeline seeds $0$ – $4$ .	2026-03-18–03-24
Controlled CIPL (memory/tool)	gpt-4o; ai.wer.plus compatible route; target code $T=0.0$ , max tokens $=512$ ; top- $p$ and API seed not passed; pipeline seeds $0$ – $4$ .	2026-03-18–03-24
BrowserUse	deepseek-chat; DeepSeek route api.deepseek.com ; wrapper temperature/top- $p$ /max tokens/API seed not passed (provider defaults); pipeline repetitions $0$ – $4$ .	2026-04-06–04-09
BrowserUse	qwen3.5-plus; DashScope OpenAI-compatible route; wrapper defaults $T=0.2$ , frequency penalty $=0.3$ , max completion tokens $=4096$ ; top- $p$ /API seed not passed; pipeline repetitions $0$ – $4$ .	2026-04-06–04-09

Appendix

Table G.1: Model, API, and decoding configuration used in the reported evaluations. “Not passed” means that the parameter was omitted from the API request; “pipeline seed” denotes repeated evaluation state and is not an API generation seed.

Figure H.1: Overview of the BrowserUse target workflow and CIPL instantiation.

Target	Provider	RN	EN	EE	CER	AER	ExecErr
browser_use (final_result)	DeepSeek	29	$9.4\pm 2.0$	$0.16\pm 0.03$	$0.21\pm 0.05$	$0.25\pm 0.06$	$0.0\pm 0.0$
browser_use (final_result)	GPT-4o	29	$8.4\pm 1.5$	$0.14\pm 0.03$	$0.18\pm 0.04$	$0.22\pm 0.05$	$0.0\pm 0.0$
browser_use (final_result)	MiniMax-M2.5	29	$24.6\pm 1.8$	$0.41\pm 0.03$	$0.65\pm 0.05$	$0.80\pm 0.04$	$0.0\pm 0.0$
browser_use (final_result)	MiniMax-M2.7	29	$20.4\pm 1.6$	$0.34\pm 0.03$	$0.52\pm 0.06$	$0.68\pm 0.05$	$0.0\pm 0.0$
browser_use (final_result)	qwen3.5-plus	29	$22.0\pm 1.4$	$0.37\pm 0.02$	$0.59\pm 0.05$	$0.76\pm 0.04$	$0.0\pm 0.0$
browser_use (process)	DeepSeek	30	$27.2\pm 1.2$	$0.45\pm 0.02$	$0.75\pm 0.03$	$0.86\pm 0.03$	$0.0\pm 0.0$

Appendix

Table H.1: Main Results for BrowserUse. Intermediate execution processes ( process ) demonstrate severe leakage across multiple models. Notably, for models like DeepSeek and GPT-4o, process leakage significantly exceeds final result leakage, confirming that observable execution artifacts in live agents pose a substantial privacy risk even when the final output appears secure.

Mode	Provider	RN	EN	EE	CER	AER	ExecErr
final_result	DeepSeek	31	$0.0\pm 0.0$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.0\pm 0.0$
final_result	GPT-4o	31	$0.0\pm 0.0$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.0\pm 0.0$
final_result	MiniMax-M2.5	31	$0.8\pm 0.8$	$0.01\pm 0.01$	$0.01\pm 0.01$	$0.02\pm 0.02$	$0.0\pm 0.0$
final_result	MiniMax-M2.7	31	$0.4\pm 0.5$	$0.01\pm 0.01$	$0.00\pm 0.00$	$0.01\pm 0.01$	$0.0\pm 0.0$
final_result	qwen3.5-plus	31	$1.2\pm 1.2$	$0.02\pm 0.02$	$0.01\pm 0.02$	$0.03\pm 0.02$	$0.0\pm 0.0$
process	DeepSeek	38	$2.0\pm 0.0$	$0.03\pm 0.00$	$0.02\pm 0.02$	$0.05\pm 0.02$	$0.0\pm 0.0$

Appendix

Table H.2: Weak-Control Results for BrowserUse. Once explicit extraction cues are removed, leakage is sharply suppressed to near-zero levels across both observation modes ( final_result and process ) and all evaluated models. This confirms that severe privacy risks in live agents are heavily gated by task-oriented extraction prompts.

Ablation	Setting	Provider	RN	EN	EE	CER	AER	ExecErr
k Ablation	k=1	DeepSeek	18	$14.7\pm 0.9$	$0.49\pm 0.03$	$0.66\pm 0.06$	$0.66\pm 0.06$	$0.0\pm 0.0$
k Ablation	k=1	qwen3.5-plus	18	$12.7\pm 0.9$	$0.42\pm 0.03$	$0.56\pm 0.04$	$0.56\pm 0.04$	$0.0\pm 0.0$
k Ablation	k=2	DeepSeek	30	$25.3\pm 1.3$	$0.42\pm 0.02$	$0.69\pm 0.06$	$0.86\pm 0.02$	$0.0\pm 0.0$
k Ablation	k=2	qwen3.5-plus	30	$18.3\pm 1.9$	$0.31\pm 0.03$	$0.37\pm 0.03$	$0.53\pm 0.03$	$0.0\pm 0.0$
k Ablation	k=4	DeepSeek	46	$44.0\pm 0.8$	$0.37\pm 0.01$	$0.80\pm 0.05$	$0.89\pm 0.02$	$0.0\pm 0.0$
k Ablation	k=4	qwen3.5-plus	46	$30.7\pm 0.9$	$0.26\pm 0.01$	$0.38\pm 0.07$	$0.59\pm 0.06$	$0.0\pm 0.0$

Appendix

Table H.3: Retrieval Depth and Method Ablation for BrowserUse (using three seeds 0,1,2). Increasing

k

exposes more units (RN), but its effect on Complete Extraction Rate (CER) varies by model: DeepSeek’s CER increases with greater exposure, whereas Qwen’s CER decreases, indicating that more retrieved context does not universally yield stronger complete leakage.

Human-audit sample size	120
Human raw agreement	0.8917
Human Cohen’s kappa	0.4598
Human disagreements adjudicated	13
Assessor–human agreement	0.9250
Assessor–human Cohen’s kappa	0.3647
Assessor precision (LEAKED)	0.600

Appendix

Table I.1: Human validation of historical PrivacyInAction runtime-assessor labels.

Mode	Provider	RN	EN	EE	CER	AER	ExecErr
audit_deception	MiniMax-M2.5	154	$5.4\pm 1.7$	$0.18\pm 0.06$	$0.00\pm 0.00$	$0.17\pm 0.05$	$0.0\pm 0.0$
audit_deception	MiniMax-M2.7	154	$5.2\pm 1.0$	$0.17\pm 0.03$	$0.00\pm 0.00$	$0.13\pm 0.04$	$0.0\pm 0.0$
audit_deception	qwen3.5-plus	154	$10.6\pm 2.1$	$0.35\pm 0.07$	$0.00\pm 0.00$	$0.28\pm 0.05$	$0.0\pm 0.0$
audit_bypass	MiniMax-M2.5	154	$75.8\pm 4.8$	$2.53\pm 0.16$	$0.10\pm 0.02$	$0.92\pm 0.02$	$0.0\pm 0.0$
audit_bypass	MiniMax-M2.7	154	$62.8\pm 4.6$	$2.09\pm 0.15$	$0.13\pm 0.04$	$0.85\pm 0.03$	$0.0\pm 0.0$
audit_bypass	qwen3.5-plus	154	$37.8\pm 5.8$	$1.26\pm 0.19$	$0.03\pm 0.02$	$0.67\pm 0.05$	$0.0\pm 0.0$

Appendix

Table I.2: Main results for the MCP agent protected by PrivacyInAction under two defense-aware attack-condition variants: audit_deception and audit_bypass . Shifting leakage realization from a monitored tool path to an unmonitored attacker-visible output channel ( audit_bypass ) yields substantially higher leakage across EN, EE, CER, and AER. For the PrivacyInAction keyword-level target, each trial corresponds to one task item, so EE is computed as recovered privacy keywords per query.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 272 papers

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
May 29, 2026Mingxuan Zhang, Jiahui Han, Dadi Guo +5Data Leakage Privacy Auditing
Shanghai Artificial Intelligence Laboratory · Southeast University
AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents
Oct 13, 2026Chengquan Guo, Zhaorun Chen, Zinan Lin +2Privacy Protection for LLM Agents Contextual Integrity
University of Chicago · Microsoft Research · University of Illinois Urbana-Champaign
Loose Lips: Understanding Privacy Leakage in Agentic Dialogue
Oct 7, 2026Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi +4Privacy Leakage in Language Models LLM Agent Security
Mila – Quebec AI Institute · McGill University · Simon Fraser University +3

May 29, 2026J/K move · Enter open · S save

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Mingxuan Zhang, Jiahui Han, Dadi Guo, Songze Li+4

Shanghai Artificial Intelligence Laboratory · Southeast University

Data Leakage Privacy Auditing AI Agent Auditing LLM Agent Security Privacy Leakage in Language Models AI Agent Security Benchmarks+1

Open paper AskPDF arXiv

LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce PrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with

1{,}182

cases across

7

acquisition behaviours and

16

application domains. Specifically, Acquisition Inspection examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. Probe Elicitation then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.

Abstract

Privacy Leakage in Language Models

Explore similar work

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue