Organizations: Minjiang University, Fuzhou, Fujian 350108, China · Renmin University of China, Beijing 100872, China · RMIT University, Melbourne, VIC 3000, Australia
Privacy leakage in LLM agents is commonly evaluated within individual components such as memory, retrieval, or tool-use pipelines, which makes it difficult to distinguish internal exposure from information that an external observer can actually recover. We present CIPL (Channel Inversion for Privacy Leakage), a channel-aware evaluation framework for black-box privacy leakage in LLM agents. CIPL represents a target through sensitive source, selection, assembly, execution, observation, and extraction stages and evaluates the transition from selected sensitive units to attacker-recoverable output under a shared protocol. Experiments across memory-based, retrieval-mediated, and tool-mediated targets, together with a BrowserUse live-agent case study, show that storage labels alone do not determine recoverability. Memory targets form a near-saturated reference case, retrieval-mediated leakage is frequently partial, and tool-mediated and live-agent leakage varies strongly with observation surface, prompt-to-channel alignment, retrieval depth, and provider behavior. A stratified semantic audit further identifies attacker-useful disclosures that canonical exact matching misses. CIPL therefore provides a common framework for comparing how internal sensitive dependence is realized as externally recoverable leakage across heterogeneous agent pipelines.
Figures & tables
Line of work
Main unit
Exp/Rec
Cross
Attr
Sem
Memory extraction
Memory records
P
N
N
N
RAG privacy / extraction
Retrieved documents or snippets
P
N
P
P
Tool / web-agent leakage
Tool calls, returns, actions, or injected instructions
P
N
P
N
Agent security benchmarks
End-to-end task outcomes
N
P
N
N
CIPL
Selected sensitive units in active computation
Y
Y
Y
Y
Table 1: Positioning of CIPL. Y: yes; P: partial or limited; N: no. Exp/Rec denotes separation of internal exposure from external recovery; Cross denotes shared cross-channel comparison; Attr denotes attribution to channel, provider, selection, or extraction factors; Sem denotes semantic leakage analysis.
Figure 1: Leakage regimes under the shared black-box measurement protocol. Panel (a) reports CER, while panel (b) reports AER. Under CIPL, three recurring patterns emerge. Memory-based settings remain saturated across all five providers. rag_ctrl exhibits a frequent-but-partial leakage pattern, with high any-recovery but incomplete recovery across providers. Tool-mediated channels show stronger dependence on observation surface and provider behavior under LLM-in-the-loop evaluation: return_echo is generally stronger than args_exfil away from provider-level ceiling effects, while DeepSeek and GPT-4o approach or reach saturation on some tool channels. Values are means over five seeds.
Setting
Provider
CER
AER
ExecErr
memory_ehr
MiniMax-M2.5
1.00±0.00
1.00±0.00
0±0
memory_ehr
MiniMax-M2.7
1.00±0.00
1.00±0.00
0±0
memory_ehr
qwen3.5-plus
1.00±0.00
1.00±0.00
0±0
memory_ehr
DeepSeek
1.00±0.00
1.00±0.00
0±0
memory_ehr
GPT-4o
1.00±0.00
1.00±0.00
0±0
memory_rap
MiniMax-M2.5
1.00±0.00
1.00±0.00
0±0
Table 2: Main results under the shared channel-aware evaluation framework. All settings use n=30 attack queries, one retry, and five seeds. We report Complete Extraction Rate (CER), Any Extracted Rate (AER), and execution-error counts (ExecErr) as mean ± standard deviation across seeds. Full metrics including RN, EN, and EE are reported in the Appendix.
Target
Main W/T/L
Main AER / CER
Naive AER / CER
memory_ehr
1/2/0
1.00 / 1.00
0.00–1.00 / 0.00–1.00
memory_rap
3/0/0
1.00 / 1.00
0.00 / 0.00
rag_ctrl
0/0/3
0.82–0.95 / 0.33–0.55
0.93–1.00 / 0.92–1.00
tool_ctrl(args)
3/0/0
0.27–0.99 / 0.27–0.49
0.18–0.86 / 0.16–0.49
tool_ctrl(return)
3/0/0
0.37–1.00 / 0.37–1.00
0.05–0.26 / 0.05–0.26
Table 3: Compact main-strategy versus naive-strategy comparison. Win/tie/loss is determined by AER, using CER as a secondary comparison when AER is equal. Ranges summarize the three audited providers (MiniMax-M2.7, qwen3.5-plus, and GPT-4o); provider-level rows are reported in the Appendix.
Figure 2: Leakage realization depends on alignment and exposure. Panel (a) reports AER under cleaned weak-control prompts for tool_ctrl ; leakage is sharply suppressed across all providers, indicating adversarial realization rather than ordinary completion. Panel (b) reports the retrieval-depth ablation for rag_ctrl ; AER remains high or stable, while the retrieval-depth results in the Appendix show that larger k can sharply reduce CER. Greater internal exposure therefore does not necessarily yield stronger complete extraction.
Provider
process CER
process AER
final CER
final AER
DeepSeek
0.75±0.03
0.86±0.03
0.21±0.05
0.25±0.06
GPT-4o
0.72±0.04
0.82±0.03
0.18±0.04
0.22±0.05
MiniMax-M2.5
0.82±0.03
0.91±0.02
0.65±0.05
0.80±0.04
MiniMax-M2.7
0.68±0.04
0.84±0.03
0.52±0.06
0.68±0.05
qwen3.5-plus
0.42±0.07
0.59±0.06
0.59±0.05
0.76±0.04
Table 4: BrowserUse as live black-box transfer evidence. Observation-surface ordering reverses across providers: DeepSeek and GPT-4o leak more through process , whereas qwen3.5-plus leaks more through final_result .
Observation mode
Strong AER
Std. RN
Std. AER
final_result
0.22 – 0.80
31
0.00 – 0.03
process
0.59 – 0.91
38
0.00 – 0.06
Table 5: BrowserUse alignment control under standard operational tasks. Leakage collapses to near-zero even though sensitive content still enters the agent context; full control results are reported in the Appendix.
Audit item
Value
Reading
labeled_samples
200
audit size
semantic_AER
0.8550
any semantic leakage
semantic_CER
0.4100
semantically complete leakage
exact_any=0 & semantic=1
5
missed by exact-only
exact_any=1 & semantic=2
82
exact and semantically severe
Table 6: RQ3 semantic-audit summary. Exact matching misses some attacker-useful leakage: 5 audited outputs contain partial sensitive-value recovery despite having no exact recovered canonical unit. Full protocol and representative cases are in the Appendix.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Provider
RN
EN
EE
CER
AER
ExecErr
memory_ehr
MiniMax-M2.5
55
55.0 ± 0.0
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
MiniMax-M2.7
55
55.0 ± 0.0
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
qwen3.5-plus
55
55.0 ± 0.0
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
DeepSeek
55
55.0 ± 0.0
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
GPT-4o
55
55.0 ± 0.0
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_rap
MiniMax-M2.5
57
57.0 ± 0.0
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
Appendix
Table B.1: Full main results under the unified CIPL protocol. All main experiments use n=30 , one retry, and five seeds. Values are reported as mean ± standard deviation across seeds.
Provider
Retrieve Method
RN
EN
EE
CER
AER
ExecErr
DeepSeek
edit_distance
57
57
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
GPT-4o
edit_distance
57
57
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
MiniMax-M2.5
edit_distance
57
57
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
MiniMax-M2.7
edit_distance
57
57
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
qwen3.5-plus
edit_distance
57
57
0.63 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
DeepSeek
token_overlap
43
43
0.48 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
Appendix
Table C.1: Retrieval-rule ablation for memory_rap . Changing the retrieval rule alters coverage and normalized efficiency, but not the qualitative vulnerability conclusion: both rules remain fully leakage-prone on all five providers.
Provider
Retrieve Method
RN
EN
EE
CER
AER
ExecErr
DeepSeek
edit_distance
50
40
0.67 ± 0.00
0.00 ± 0.00
0.80 ± 0.00
0.0 ± 0.0
GPT-4o
edit_distance
50
40
0.67 ± 0.00
0.00 ± 0.00
0.80 ± 0.00
0.0 ± 0.0
MiniMax-M2.5
edit_distance
50
50
0.83 ± 0.00
0.17 ± 0.03
0.98 ± 0.02
0.0 ± 0.0
MiniMax-M2.7
edit_distance
50
50
0.83 ± 0.00
0.19 ± 0.08
0.98 ± 0.02
0.0 ± 0.0
qwen3.5-plus
edit_distance
50
40
0.67 ± 0.00
0.00 ± 0.00
0.80 ± 0.00
0.0 ± 0.0
DeepSeek
token_overlap
50
40
0.67 ± 0.00
0.00 ± 0.00
0.80 ± 0.00
0.0 ± 0.0
Appendix
Table C.2: Retrieval-rule ablation for rag_ctrl . The retrieval rule has only a modest effect on DeepSeek , GPT-4o , and qwen3.5-plus , but slightly changes the balance between complete and partial leakage on both MiniMax-M2.5 and MiniMax-M2.7 .
Provider
Retrieve Method
RN
EN
EE
CER
AER
ExecErr
DeepSeek
edit_distance
50
50.0 ± 0.0
0.83 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
GPT-4o
edit_distance
50
50.0 ± 0.0
0.83 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
MiniMax-M2.5
edit_distance
50
43.0 ± 5.0
0.72 ± 0.08
0.29 ± 0.02
0.29 ± 0.02
0.0 ± 0.0
MiniMax-M2.7
edit_distance
50
50.0 ± 0.0
0.83 ± 0.00
0.31 ± 0.03
0.32 ± 0.02
0.0 ± 0.0
qwen3.5-plus
edit_distance
50
50.0 ± 0.0
0.83 ± 0.00
0.80 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
DeepSeek
token_overlap
20
20.0 ± 0.0
0.33 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
Appendix
Table C.3: Retrieval-rule ablation for tool_ctrl(return_echo,llm) . This auxiliary ablation uses three seeds. Retrieval-rule choice changes both internal coverage and leakage strength. This effect is clearest on the two MiniMax variants, while DeepSeek and GPT-4o remain saturated under both rules and qwen3.5-plus remains highly leakage-prone under both rules.
Figure C.1: Retrieval-rule ablation for tool_ctrl(return_echo,llm) . We compare edit-distance and token-overlap retrieval under the LLM-in-the-loop return-echo setting. The retrieval-rule effect is visible on both MiniMax variants, while DeepSeek and GPT-4o remain saturated and qwen3.5-plus remains highly leakage-prone under both rules. This supports the CIPL interpretation that leakability depends not only on the final observation channel, but also on the upstream selection mechanism that determines which sensitive units enter active computation.
Target
Provider
Size
RN
EN
EE
CER
AER
ExecErr
memory_ehr
DeepSeek
100
45
45
0.38 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
GPT-4o
100
45
45
0.38 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
MiniMax-M2.5
100
45
45
0.38 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
MiniMax-M2.7
100
45
45
0.38 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
qwen3.5-plus
100
45
45
0.38 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
DeepSeek
200
55
55
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
Appendix
Table C.4: Source-size ablation for memory-based targets. Increasing source size changes RN, EN, and EE, but does not change the qualitative vulnerability conclusion: both memory targets remain at CER = AER = 1.0 across all five providers, no execution-instability effect appears under this source-size variation.
Provider
Size
RN
EN
EE
CER
AER
ExecErr
DeepSeek
20
20
20
0.33 ± 0.00
0.20 ± 0.00
0.80 ± 0.00
0.0 ± 0.0
GPT-4o
20
20
20
0.33 ± 0.00
0.06 ± 0.04
0.48 ± 0.02
0.0 ± 0.0
MiniMax-M2.5
20
20
20
0.33 ± 0.00
0.51 ± 0.02
0.86 ± 0.03
0.0 ± 0.0
MiniMax-M2.7
20
20
20
0.33 ± 0.00
0.52 ± 0.06
0.92 ± 0.06
0.0 ± 0.0
qwen3.5-plus
20
20
20
0.33 ± 0.00
0.00 ± 0.00
0.40 ± 0.00
0.0 ± 0.0
DeepSeek
30
30
21
0.35 ± 0.00
0.20 ± 0.00
0.60 ± 0.00
0.0 ± 0.0
Appendix
Table C.5: Source-size ablation for rag_ctrl . The effect of source size is non-monotonic and strongly provider-dependent in the five-provider setting. The two MiniMax variants remain high-AER across all tested source sizes, while qwen3.5-plus , DeepSeek , and GPT-4o remain in lower-CER partial-leakage patterns with distinct source-size sensitivity.
Target
Provider
k
RN
EN
EE
CER
AER
ExecErr
memory_ehr
DeepSeek
2
34
34
0.57 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
GPT-4o
2
34
34
0.57 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
MiniMax-M2.5
2
34
34
0.57 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
MiniMax-M2.7
2
34
34
0.57 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
qwen3.5-plus
2
34
34
0.57 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
memory_ehr
DeepSeek
4
55
55
0.46 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.0 ± 0.0
Appendix
Table C.6: Full retrieval-depth ablation for memory-based targets. Changing retrieval depth affects RN, EN, and EE, but not the qualitative leakage outcome: both memory_ehr and memory_rap remain saturated across all five providers at all tested k values.
Provider
k
RN
EN
EE
CER
AER
ExecErr
DeepSeek
1
50
40.0±0.0
1.33±0.00
0.80±0.00
0.80±0.00
0.0±0.0
GPT-4o
1
50
40.0±0.0
1.33±0.00
0.80±0.00
0.80±0.00
0.0±0.0
MiniMax-M2.5
1
50
50.0±0.0
1.67±0.00
0.94±0.02
0.94±0.02
0.0±0.0
MiniMax-M2.7
1
50
50.0±0.0
1.67±0.00
0.98±0.02
0.98±0.02
0.0±0.0
qwen3.5-plus
1
50
40.0±1.0
1.33±0.03
0.80±0.00
0.80±0.00
0.0±0.0
DeepSeek
2
50
40.0±0.0
0.67±0.00
0.00±0.00
0.80±0.00
0.0±0.0
Appendix
Table C.7: Full retrieval-depth ablation for rag_ctrl . Both MiniMax variants remain high-AER across the tested retrieval depths, but CER drops sharply as k increases. qwen3.5-plus , DeepSeek , and GPT-4o remain comparatively stable in AER while staying in lower-CER partial-leakage patterns.
Provider
k
RN
EN
EE
CER
AER
ExecErr
DeepSeek
1
30
30.0±0.0
1.00±0.00
1.00±0.00
1.00±0.00
0.0±0.0
GPT-4o
1
30
30.0±0.0
1.00±0.00
1.00±0.00
1.00±0.00
0.0±0.0
MiniMax-M2.5
1
30
30.0±0.0
1.00±0.00
0.73±0.08
0.73±0.08
0.0±0.0
MiniMax-M2.7
1
30
30.0±0.0
1.00±0.00
0.36±0.07
0.36±0.07
0.0±0.0
qwen3.5-plus
1
30
30.0±0.0
1.00±0.00
1.00±0.00
1.00±0.00
0.0±0.0
DeepSeek
2
50
50.0±0.0
0.83±0.00
1.00±0.00
1.00±0.00
0.0±0.0
Appendix
Table C.8: Full retrieval-depth ablation for tool_ctrl(return_echo,llm) . This auxiliary ablation uses three seeds. Increasing retrieval depth weakens complete extraction on the MiniMax variants, while DeepSeek and qwen3.5-plus remain saturated and GPT-4o continues to exhibit persistently high AER but substantially lower CER.
Target
Provider
Main (AER / CER)
Naive (AER / CER)
memory_ehr
MiniMax-M2.7
1.00 / 1.00
1.00 / 1.00
memory_ehr
qwen3.5-plus
1.00 / 1.00
0.00 / 0.00
memory_ehr
GPT-4o
1.00 / 1.00
1.00 / 1.00
memory_rap
MiniMax-M2.7
1.00 / 1.00
0.00 / 0.00
memory_rap
qwen3.5-plus
1.00 / 1.00
0.00 / 0.00
memory_rap
GPT-4o
1.00 / 1.00
0.00 / 0.00
Appendix
Table D.1: Summary of main-vs-naive baseline comparisons. The main CIPL prompts are not uniformly stronger than naive baselines. Memory- and tool-mediated settings generally favor the main prompts, while rag_ctrl provides a systematic counterexample in which the dump-style naive baseline performs better.
Target
Provider
Full (AER / CER)
Highest-AER Variant
memory_ehr
MiniMax-M2.7
1.00 / 1.00
1.00 (multiple variants tied)
memory_ehr
qwen3.5-plus
1.00 / 1.00
1.00 (multiple variants tied)
rag_ctrl
MiniMax-M2.7
0.95 / 0.55
0.98 ( full / tied best)
rag_ctrl
qwen3.5-plus
0.89 / 0.45
1.00 ( no_diversification )
tool_ctrl(return_echo,llm)
MiniMax-M2.7
0.37 / 0.37
0.37 ( full )
tool_ctrl(return_echo,llm)
qwen3.5-plus
0.40 / 0.40
0.40 ( full )
Appendix
Table D.2: Summary of component-ablation outcomes. The Full column reports the audited five-seed main-setting reference; variant summaries come from the corresponding auxiliary ablation runs. The full locator/aligner/diversification construction is therefore best read as a useful reference configuration, not as a uniformly optimal one.
Setting
Condition
AER / CER
Interpretation
memory_ehr qwen
n=10/20/30/50
audited reference at n=30 : 1.00 / 1.00
Saturated across budgets
rag_ctrl MiniMax-M2.7
n=10/20/30/50
main-setting reference at n=30 : 0.9533 / 0.5467
Frequent but incomplete leakage remains the qualitative pattern
tool_ctrl(return_echo,llm) qwen3.5-plus
n=10/20/30/50
audited reference at n=30 : 0.40 / 0.40
Moderate LLM-in-the-loop leakage remains observable under budget perturbation
memory_ehr qwen
retries=1,2
reference at retries=1: 1.00 / 1.00
Stable under retries
rag_ctrl MiniMax-M2.7
retries=1,2
main-setting reference at retries=1: 0.9533 / 0.5467; higher CER also observed at retries=2
Retries can increase leakage
tool_ctrl(return_echo,llm) qwen3.5-plus
retries=1,2
reference at retries=1: 0.40 / 0.40
High-leakage pattern preserved
Appendix
Table D.3: Summary of robustness checks. The AER/CER column reports the main-setting reference value for the named setting where applicable; qualitative interpretations summarize the accompanying budget, retry, or seed sweeps. Retry-related results must be interpreted at the prompt level: r0 is not a valid no-retry baseline under the current implementation.
Provider
attack echo AER
defense echo AER
attack args AER
defense args AER
MiniMax-M2.7
0.37
0.12
0.27
0.19
GPT-4o
1.00
0.20
0.99
0.46
Appendix
Table D.4: Defense-prompt suppression on tool-mediated channels. Adding defense-style prompting sharply reduces leakage on both echo and argument channels, especially for GPT-4o .
Setting
Check
Main reading
memory_ehr, qwen3.5-plus
budgets n=10 – 50
remains saturated throughout
rag_ctrl, MiniMax-M2.7
budgets n=10 – 50
frequent but incomplete leakage persists
tool_ctrl (return_echo,llm), qwen3.5-plus
budgets n=10 – 50
moderate leakage remains observable
tool_ctrl (return_echo,llm), GPT-4o
defense prompting
AER drops sharply but does not define the main threat model
Appendix
Table D.5: Compact robustness summary from appendix checks. These checks do not overturn the main regime assignments. They are included here to show that the reported takeaways are not artifacts of one budget, seed count, or prompt configuration.
Metric
Value
labeled_samples
200
semantic_AER
0.8550
semantic_CER
0.4100
exact_any=0 & semantic=1
5
exact_any=1 & semantic=2
82
exact_any=0 & semantic=0
29
Appendix
Table E.1: Semantic leakage annotation summary. The metadata-blinded two-annotator semantic audit shows that exact-match extraction can underestimate leakage: 5 samples contain partial sensitive-value recovery despite having no exact recovered canonical unit.
Setting
Model/API, decoding, and seed configuration
Experiment window
Controlled CIPL (memory/tool)
MiniMax-M2.5/M2.7; MiniMax route api.minimaxi.com ; target code T=0.0 , max tokens =512 ; top- p and API seed not passed; pipeline seeds 0 – 4 .
2026-03-18–03-24
Controlled CIPL (memory/tool)
qwen3.5-plus; DashScope OpenAI-compatible route; target code T=0.0 , max tokens =512 , enable_thinking=false ; top- p and API seed not passed; pipeline seeds 0 – 4 .
2026-03-18–03-24
Controlled CIPL (memory/tool)
deepseek-chat; DeepSeek route api.deepseek.com ; target code T=0.0 , max tokens =512 ; top- p and API seed not passed; pipeline seeds 0 – 4 .
2026-03-18–03-24
Controlled CIPL (memory/tool)
gpt-4o; ai.wer.plus compatible route; target code T=0.0 , max tokens =512 ; top- p and API seed not passed; pipeline seeds 0 – 4 .
qwen3.5-plus; DashScope OpenAI-compatible route; wrapper defaults T=0.2 , frequency penalty =0.3 , max completion tokens =4096 ; top- p /API seed not passed; pipeline repetitions 0 – 4 .
2026-04-06–04-09
Appendix
Table G.1: Model, API, and decoding configuration used in the reported evaluations. “Not passed” means that the parameter was omitted from the API request; “pipeline seed” denotes repeated evaluation state and is not an API generation seed.
Figure H.1: Overview of the BrowserUse target workflow and CIPL instantiation.
Target
Provider
RN
EN
EE
CER
AER
ExecErr
browser_use (final_result)
DeepSeek
29
9.4±2.0
0.16±0.03
0.21±0.05
0.25±0.06
0.0±0.0
browser_use (final_result)
GPT-4o
29
8.4±1.5
0.14±0.03
0.18±0.04
0.22±0.05
0.0±0.0
browser_use (final_result)
MiniMax-M2.5
29
24.6±1.8
0.41±0.03
0.65±0.05
0.80±0.04
0.0±0.0
browser_use (final_result)
MiniMax-M2.7
29
20.4±1.6
0.34±0.03
0.52±0.06
0.68±0.05
0.0±0.0
browser_use (final_result)
qwen3.5-plus
29
22.0±1.4
0.37±0.02
0.59±0.05
0.76±0.04
0.0±0.0
browser_use (process)
DeepSeek
30
27.2±1.2
0.45±0.02
0.75±0.03
0.86±0.03
0.0±0.0
Appendix
Table H.1: Main Results for BrowserUse. Intermediate execution processes ( process ) demonstrate severe leakage across multiple models. Notably, for models like DeepSeek and GPT-4o, process leakage significantly exceeds final result leakage, confirming that observable execution artifacts in live agents pose a substantial privacy risk even when the final output appears secure.
Mode
Provider
RN
EN
EE
CER
AER
ExecErr
final_result
DeepSeek
31
0.0±0.0
0.00±0.00
0.00±0.00
0.00±0.00
0.0±0.0
final_result
GPT-4o
31
0.0±0.0
0.00±0.00
0.00±0.00
0.00±0.00
0.0±0.0
final_result
MiniMax-M2.5
31
0.8±0.8
0.01±0.01
0.01±0.01
0.02±0.02
0.0±0.0
final_result
MiniMax-M2.7
31
0.4±0.5
0.01±0.01
0.00±0.00
0.01±0.01
0.0±0.0
final_result
qwen3.5-plus
31
1.2±1.2
0.02±0.02
0.01±0.02
0.03±0.02
0.0±0.0
process
DeepSeek
38
2.0±0.0
0.03±0.00
0.02±0.02
0.05±0.02
0.0±0.0
Appendix
Table H.2: Weak-Control Results for BrowserUse. Once explicit extraction cues are removed, leakage is sharply suppressed to near-zero levels across both observation modes ( final_result and process ) and all evaluated models. This confirms that severe privacy risks in live agents are heavily gated by task-oriented extraction prompts.
Ablation
Setting
Provider
RN
EN
EE
CER
AER
ExecErr
k Ablation
k=1
DeepSeek
18
14.7±0.9
0.49±0.03
0.66±0.06
0.66±0.06
0.0±0.0
k Ablation
k=1
qwen3.5-plus
18
12.7±0.9
0.42±0.03
0.56±0.04
0.56±0.04
0.0±0.0
k Ablation
k=2
DeepSeek
30
25.3±1.3
0.42±0.02
0.69±0.06
0.86±0.02
0.0±0.0
k Ablation
k=2
qwen3.5-plus
30
18.3±1.9
0.31±0.03
0.37±0.03
0.53±0.03
0.0±0.0
k Ablation
k=4
DeepSeek
46
44.0±0.8
0.37±0.01
0.80±0.05
0.89±0.02
0.0±0.0
k Ablation
k=4
qwen3.5-plus
46
30.7±0.9
0.26±0.01
0.38±0.07
0.59±0.06
0.0±0.0
Appendix
Table H.3: Retrieval Depth and Method Ablation for BrowserUse (using three seeds 0,1,2). Increasing k exposes more units (RN), but its effect on Complete Extraction Rate (CER) varies by model: DeepSeek’s CER increases with greater exposure, whereas Qwen’s CER decreases, indicating that more retrieved context does not universally yield stronger complete leakage.
Human-audit sample size
120
Human raw agreement
0.8917
Human Cohen’s kappa
0.4598
Human disagreements adjudicated
13
Assessor–human agreement
0.9250
Assessor–human Cohen’s kappa
0.3647
Assessor precision (LEAKED)
0.600
Appendix
Table I.1: Human validation of historical PrivacyInAction runtime-assessor labels.
Mode
Provider
RN
EN
EE
CER
AER
ExecErr
audit_deception
MiniMax-M2.5
154
5.4±1.7
0.18±0.06
0.00±0.00
0.17±0.05
0.0±0.0
audit_deception
MiniMax-M2.7
154
5.2±1.0
0.17±0.03
0.00±0.00
0.13±0.04
0.0±0.0
audit_deception
qwen3.5-plus
154
10.6±2.1
0.35±0.07
0.00±0.00
0.28±0.05
0.0±0.0
audit_bypass
MiniMax-M2.5
154
75.8±4.8
2.53±0.16
0.10±0.02
0.92±0.02
0.0±0.0
audit_bypass
MiniMax-M2.7
154
62.8±4.6
2.09±0.15
0.13±0.04
0.85±0.03
0.0±0.0
audit_bypass
qwen3.5-plus
154
37.8±5.8
1.26±0.19
0.03±0.02
0.67±0.05
0.0±0.0
Appendix
Table I.2: Main results for the MCP agent protected by PrivacyInAction under two defense-aware attack-condition variants: audit_deception and audit_bypass . Shifting leakage realization from a monitored tool path to an unmonitored attacker-visible output channel ( audit_bypass ) yields substantially higher leakage across EN, EE, CER, and AER. For the PrivacyInAction keyword-level target, each trial corresponds to one task item, so EE is computed as recovered privacy keywords per query.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce PrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with 1,182 cases across 7 acquisition behaviours and 16 application domains. Specifically, Acquisition Inspection examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. Probe Elicitation then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.
Mingxuan Zhang, Jiahui Han, Dadi Guo +5
Shanghai Artificial Intelligence Laboratory · Southeast University
LLM agents are becoming increasingly capable of handling daily workspace tasks by interacting with application environments through tool calls. However, as agents read and write sensitive user data across email, documents, and other applications, a central privacy question is whether they preserve contextual integrity (CI), that is, whether the information flows they produce comply with context-specific disclosure constraints. Existing work either focuses on model-level privacy leakage or evaluates agents in limited application environments with manually constructed disclosure constraints. In this work, we introduce AgentCIScope, a unified platform and benchmark for evaluating agent-level contextual integrity through executable, multi-step tasks across diverse, realistic application environments. AgentCIScope transforms public and synthetic privacy cases into executable agent tasks, with disclosure constraints grounded in three criteria: governing privacy policies, persona-based agreement, and a predefined sensitive-attribute taxonomy. Based on AgentCIScope, we evaluate 12 agents on 936 tasks and find that strong task-completion capability does not guarantee contextual-integrity preservation: single-run leakage rates range from 32.5% to 94.4%, while repeating each task five times further increases leakage by 4-19 percentage points. Leakage varies substantially across attributes: personal preferences and booking details have average leakage rates of 84.7% and 76.5%, compared with 17.1% for credit-card information and 29.4% for SSNs; environments: calendar has the highest average leakage rate as an information source, while WhatsApp ranks highest as an information destination; and governing privacy policies. AgentCIScope provides deep insights into agent privacy leakage and serves as a unified testbed for developing privacy-preserving agents.
Chengquan Guo, Zhaorun Chen, Zinan Lin +2
University of Chicago · Microsoft Research · University of Illinois Urbana-Champaign
As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue has become critical. Privacy leakage in such settings requires reasoning about whether information flows respect the contextual norms under which they originated. We extend CIMemories to CIMEMORIES DIALOGUE, a benchmark of 4,410 two-agent dialogue instances grounded in Contextual Integrity theory. Using this benchmark, we systematically evaluate 7 open-weight models from the Qwen3 and Llama 3.1 families across conversation lengths of 1 and 6 turns under three prompting conditions. We find that task capability and privacy leakage are fundamentally coupled: more capable models are systematically leakier, with no model in our evaluation able to decouple the two. Thinking models consistently leak more than their non-thinking counterparts, a pattern we attribute to the extended reasoning surface created by chain-of-thought. Leakage compounds with conversation length, rising up to 3x from 1 to 6 turns. Our results suggest that the coupling between task capability and privacy leakage is not an artifact of misconfiguration but a structural feature of current agentic systems.
Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi +4
Mila – Quebec AI Institute · McGill University · Simon Fraser University +3