PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents

Lucas Biechy; Cédric Eichler; Héber H. Arcolezi; Nicolas Anciaux

cs.CRSep 28, 2026

PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents

Authors: Lucas Biechy, Cédric Eichler, Héber H. Arcolezi, Nicolas Anciaux

Organizations: Petscraft, Inria · Université Paris-Saclay · INSA CVL · Université d’Orléans · LIFO · ÉTS Montréal

Abstract

While prior work has documented privacy failures in LLM agents, it remains unclear how the presentation of privacy guidance influences their choice of information sources. We introduce PrivacySkills, a controlled framework for evaluating how agents choose among acquisition pathways that provide the same task-relevant value: consulting publicly available personal information, accessing confidential sources, or interacting with the user. The evaluation framework comprises 55 synthetic tasks spanning 11 categories of personal information, with 169 associated skills that describe the available acquisition pathways. We consider privacy guidance through system-level instructions, skill-level metadata labels, or both. Separately, we vary user availability and urgency framing. With users available and no privacy guidance, agents access confidential sources in 30% of valid runs on average across five open-weight models, despite sufficient alternatives. This rate increases to 45% when users are unavailable, whereas urgency framing has no detectable effect. System-level privacy instructions alone have limited effects on confidential access, while skill-level intrusiveness labels produce a modest reduction (24% on average), but combining the two roughly halves confidential access. Our findings motivate incorporating privacy annotations into skill specifications and evaluating their effectiveness alongside system-level instructions.

Figures & tables

Figure 1: PrivacySkills . To book the ride, the agent must first obtain the airport pickup point through one of three acquisition skills: user interaction, a public source, or a confidential source. All three return the needed value (blue) and lead to the same booking, but the confidential source also exposes unrelated personal data (red). PrivacySkills measures which path agents take across 55 tasks spanning 11 categories of personal information.

Figure 2: Experimental design. Each run pairs one task with one of four privacy conditions, crossing intrusiveness labels and a privacy instruction. Only the dashed parts of the system prompt vary; a separate control adds a user-unavailable note. Runs cover all six orders of the three acquisition skills (user interaction, public source, confidential source), with 10 samples each.

Figure 3: Acquisition tiers distribution per model with no situational pressure (Baseline), under urgency, and under user unavailability.

Figure 4: Change in HIR relative to neutral guidance, per model and privacy-guidance condition, without situational pressure.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Runs	Valid (%)	ER (%)
Gemma 4 (26B)	12,072	95.9	3.6
GPT-OSS (120B)	12,579	95.1	0.5
Mistral Small 3.2	25,024	61.2	3.8
Nemotron Cascade 2 (30B)	14,406	94.7	1.0
Qwen3 (35B)	17,296	98.1	1.0
Total	81,377	85.4	2.2

Appendix

Table 1: Attrition and escalation, by model (conditions pooled), under the six settings (neutral, urgent, user unavailable, instruction, label, label + instruction). Valid and ER are shares of all runs.

Condition	Runs	Valid (%)	ER (%)
Baseline	14,179	87.1	2.7
Urgent	12,306	83.8	2.0
User unavailable	12,377	81.0	2.4
Instruction	13,953	89.6	2.3
Label	14,417	83.3	1.8
Label + instruction	14,145	86.9	2.0

Appendix

Table 2: Attrition and escalation, by condition (models pooled), under the six conditions (neutral, urgent, user unavailable, instruction, label, label + instruction). Valid and ER are shares of all runs.

Dataset category	GDPR	CCPA	LGPD	PIPL
Health	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Biometric / genetic	$\checkmark$	$\checkmark$	$\checkmark$	Biometrics
Religious belief	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Racial / ethnic origin	$\checkmark$	$\checkmark$	$\checkmark$	–
Sex life / orientation	$\checkmark$	$\checkmark$	Sex life	–
Political / union affiliation	$\checkmark$	Union membership	$\checkmark$	–

Appendix

Table 3: Legal coverage of the eleven dataset categories . A checkmark indicates explicit inclusion in the cited provision, subject to its conditions. Named entries identify coverage of part of a combined category. A dash means the type is not separately listed in that provision. It may still receive legal protection, including under PIPL’s open-ended definition.

Figure 5: Heatmap of HIR per data category and model under no privacy guidance and no situational pressure

Figure 6: Source choice under correct and inverted intrusiveness labels. Share of valid runs by acquisition tiers, under no situational pressure. All three panels include the privacy instruction. In the inverted panel, asking the user is labeled extreme and the confidential source is labeled minimal . The public source keeps the label low .

Figure 7: Label and score versions of the same ranking. Change in HIR compared with the neutral condition, computed as

\mathrm{HIR}(\text{neutral})-\mathrm{HIR}(\text{guidance})

, in percentage points Positive values mean fewer acquisitions from the confidential source. The label shows minimal , low , and extreme . The score shows 1/10 , 3/10 , and 10/10 . No situational pressure: non-urgent instances with the user available.

Explore similar work

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
May 29, 2026Mingxuan Zhang, Jiahui Han, Dadi Guo +5Privacy Attacker Large Language Model
Shanghai Artificial Intelligence Laboratory · Southeast University
PrivacyAlign: Contextual Privacy Alignment for LLM Agents
Jun 19, 2026Manveer Singh Tamber, Abhay Puri, Marc-Etienne Brunet +3Large Language Model Agents Privacy
University of Waterloo · †Work done during internship at ServiceNow AI Research · 2ServiceNow AI Research +2
Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents
Jun 25, 2026Nada Lahjouji, Ashwin Gerard ColacoPrivacy Large Language Model Agents
University of California, Irvine Irvine, California, USA

May 29, 2026, cs.CRJ/K move · Enter open · S save

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Mingxuan Zhang, Jiahui Han, Dadi Guo, Songze Li+4

Shanghai Artificial Intelligence Laboratory · Southeast University

Privacy Attacker Large Language Model Model Auditing Acquisition

Open paper AskPDF arXiv

LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce \emph{PrivacyPeek}, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with

1{,}182

cases across

7

acquisition behaviours and

16

application domains. Specifically, \emph{Acquisition Inspection} examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. \emph{Probe Elicitation} then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.

Abstract

Explore similar work

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents