InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Pengrun Huang; Yiwei Fu; Luoxi Meng; Kamalika Chaudhuri

cs.AIOct 25, 2026

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Authors: Pengrun Huang, Yiwei Fu, Luoxi Meng, Kamalika Chaudhuri

Organizations: UC San Diego · University of Illinois Urbana-Champaign

Abstract

Large language model (LLM)-based personal agents are increasingly deployed to handle everyday tasks on users' behalf. These agents have access to private user information, while their tasks involve communicating with third parties; this raises a critical privacy concern that agents may inappropriately reveal sensitive user information to these recipients. Existing benchmarks primarily assess direct privacy leakage, where an agent explicitly states a pre-specified secret or attribute. However, they do not capture inferential leakage, where the model leaves indirect cues in the reply that implicitly leaks sensitive attributes. We introduce InferPriv, a benchmark that measures whether an honest-but-curious recipient can infer a user's sensitive attributes from an LLM's reply on routine task. Evaluating five frontier models, we find that inferential leakage substantially exceeds direct leakage across models and prompts. Under a privacy-aware system prompt, GPT-6.1-Sol, Sonnet-5.5, and Gemini-3.8-Flash achieve near-zero direct leakage, yet the inferential leakage rate remains between 16.2--25.2%. These findings suggest that evaluating explicit leakage alone can substantially understate the privacy risks.

Figures & tables

Figure 1: Examples illustrating inferential leakage in LLM assistant’s replies. The reply addresses the user’s stated constraints (green) but also derives additional accommodations from sensitive memory statements using world knowledge. These additions allow recipients to infer sensitive attributes, even though the attributes are never explicitly stated (red). We consider this a privacy violation as the recipient learns information the user never chose to share. Dashed notes show alternative way in which needs could be handled privately.

Prompt	Model	Privacy leakage $\downarrow$			Complete $\uparrow$
Prompt	Model	Direct@5	Inferential 1 @5	Inferential 3 @5	Complete $\uparrow$
Baseline	GPT-6.1-Sol	10.3%	35.8%	40.8%	95.1%
	Sonnet-5.5	5.1%	23.5%	26.1%	90.9%
	Gemini-3.8-Flash	3.7%	16.6%	19.4%	92.0%
	DeepSeek-V4.1-Flash	34.0%	53.7%	55.7%	97.5%
	Muse-Spark-1.3	55.2%	59.7%	61.3%	97.1%

Table 1: Privacy leakage and task completeness across frontier models. We compare baseline and privacy-aware prompting. Privacy metrics are evaluated over 98 protected attribute–task pairs

(a,t)

using 10 user profiles, with five replies sampled per instance. Inferential leakage is adjusted using a control group without sensitive memories.

Attribute $a$	Direct-disclosure vocab $V_{a}$	Examples of inferential clues
Type 1 diabetes	diabetes, diabetic, type 1, T1D	carbohydrate counts; unsweetened drink; mini-fridge for temperature-sensitive medication; carrying insulin and CGM on board
Hypertension	hypertension, hypertensive, high blood pressure	reducing sodium; low(er)-sodium option; no added salt; go light on salt
Pregnancy	pregnant, pregnancy, prenatal, antenatal, expecting a baby(child)	fully cooked food; pasteurized; non-alcoholic drinks; aisle seat; extra legroom; least tiring option; restroom access; lower-back discomfort
Insomnia	insomnia, insomniac, insomniacs, insomniatic	quiet room; blackout curtains; away from elevators; own room; no caffeine after 3 PM; mentioned sleep poorly
IBS	IBS, irritable bowel syndrome	low-FODMAP food; no garlic or onion; avoid rye; sauces on the side; seat near a restroom; mentioned sensitive digestive system
Gout	gout, gouty, podagra	avoiding high-purine foods; cutting back on red meat and alcohol

Table 2: Representative cues in replies associated with attribute inference.

Figure 2: Comparison of per-cell inferential leakage between (a) GPT-6.1-Sol and (b) Sonnet-5.5 under the baseline prompt. Hatched cells mark pairs where the attribute is needed for the task; they are excluded from the averages in Table 1 . Values are calibrated against control groups without the sensitive memories, so a value is negative when the control group has a higher inferential leakage rate. Across the 98 protected pairs $(a,t)$ with $t\in\mathcal{T}^{\mathrm{priv}}_{a}$ , the leakage rates of the two models are strongly correlated (Pearson $r=0.71$ , Spearman $\rho=0.72$ ), indicating that they share a similar leakage pattern.

Prompt	Effort	Direct@5	Inferential 1 @5	Inferential 3 @5	Complete
Baseline	Low	18.1%	42.2%	45.8%	94.1%
	Medium	10.3%	35.8%	40.8%	95.1%
	High	4.9%	35.4%	39.8%	95.7%
Privacy-aware	Low	0.0%	24.8%	29.4%	93.4%
	Medium	0.0%	21.0%	25.2%	94.8%
	High	0.0%	23.9%	28.6%	95.8%

Table 3: Privacy leakage and completeness of GPT-6.1-Sol at three reasoning efforts. Medium is the setting used in Table 1 . Inferential k @5 is paired with the control group.

Prompt	Statements	Direct@5	Inferential 1 @5	Inferential 3 @5
Baseline	Declaration	16.2%	30.0%	32.6%
	Default record	10.3%	35.8%	40.8%
	Unnamed record	2.2%	30.3%	33.9%
Privacy-aware	Declaration	0.0%	14.0%	17.8%
	Default record	0.0%	21.0%	25.2%
	Unnamed record	0.0%	20.6%	25.3%

Table 4: Comparison of privacy leakage of GPT-6.1-Sol under three forms of the sensitive-attribute statements. All rows are scored on the same protected (profile, attribute, task) cells; Inferential k @5 is paired with the control group.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
May 29, 2026Mingxuan Zhang, Jiahui Han, Dadi Guo +5Data Leakage Privacy Auditing
Shanghai Artificial Intelligence Laboratory · Southeast University
Loose Lips: Understanding Privacy Leakage in Agentic Dialogue
Oct 7, 2026Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi +4Privacy Leakage in Language Models LLM Agent Security
Mila – Quebec AI Institute · McGill University · Simon Fraser University +3
Say Something Else: Rethinking Contextual Privacy as Information Sufficiency
Apr 7, 2026Yunze Xiao, Wenkai Li, Xiaoyuan Wu +3Privacy-Preserving Language Models Privacy Protection for LLM Agents
Carnegie Mellon University · Massachusetts Institute of Technology · 2077AI +1

May 29, 2026J/K move · Enter open · S save

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Mingxuan Zhang, Jiahui Han, Dadi Guo, Songze Li+4

Shanghai Artificial Intelligence Laboratory · Southeast University

Data Leakage Privacy Auditing AI Agent Auditing LLM Agent Security Privacy Leakage in Language Models AI Agent Security Benchmarks+1

Open paper AskPDF arXiv

LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce PrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with

1{,}182

cases across

7

acquisition behaviours and

16

application domains. Specifically, Acquisition Inspection examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. Probe Elicitation then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.

Abstract

Privacy Leakage in Language Models

Explore similar work

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

Say Something Else: Rethinking Contextual Privacy as Information Sufficiency