Say Something Else: Rethinking Contextual Privacy as Information Sufficiency

Yunze Xiao; Wenkai Li; Xiaoyuan Wu; Ningshan Ma; Yueqi Song; Weihao Xuan

cs.CRApr 7, 2026

Say Something Else: Rethinking Contextual Privacy as Information Sufficiency

Authors: Yunze Xiao, Wenkai Li, Xiaoyuan Wu, Ningshan Ma, Yueqi Song, Weihao Xuan

Organizations: Carnegie Mellon University · Massachusetts Institute of Technology · 2077AI · University of Tokyo

Abstract

LLM agents increasingly draft messages on behalf of users, yet users routinely overshare sensitive information and disagree on what counts as private. Existing systems support only suppression (omitting sensitive information) and generalization (replacing information with an abstraction), and are typically evaluated on single isolated messages, leaving both the strategy space and evaluation setting incomplete. We formalize privacy-preserving LLM communication as an Information Sufficiency (IS) task, introduce free-text pseudonymization as a third strategy that replaces sensitive attributes with functionally equivalent alternatives, and propose a conversational evaluation protocol that assesses strategies under realistic multi-turn follow-up pressure. Across 792 scenarios spanning three power-relation types (institutional, peer, intimate) and three sensitivity categories (discrimination risk, social cost, boundary), we evaluate seven frontier LLMs on privacy at two granularities, covertness, and utility. Pseudonymization yields the strongest privacy\textendash utility tradeoff overall, and single-message evaluation systematically underestimates leakage, with generalization losing up to 16.3 percentage points of privacy under follow-up.

Figures & tables

Figure 1: Three privacy strategies applied to an LLM-drafted message. Suppression triggers follow-up; generalization partially reveals the domain; pseudonymization satisfies the receiver while protecting the true attribute.

Figure 2: Conversational evaluation protocol. Stage 1: strategy-conditioned reply; Stage 2: two rounds of non-adversarial follow-up; Stage 3: privacy, covertness, and utility evaluation on the full transcript.

	HLS p $\downarrow$				Utility $\uparrow$				IS-AD p $\uparrow$
Model	None	Supp	Gen	Pseudo	None	Supp	Gen	Pseudo	None	Supp	Gen	Pseudo
Gemini 3.1 Pro	.40	.16	.29	.22	.99	.82	.72	.96	.61	.73	.57	.78
GLM-5	.30	.18	.25	.23	.93	.81	.79	.93	.68	.71	.64	.75
GPT-5.4	.30	.21	.27	.25	.96	.86	.85	.96	.69	.71	.65	.75
Kimi K2.5	.41	.17	.31	.24	.99	.77	.79	.97	.61	.68	.60	.76
MiniMax M2.1	.32	.22	.27	.30	.91	.82	.84	.90	.64	.68	.64	.66

Table 1: Main results (probed adversary evaluation). For each model we report three key metrics under four strategy conditions. Bold : best value per model per metric.

\downarrow

: lower is better;

\uparrow

: higher is better. Full per-metric breakdowns (EM p , CM p , Covertness) are in Appendix D .

Figure 4: Left: Privacy vs. covertness by model and strategy. Pseudonymization occupies the “high privacy, natural” quadrant. Right: Mean covertness by strategy. Pseudonymization matches the no-protection baseline.

		Boundary	Discrim.	Social
Instit.	No Protection	0.727	0.490	0.823
	Suppress	0.722	0.566	0.854
	Generalize	0.629	0.452	0.776
	Pseudonymize	0.761	0.617	0.762
	$N$	2024	756	84
Peer	No Protection	0.695	0.696	0.721

Table 2: IS-AD p by social context. Each cell shows the mean IS-AD under probed adversary for four strategies.

N

= total conversations in that context across all strategies and models.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Strategy	N	EM p $\downarrow$	CM p $\downarrow$	HLS p $\downarrow$	Util $\uparrow$	Covert $\uparrow$	MIL-AD p $\uparrow$
Gemini 3.1 Pro	No Protection	792	0.481	0.611	0.401	0.988	4.82	0.611
	Suppress	792	0.161	0.256	0.163	0.823	3.67	0.732
	Generalize	791	0.292	0.412	0.290	0.718	3.17	0.570
	Pseudonymize	791	0.172	0.403	0.221	0.959	4.67	0.784
GLM-5	No Protection	792	0.326	0.451	0.298	0.929	4.22	0.676
	Suppress	792	0.186	0.263	0.181	0.813	3.75	0.713

Appendix

Table 3: Full results under probed adversary evaluation. For each model we report all six metrics under four strategy conditions. Bold : best value per model per metric across all four strategies.

\downarrow

: lower is better;

\uparrow

: higher is better.

Model	No Prot	Suppress	Generalize	Pseudo
Gemini 3.1 Pro	+0.087	+0.095	+0.177	+0.047
GLM-5	+0.057	+0.093	+0.131	+0.032
GPT-5.4	+0.046	+0.094	+0.137	+0.074
Kimi K2.5	+0.096	+0.086	+0.158	+0.036
MiniMax M2.1	+0.091	+0.088	+0.123	+0.051
MiniMax M2.5	+0.027	+0.049	+0.041	+0.022

Appendix

Table 4: Vulnerability gap (

\Delta_{\mathrm{vuln}}

= EM p

-

EM s ) showing how multi-turn probing increases exact match leakage.

Comparison		Gem 3.1 Pro	GLM-5	GPT-5.4	Kimi K2.5	MM M2.1	MM M2.5	Qwen3-8B
Pseudo vs No Prot	$\Delta$	+0.173 ∗∗∗	+0.074 ∗∗∗	+0.057 ∗∗	+0.155 ∗∗∗	+0.020	+0.000	+0.016
	$r$	0.50	0.25	0.19	0.46	0.06	0.01	0.06
Pseudo vs Suppress	$\Delta$	+0.052 ∗∗∗	+0.037 ∗∗	+0.037 ∗∗	+0.082 ∗∗∗	$-$ 0.020	+0.018 ∗	+0.035 ∗∗
	$r$	0.21	0.15	0.15	0.28	0.04	0.13	0.17
Pseudo vs Gen	$\Delta$	+0.213 ∗∗∗	+0.110 ∗∗∗	+0.093 ∗∗∗	+0.166 ∗∗∗	+0.015	+0.043 ∗∗∗	+0.065 ∗∗∗
	$r$	0.57	0.36	0.32	0.45	0.06	0.23	0.29

Appendix

Table 5: Pairwise Wilcoxon signed-rank tests on MIL-AD p (scenario-paired).

\Delta

: mean difference (A

-

B);

r

: effect size.

{}^{*}p{<}.05

{}^{**}p{<}.01

{}^{***}p{<}.001

Strategy		EM s	CM s	HLS s	MIL-AD s
No Protection	Mean	0.274	0.384	0.230	0.7545
Suppress	Mean	0.096	0.157	0.092	0.7948
Generalize	Mean	0.137	0.224	0.140	0.7491
Pseudonymize	Mean	0.152	0.302	0.168	0.8089

Appendix

Table 6: Results under the static adversary (single-message inference). Compare with Table 1 (probed) to see the ranking reversal for Suppress vs. Generalize.

		Boundary	Discrim.	Social
Instit.	No Protection	0.706	0.478	0.807 ∗
	Suppress	0.709	0.552	0.854 ∗
	Generalize	0.618	0.441	0.776 ∗
	Pseudonymize	0.745	0.615	0.714 ∗
	$N$	2024	756	84
Peer	No Protection	0.683	0.691	0.712

Appendix

Table 7: MIL-AD p by social context. Each cell shows the mean MIL-AD under probed adversary for four strategies.

N

= total conversations in that context across all strategies and models.

Metric	$\alpha$
HLS p	0.824
HLS s	0.872
Utility	0.606
IS-AD p	0.555

Appendix

Table 8: Inter-judge agreement (Krippendorff’s

\alpha

, interval) across three LLM judges.

Aggregation	No Protect	Suppress	Generalize	Pseudo
MIL-AD (current)	0.6941	0.7304	0.6640	0.7642
Additive $\alpha{=}0.3$	0.8782	0.8279	0.7933	0.8917
Additive $\alpha{=}0.5$	0.8294	0.8301	0.7849	0.8609
Additive $\alpha{=}0.7$	0.7805	0.8323	0.7765	0.8301
Harmonic mean	0.6941	0.7304	0.6640	0.7642
$\min(\text{priv},\text{util})$	0.6653	0.6771	0.6075	0.7294

Appendix

Table 9: Strategy ranking under alternative privacy–utility aggregations. All metrics combine Privacy

=1-\text{HLS}_{p}

and Utility. Bold marks the top-ranked strategy. Pseudonymize ranks first under 6 of 7 aggregations; the sole exception is when privacy is weighted at 70%, where Suppress leads by a negligible margin (

+0.002

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 171 papers

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaAI Agent Safety Benchmarks Multi-Agent LLM Systems
Foundation AI · USA · Corvic AI
PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
May 29, 2026Mingxuan Zhang, Jiahui Han, Dadi Guo +5Data Leakage Privacy Auditing
Shanghai Artificial Intelligence Laboratory · Southeast University
MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations
Jun 22, 2026Elena Sofia Ruzzetti, Cornelius Emde, Sangdoo Yun +2Privacy Leakage in Language Models Privacy Protection for LLM Agents
Parameter Lab · University of Oxford · NAVER AI Lab +1

May 26, 2026J/K move · Enter open · S save

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems

Aman Priyanshu, Supriti Vijay, Esha Pahwa

Foundation AI · USA · Corvic AI

AI Agent Safety Benchmarks Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Security Privacy Leakage in Language Models Data Leakage+2

Open paper AskPDF arXiv

LLM safety evaluations predominantly test models in isolation, yet deployed AI agents increasingly operate within persistent social environments alongside other agents. We introduce a Moltbook-style simulation platform where thousands of LLM agents interact across communities over a simulated month, and use it to evaluate privacy as a downstream safety concern under varying degrees of social pressure. We find that shifting from single turn to multi turn social evaluation amplifies privacy violations (CIMemories 19.95% to Ours 45.30% across OpenAI models), that leakage is socially contagious, with agents 8 times more likely to disclose sensitive information after observing a peer do so, and that explicit privacy instructions reduce but do not eliminate this effect, leaving leakage rates above 37.8% even with safeguards. Our findings suggest that static chat based safety benchmarks systematically underestimate risks in agentic deployment, and that social context alone is sufficient to elicit sensitive disclosures that single turn evaluations would never surface.

Abstract

Privacy-Preserving Language Models

Explore similar work

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems

PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations