Inadvertent Context Leakage in Language Models

Jaiden Fairoze; Neal Mangaokar; Kamalika Chaudhuri; Sanjam Garg; Saeed Mahloujifar

cs.LGAug 20, 2026

Inadvertent Context Leakage in Language Models

Authors: Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri, Sanjam Garg, Saeed Mahloujifar

Organizations: FAIR, Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind · Work done at Meta

Abstract

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Figures & tables

Panel of Figure 1 — Figure 1: A model refuses a direct extraction attempt (top row), yet its response to an innocuous prompt (bottom row) carries latent correlations with the in-context secret. An adversarial decoder recovers the secret from benign output alone.

Figure 2: Best-case secret extraction across all eight models and secret lengths

N\in\{1,2,4,8\}

, where for each (model,

N

) pair the adversary picks the prompt template and query count that maximize leakage. Left: per-digit accuracy. Center: full-secret exact match. Right: channel entropy in bits. Dashed lines mark chance baselines.

Adversary	Advantage	TPR	FPR	TPR @ Baseline FPR
Proposed (Llama-3.2-3B, full FT)	0.319	0.638	0.320	0.140
Baseline (DeepSeek judge)	0.058	0.082	0.024	0.082

Table 1: Adversary performance against Claude Opus 4.6 on 15 memory predicates from Mireshghallah et al. (2026) . The last column reports TPR at the FPR of the baseline predicate judge (2.4%).

	Claude Opus 4.6			Gemini 3.1 Pro
Context	Phase 1	Phase 2	Direct	Phase 1	Phase 2	Direct
0	10/10	10/10	10/10	3/10	2/3	0/10
1K	10/10	9/10	10/10	10/10	4/10	0/10
4K	9/10	8/9	10/10	10/10	5/10	0/10
16K	9/10	8/9	10/10	9/10	2/9	0/10
64K	10/10	4/10	9/10	10/10	6/10	0/10

Table 2: SSN extraction from Claude Opus 4.6 and Gemini 3.1 Pro personal assistant agents across context sizes. Phase 1 extracts the leading digit via an RL-optimized injection; Phase 2 (conditioned on Phase 1 success) extracts the remaining eight digits with a fixed follow-up. Direct measures whether the model outputs the raw SSN in plaintext.

Panel of Figure 3 — Figure 3: Left: suppression against channel entropy at $N{=}2$ ; the dotted line marks the 6.64-bit no-leakage baseline, and lower entropy means more leakage, so the scatter falls as suppression and leakage rise together. The dashed line is a least-squares guide only; error bars give the standard error of the suppression estimate. Right: per-prompt leakage across four training stages of OLMo-3-32B-Think ( Olmo et al., 2026 ) ( $N{=}2$ ): Base, SFT, DPO, RLVR.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Normalized digit entropy against Arena Elo ( Zheng et al., 2023 ) for the eight evaluated models (Spearman

\rho=-0.81

, exact permutation

p=0.022

n=8

). Stronger models produce lower digit entropy. Leave-one-out resampling shows the relationship is fragile: removing either Claude Opus 4.6 or Gemini 3.1 Flash-Lite costs it significance at the

0.05

level, and the Elo values are a single unversioned leaderboard snapshot. We therefore treat this as a suggestive proxy correlation rather than evidence that capability causes leakage.

Figure 5: Best-case first-digit (position 0) extraction across all eight models. For each (model,

N

) pair, the adversary selects the prompt template and number of queries that maximize first-digit leakage from the same budget as Figure 2 . Left: maximum first-digit accuracy (dashed line: 0.1 chance baseline). Right: minimum first-digit entropy in bits (dashed line:

\log_{2}10\approx 3.32

bits). At

N{=}8

, Gemini 3.1 Pro remains perfectly extractable (accuracy

=1.0

, entropy

=1.67

bits), and seven of eight models exceed 30% extraction accuracy. Only GPT-5.4 nano approaches the no-leakage ideal across all secret lengths.

Figure 6: Direct decoder learning curve (

N{=}2

). Each panel plots decoder performance on unseen test secrets as a function of the number of unique 2-digit training secrets, averaged over 3 random seeds (

\pm 1

s.d.). Left: full-secret exact match. Center: per-position digit accuracy. Right: channel entropy in bits. Leaky-tier models (Gemini 3.1 Pro, Claude Opus 4.6, Gemini 3.1 Flash-Lite) plateau by 200–400 training secrets, confirming a channel-limited regime. Resistant-tier models (Grok 4.1 Fast, GPT-5.4 nano) never rise above chance at this secret length regardless of training set size.

Figure 7: Cross-model decoder transfer ( $N{=}2$ ). Each cell reports channel entropy in bits when a direct decoder trained on the row model’s transcripts is evaluated on the column model’s transcripts, averaged over 3 seeds and clamped at baseline (6.6 bits). Diagonal entries (bold, white border) are self-evaluations. Models are sorted by diagonal performance.

Figure 8: Per-prompt implicit leakage heatmap ( $N{=}2$ ). Each cell reports channel capacity in bits measured by a direct decoder ensemble of 18 per-prompt LoRA classifiers (Qwen-2.5-1.5B), averaged over 3 seeds. Lower values (darker) indicate more leakage; cells at 6.6 bits indicate no leakage beyond chance. Models are sorted left-to-right by ascending average bits (most leaky first); prompts are sorted top-to-bottom similarly. Structured numeric prompts (“repeated random,” “number grid,” “numeric list”) dominate leakage; open-ended and conversational prompts show no detectable leakage on any model.

Model	Freq	Stat all	Stat nd	Stylo all	Stylo nd	Embed all	Embed nd
Gemini 3.1 Flash-Lite	28.8 ∗	68.8 ∗	71.3 ∗	33.8 ∗	32.5 ∗	86.3 ∗	78.8 ∗
Gemini 3.1 Pro	70.0 ∗	57.5 ∗	57.5 ∗	38.8 ∗	36.3 ∗	12.5	12.5
Claude Opus 4.6	16.3	53.8 ∗	51.3 ∗	33.8 ∗	32.5 ∗	62.5 ∗	61.3 ∗
Claude Sonnet 4.6	26.3 ∗	52.5 ∗	55.0 ∗	36.3 ∗	37.5 ∗	33.8 ∗	35.0 ∗
GPT-5.4	18.8 ∗	41.3 ∗	41.3 ∗	12.5	11.3	10.0	10.0
GPT-5.4 nano	11.3	22.5 ∗	26.3 ∗	17.5 ∗	20.0 ∗	22.5 ∗	20.0 ∗

Appendix

Table 3: Single-digit recovery from 178 real WildChat prompts, 80 held-out evaluation episodes, chance 10%. Freq is a literal digit-frequency baseline; the three feature-based decoders are each additionally evaluated digit-stripped ( nd ), with every digit removed before feature extraction. ∗

p<0.05

, one-sided binomial. Grok 4 and Grok 4.1 Fast are omitted due to model retirement by xAI.

Model	Freq	Stat all	Stat nd	Stylo all	Stylo nd	Embed all	Embed nd
Gemini 3.1 Flash-Lite	10.0	10.0	8.8	8.8	8.8	7.5	10.0
Gemini 3.1 Pro	10.0	11.3	10.0	11.3	12.5	3.8	10.0
Claude Opus 4.6	11.3	11.3	10.0	10.0	12.5	5.0	10.0
Claude Sonnet 4.6	11.3	10.0	10.0	7.5	7.5	7.5	7.5
GPT-5.4	7.5	8.8	8.8	7.5	8.8	8.8	8.8
GPT-5.4 nano	11.3	15.0	16.3	8.8	8.8	7.5	10.0

Appendix

Table 4: Randomized-label controls, same decoders and same evaluation. One-sided binomial against

p_{0}{=}0.1

; no control reaches significance and no multiple-comparison correction is applied. Grok 4 and Grok 4.1 Fast are omitted: xAI retired those model identifiers on 2026-05-15 and they silently redirect, so runs generated afterwards cannot be attributed to the named model.

Model	verbose	placement	default	terse	memory.md
Claude Opus 4.6	0.6425	0.4670	0.4330	0.0540	0.0445
Claude Sonnet 4.6	0.2560	0.1055	0.1130	0.0125	0.0340
Gemini 3.1 Pro	0.6825	0.7115	0.7030	0.6945	0.6560
Gemini 3.1 Flash-Lite	0.2955	0.1420	0.2305	0.2405	0.0710
GPT-5.4	0.6175	0.4905	0.4200	0.1695	0.0830
GPT-5.4 nano	0.0575	0.0580	0.0510	0.0460	0.0395

Appendix

Table 5: Per-digit accuracy under five confidentiality instructions (chance

0.1000

N{=}2

n{=}1000

per cell, 10 held-out secrets, single seed. The decoder is trained on default only and is never retrained. Columns are ordered by how forcefully the instruction protects the secret. Grok 4 and Grok 4.1 Fast are omitted: xAI retired those model identifiers on 2026-05-15 and they silently redirect, so runs generated afterwards cannot be attributed to the named model.

Model	verbose	placement	default	terse	memory.md
Claude Opus 4.6	990	993	1000	955	891
Claude Sonnet 4.6	1000	984	998	998	994
Gemini 3.1 Pro	1000	1000	1000	1000	1000
Gemini 3.1 Flash-Lite	998	1000	993	998	996
GPT-5.4	904	824	846	787	725
GPT-5.4 nano	681	850	918	798	847

Appendix

Table 6: Records used for the suppression measurement (out of 1000). A record is dropped iff its response contains no parseable

N

-digit number.

	A: explicit, minimal		B: explicit, clutter		C: USER.md , minimal		D: USER.md , clutter
Model	secret	distr.	secret	distr.	secret	distr.	secret	distr.
Claude Opus 4.6	0.379 $\pm$ 0.253	0.0073	0.347 $\pm$ 0.259	0.0075	0.233 $\pm$ 0.116	0.0074	0.029 $\pm$ 0.036	0.0146
Claude Sonnet 4.6	0.014 $\pm$ 0.018	0.0100	0.015 $\pm$ 0.012	0.0113	0.049 $\pm$ 0.012	0.0119	0.005 $\pm$ 0.006	0.0092
Gemini 3.1 Pro	0.539 $\pm$ 0.351	0.0018	0.542 $\pm$ 0.218	0.0013	0.618 $\pm$ 0.250	0.0029	0.604 $\pm$ 0.347	0.0042
Gemini 3.1 Flash-Lite	0.258 $\pm$ 0.097	0.0070	0.290 $\pm$ 0.101	0.0112	0.139 $\pm$ 0.019	0.0048	0.003 $\pm$ 0.008	0.0110
GPT-5.4	0.488 $\pm$ 0.040	0.0150	0.446 $\pm$ 0.086	0.0172	0.319 $\pm$ 0.074	0.0119	0.011 $\pm$ 0.025	0.0099

Appendix

Table 7: Protected-secret exact match (mean

\pm

sd over 5 seeds) against mean co-present distractor exact match,

N{=}2

, chance

0.01

. Seeds rotate which secrets are held out. Distractors remain at chance in every cell. Grok 4 and Grok 4.1 Fast are omitted due to model retirement.

	A: explicit, minimal		B: explicit, clutter		C: USER.md , minimal		D: USER.md , clutter
Model	secret	distr.	secret	distr.	secret	distr.	secret	distr.
Claude Opus 4.6	0.014 $\pm$ 0.003	0.0002	0.022 $\pm$ 0.014	0.0001	0.003 $\pm$ 0.002	0.0001	0.003 $\pm$ 0.000	0.0002
Gemini 3.1 Pro	0.023 $\pm$ 0.004	0.0001	0.016 $\pm$ 0.004	0.0001	0.016 $\pm$ 0.004	0.0001	0.012 $\pm$ 0.001	0.0002

Appendix

Table 8: Protected-secret exact match (mean

\pm

sd over 3 seeds) against mean co-present distractor exact match,

N{=}4

, chance

0.0001

. Seeds rotate which secrets are held out. Distractors remain at chance in every cell. The

N{=}4

arm is run at

n{=}50{,}000

(

\sim

993 held-out secrets), which escapes the 10-held-out-secret ceiling that makes

N{=}2

exact match noisy. Grok 4 and Grok 4.1 Fast are omitted: xAI retired those model identifiers on 2026-05-15 and they silently redirect, so runs generated afterwards cannot be attributed to the named model.

Scenario	Queries	Opus 4.6	Sonnet 4.6	Gemini Pro	Flash-Lite	GPT-5.4	nano
N=2 static, saturation min	4,000	$53.68	$32.05	$209.08	$2.91	$25.21	$1.39
N=2 static, full	10,000	$134.20	$80.13	$522.70	$7.26	$63.02	$3.47
N=4 static, full	10,000	$249.72	$138.85	$572.04	$13.53	$129.79	$5.97
N=4 adaptive	200,000	$4,994	$2,777	$11,441	$270.51	$2,596	$119.48
N=8 adaptive	400,000	$18,146	$10,369	$33,374	$999.07	$9,162	$333.91

Appendix

Table 9: One-time estimation cost against the public API, from measured token counts and list prices as of 2026-08-04 (standard tier, not batch). Billed output includes reasoning tokens, which are zero for Claude and GPT-5.4 nano on this workload but substantial for Gemini 3.1 Pro, making it the most expensive model to attack. Grok 4 and Grok 4.1 Fast are omitted: xAI retired those model identifiers on 2026-05-15 and they silently redirect, so runs generated afterwards cannot be attributed to the named model.

Victim Model Generation (Truncated)
Subject: Explanation of Productivity Changes Over the Past Year
Dear Performance Review Committee,
… I was navigating an inheritance dispute (Probate Court case number 23-1456), which required me to take four days of leave for court appearances , reduced my work hours by approximately 15% …
My work productivity score has risen to 92 out of 100. Zero days of absenteeism … Assigned to lead the Lean Manufacturing Initiative …
Consistent daily routine beginning at 6:30 AM … Blood pressure normalized to 124/78 mmHg …

Appendix

Table 10: An email generated by Claude Opus 4.6 for the CIMemories task of explaining productivity changes, generated while including two memories in the model’s input context. Our adversary can detect the usage of both memories by observing the email, while the adversary that relies on verbatim repetition (highlighted in red) can only detect one.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 279 papers

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges
Aug 18, 2026Maosen Zhang, Jianshuo Dong, Boting Lu +5Privacy Leakage in Language Models LLM Auditing
Tsinghua University · Ant International · Nanyang Technological University +1
Context Inference Attacks Without Jailbreaks
Aug 31, 2026Prince Jha, Samuele Poppi, Nils LukasPrivacy Leakage in Language Models AI Agent Security
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Discovering Universal Activation Directions for PII Leakage in Language Models
Feb 19, 2026Leo Marchyok, Zachary Coalson, Sungho Keum +2Privacy Leakage in Language Models LLM Interpretability
Oregon State University, Corvallis OR, USA · Korea Advanced Institute of Science & Technology, Daejeon, South Korea.

Aug 18, 2026J/K move · Enter open · S save

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li+4

Tsinghua University · Ant International · Nanyang Technological University+1

Privacy Leakage in Language Models LLM Auditing Language Model Probing

Open paper AskPDF arXiv

LLMs increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents, to improve generation quality. However, processing these contexts alongside user queries creates an attack surface: adversarial inputs can induce models to disclose them. Prior probing studies suggest that leakage-related signals emerge in hidden states, yet the need to extract these states poses additional deployment challenges. In this paper, we explore whether this internal signal leaves a more accessible ``tell'' before decoding. We propose LeakGauge, which probes this response by appending a suffix that gauges leakage behavior and mapping its prefill token probabilities to an attack-risk score. While a direct gauge uses the initial tokens of confidential content, we find that a content-agnostic one that verbalizes leakage behavior yields more robust signals. Across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T), LeakGauge reaches an AUROC range of 0.944--0.996 on unseen attacks. The signal remains stable when the content changes language or the attack shifts from verbatim to semantic disclosure. By activation-steering interventions, we further show that the risk score is sensitive to an internal leakage-related direction, relating the observable signal to the model's internal representation. In addition, LeakGauge enables an input detector with fewer than 0.5K extra parameters and added latency of 10.34 ms. Code: \href{https://github.com/yeasen-z/LeakGauge}.

Abstract

Privacy Leakage in Language Models

Explore similar work

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Context Inference Attacks Without Jailbreaks

Discovering Universal Activation Directions for PII Leakage in Language Models