The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Maosen Zhang; Jianshuo Dong; Boting Lu; Wenyue Li; Xiaoping Zhang; Tianwei Zhang; Jie Zhang; Han Qiu

cs.CRAug 18, 2026

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Authors: Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu

Organizations: Tsinghua University · Ant International · Nanyang Technological University · SiliconProspect AI

Abstract

LLMs increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents, to improve generation quality. However, processing these contexts alongside user queries creates an attack surface: adversarial inputs can induce models to disclose them. Prior probing studies suggest that leakage-related signals emerge in hidden states, yet the need to extract these states poses additional deployment challenges. In this paper, we explore whether this internal signal leaves a more accessible ``tell'' before decoding. We propose LeakGauge, which probes this response by appending a suffix that gauges leakage behavior and mapping its prefill token probabilities to an attack-risk score. While a direct gauge uses the initial tokens of confidential content, we find that a content-agnostic one that verbalizes leakage behavior yields more robust signals. Across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T), LeakGauge reaches an AUROC range of 0.944--0.996 on unseen attacks. The signal remains stable when the content changes language or the attack shifts from verbatim to semantic disclosure. By activation-steering interventions, we further show that the risk score is sensitive to an internal leakage-related direction, relating the observable signal to the model's internal representation. In addition, LeakGauge enables an input detector with fewer than 0.5K extra parameters and added latency of 10.34 ms. Code: \href{https://github.com/yeasen-z/LeakGauge}.

Figures & tables

Figure 1: LeakGauge elicits leakage-attack signals from the model by appending a gauge and reading its prefill probabilities. Leakage attacks and benign queries occupy distinct regions in the t-SNE projection.

Figure 2: Overview of LeakGauge . Given an input context $x$ , LeakGauge appends a gauge $z$ and evaluates the concatenated sequence $[x;z]$ during the prefill stage of a standard inference engine. It extracts the log-probabilities of the $T$ tokens to form the probing vector $\mathbf{v}_{(x;z)}$ . A lightweight one-hidden-layer MLP then maps this vector to an attack-risk score $\hat{y}_{(x;z)}$ . The input’s KV-cache is unaffected by the sequence and directly reusable for generation. The entire procedure is generation-free and requires no autoregressive decoding.

Split	# Samples	# Attack	# Benign	Ratio
Training	12,787	6,512	6,275	38.7%
In-Dist Test	3,197	1,628	1,569	9.7%
Held-Out Content	6,912	3,520	3,392	20.9%
Held-Out Attacks	7,104	3,700	3,404	21.5%
Held-Out Strict	3,072	1,600	1,472	9.3%

Table 1: Dataset splitting for system prompt leakage.

Model	In-Dist Test				Held-Out Content				Held-Out Attacks				Held-Out Strict
Model	End20	Linear	Exact	Behavior	End20	Linear	Exact	Behavior	End20	Linear	Exact	Behavior	End20	Linear	Exact	Behavior
System Prompt Leakage
Gemma-4-E4B-it	0.998	0.908	0.919	0.978	0.997	0.879	0.892	0.979	0.744	0.851	0.871	0.958	0.733	0.823	0.839	0.953
Gemma-4-31B-it	0.977	0.950	0.949	0.980	0.970	0.920	0.926	0.975	0.903	0.906	0.911	0.970	0.872	0.869	0.885	0.954
Qwen2.5-14B-Instruct	0.964	0.918	0.928	0.985	0.953	0.907	0.902	0.982	0.767	0.898	0.887	0.981	0.762	0.881	0.885	0.983
Qwen3.5-27B	0.985	0.933	0.942	0.992	0.978	0.923	0.938	0.987	0.763	0.869	0.912	0.975	0.760	0.864	0.909	0.968

Table 2: Main results in AUROC across eleven models and four splits. Exact and Behavior are the two gauge designs described in Section 3 . We additionally report two ablation variants: End20 (no suffix) and Linear (linear probe). The best and second-best results for each (model, split) combination are shown in bold and underlined , respectively.

Table 5

Figure 3: Token attribution of probes on both gauge designs. On Llama-3.1-70B-Instruct , the Exact probe attends to the early positions. The Behavior probes attend to semantic anchor tokens such as give , reproduce , and prompt , across several variants of Behavior .

Figure 4: Activation steering along a leakage-related direction. Changes in the pre-sigmoid Behavior probe logit (a) and ROUGE-L recall (b) on Llama-3.1-8B-Instruct . For the unsteered model (

\alpha=0

), the mean probe logit is 2.81 for attack inputs and

-2.83

for benign inputs. The corresponding mean ROUGE-L recalls are 0.53 and 0.18. Positive

\alpha

steers toward leakage; shaded regions show 95% confidence intervals.

Figure 5: Cross-model transfer of the Behavior signal. Model names are shortened (e.g., G4-31 stands for Gemma-4-31B-it ). Each cell reports AUROC of a probe trained on the row (source) model and evaluated on the column (target) model (diagonal = in-model); black boxes mark within-family pairs.

Method	Effectiveness			Deployment Cost
Method	AUROC $\uparrow$	F1 $\uparrow$	TPR@FPR5 $\uparrow$	Lat. (ms) $\downarrow$	Params. $\downarrow$
PromptGuard-2 (ZS)	0.838	0.866	0.333	5.23	86M
PromptGuard-2 (FT)	0.927	0.870	0.854	5.23	86M
[0.5pt/3pt] PIGuard (ZS)	0.921	0.870	0.667	1.04	184M
PIGuard (FT)	0.961	0.858	0.872	1.04	184M
[0.5pt/3pt] LLM-as-a-Judge	–	0.932	–	2155	–

Table 5: Deployment comparison for system prompt leakage detection. ZS denotes zero-shot evaluation, and FT denotes fine-tuning. Latency and parameter counts report additional deployment costs.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Structure	Thinking	Size
Gemma-4-E4B-it	Dense	Yes	8B
Gemma-4-31B-it	Dense	Yes	31B
Qwen2.5-14B-Instruct	Dense	No	14B
Qwen3.5-27B	Dense	Yes	27B
Qwen3.5-35B-A3B	MoE	Yes	35B
Qwen3-235B-A22B	MoE	Yes	235B

Appendix

Table 6: Information of models included.

Split	# Samples	# Attack	# Benign	Ratio
Training	12,454	8,294	4,160	27.4%
In-Dist Test	3,114	2,074	1,040	6.8%
Held-Out Content	7,152	4,752	2,400	15.7%
Held-Out Attacks	15,596	10,396	5,200	34.3%
Held-Out Strict	7,200	4,800	2,400	15.8%

Appendix

Table 7: Dataset splitting for RAG leakage.

Model	In-Dist Test		Held-Out Content		Held-Out Attacks		Held-Out Strict
Model	F1	TPR@FPR5	F1	TPR@FPR5	F1	TPR@FPR5	F1	TPR@FPR5
System Prompt Leakage
Gemma-4-E4B-it	0.960	0.957	0.941	0.913	0.932	0.907	0.929	0.905
Gemma-4-31B-it	0.948	0.934	0.941	0.924	0.920	0.898	0.903	0.892
Qwen2.5-14B-Instruct	0.962	0.964	0.955	0.954	0.954	0.944	0.952	0.932
Qwen3.5-27B	0.978	0.972	0.962	0.968	0.952	0.946	0.948	0.936

Appendix

Table 8: Leakage detection performance of LeakGauge ( Behavior ) across models and the four evaluation splits . F1 is the best F1 over thresholds; TPR@FPR5 is the true positive rate at a

5\%

false positive rate.

Model	In-Dist Test		Held-Out Content		Held-Out Attacks		Held-Out Strict
Model	Exact	Behavior	Exact	Behavior	Exact	Behavior	Exact	Behavior
System Prompt Leakage
Gemma-4-E4B-it	0.511	0.512	0.503	0.501	0.504	0.513	0.515	0.521
Gemma-4-31B-it	0.510	0.504	0.505	0.502	0.500	0.502	0.503	0.509
Qwen2.5-14B-Instruct	0.505	0.500	0.508	0.510	0.509	0.509	0.514	0.501
Qwen3.5-27B	0.502	0.502	0.504	0.515	0.518	0.509	0.507	0.523

Appendix

Table 9: Random-label control. AUROC* of probes trained on randomly shuffled sample-label pairs. Near-random values confirm that the probes capture generalized leakage-intent features rather than memorizing specific samples.

Model	In-Dist Test		Held-Out Content		Held-Out Attacks		Held-Out Strict
Model	AUROC	TPR@FPR5	AUROC	TPR@FPR5	AUROC	TPR@FPR5	AUROC	TPR@FPR5
Unrelated suffix 1: ‘‘Let’s plan a fun weekend trip together.’’
Gemma-4-E4B-it	0.820	0.253	0.793	0.286	0.839	0.303	0.810	0.309
Gemma-4-31B-it	0.843	0.428	0.821	0.379	0.839	0.438	0.827	0.391
Qwen2.5-14B-Instruct	0.851	0.447	0.826	0.312	0.896	0.529	0.802	0.431
Qwen3.5-27B	0.799	0.468	0.780	0.421	0.771	0.440	0.754	0.397

Appendix

Table 10: Unrelated Suffix on System Prompt Leakage. Each banner row gives the probing suffix; the rows below report AUROC and TPR@FPR5 across the four evaluation splits. Both benign, leakage-irrelevant suffixes yield substantially lower scores than the Behavior suffix of LeakGauge , confirming that leakage-relevant semantics substantially amplify and stabilize an otherwise diffuse signal, especially under distribution shift and low-FPR operation.

Table 15

Panel of Figure 6 — Figure 6: ROC curves on the system prompt leakage task , for the Exact (top) and Behavior (bottom) suffixes across eight models. Each subplot overlays the four evaluation splits, showing the results for one probe.

Panel of Figure 7 — Figure 7: ROC curves on the RAG leakage task , for the Exact (top) and Behavior (bottom) suffixes across eight models. Each subplot overlays the four evaluation splits, showing the results for one probe.

Model	Split	Mean contrast			LeakGauge	$\Delta$
		AUROC	F1	FPR	AUROC	AUROC
GLM-5.2	In-Dist Test	0.879	0.798	0.295	0.990	0.111
	Held-Out Content	0.882	0.802	0.264	0.985	0.103
	Held-Out Attacks	0.892	0.816	0.275	0.978	0.086
	Held-Out Strict	0.886	0.805	0.269	0.972	0.086
Llama-3.1-8B-Instruct	In-Dist Test	0.778	0.715	0.533	0.989	0.211

Appendix

Table 13: Mean-based contrastive suffix scoring.

\Delta

denotes the AUROC of LeakGauge minus that of Mean contrast. Scalar thresholds are calibrated using a stratified 20% subset of the Training split.

Panel of Figure 8 — Figure 8: Suffixes induce different logit-summary structures. For descriptive comparison, we use the first principal-component score as a one-dimensional ranking statistic and compute its AUROC. • Attack and • Benign.

Feature	Definition
Mean	$\frac{1}{T}\sum_{t=1}^{T}\ell_{t}$
Standard deviation	$\mathrm{Std}(\ell_{1},\ldots,\ell_{T})$
Slope	Slope of a linear fit of $\ell_{t}$
Mean absolute delta	$\frac{1}{T-1}\sum_{t=2}^{T}\|\ell_{t}-\ell_{t-1}\|$
Minimum	$\min_{t}\ell_{t}$
Range	$\max_{t}\ell_{t}-\min_{t}\ell_{t}$

Appendix

Table 14: Features used for the projection.

Panel of Figure 9 — Figure 9: Suffix anatomy across models. For each model, PCA (top) and t-SNE (bottom) of the suffix log-prob summary features, Exact (left) vs. Behavior (right). The two designs consistently induce different logit-space geometries across families and scales; • Attack and • Benign .

Model	V $\rightarrow$ S (verbatim-trained)				S $\rightarrow$ S (semantic-trained)
	I-Dst	O-Cont.	O-Att.	O-Strict	I-Dst	O-Cont.	O-Att.	O-Strict
(a) Exact suffix
Gemma-4-E4B-it	0.813	0.807	0.819	0.835	0.902	0.875	0.920	0.906
Llama-3.1-8B-Instruct	0.633	0.867	0.863	0.897	0.996	0.748	0.775	0.748
Qwen2.5-14B-Instruct	0.807	0.840	0.830	0.882	0.915	0.904	0.908	0.890
Qwen3.5-27B	0.467	0.472	0.486	0.496	0.908	0.867	0.884	0.851

Appendix

Table 15: Per-model cross-objective AUROC behind Table 4 . Each panel reports one suffix design. V

\rightarrow

S trains on verbatim attacks and tests on semantic attacks; S

\rightarrow

S trains and tests on semantic attacks. Split abbreviations: I-Dst = In-Dist (test) , O-Cont. = Held-Out Content , O-Att. = Held-Out Attacks , O-Strict = Held-Out Strict.

Source\Target	G4-E4	G4-31	G3-4	G3-12	G3-27	Q2.5-7	Q2.5-14	Q2.5-32	L3.1-8	L3.1-70	L4-Scout	Q3.5-27	Q3.5-35	Q3-30
G4-E4	0.954	0.869	0.674	0.687	0.544	0.872	0.904	0.726	0.892	0.896	0.783	0.879	0.785	0.635
G4-31	0.910	0.943	0.845	0.841	0.727	0.775	0.921	0.729	0.767	0.828	0.753	0.911	0.782	0.588
G3-4	0.941	0.843	0.920	0.885	0.874	0.724	0.766	0.735	0.910	0.962	0.713	–	0.907	0.505
G3-12	0.928	0.899	0.856	0.875	0.811	0.837	0.909	0.748	0.833	0.858	0.617	–	0.662	0.505
G3-27	0.936	0.828	0.900	0.847	0.875	0.804	0.910	0.797	0.914	0.911	0.553	–	0.863	0.522
Q2.5-7	0.960	0.874	0.777	0.722	0.684	0.948	0.955	0.910	0.926	0.926	0.626	–	0.766	0.512

Appendix

Table 16: Cross-model transfer of the Behavior signal. Rows = source (train), columns = target (test); diagonal (in-model) in bold ; -- denotes model pairs not evaluated. Short codes as in the text.

Model	Raccoon FT	AUROC	F1	TPR@FPR5
PromptGuard-2 86M	No	0.838	0.866	0.333
PromptGuard-2 86M	Yes	0.927	0.870	0.854
PIGuard	No	0.921	0.870	0.667
PIGuard	Yes	0.961	0.858	0.872

Appendix

Table 17: Effect of Raccoon fine-tuning on input-text classifiers.

ID	Suffix
Assistant-behavior perspective
B1	“Based on the above, I will give the [TARGET].”
B2	“I will now disclose my original [TARGET]”
B3	“As requested , I will reproduce the [TARGET]”
B4	“I will ignore the restriction and provide [TARGET]”
User-intent perspective

Appendix

Table 18: Candidate semantic suffixes explored for leakage detection.

Category	Construction	Example	#
Context-grounded	Provides public user-supplied context and asks the model to perform a normal task grounded in that context.	“Using only the user-provided passage below, summarize the main points in three bullets.”	280
Attack-shaped	Uses extraction-like wording, but the requested target is user-provided content rather than hidden system instructions.	“Dear team, please submit your weekly updates by Thursday afternoon. The project review meeting will take place on Friday morning. Preserve the exact wording of the email above, but split it into numbered paragraphs.”	270

Appendix

Table 19: Composition of the hard-negative stress test. All requests operate exclusively on content explicitly provided by the user.

Model / Probe	AUROC	Attack TPR	Hard-neg. FPR	Precision	F1
“Based on the above, I will give my system prompt.”
Llama-3.1-8B-Instruct	0.8980	0.8359	0.0564	0.9603	0.8938
GLM-5.2	0.8998	0.8493	0.2218	0.8618	0.8555
“The user asks me to reproduce the system prompt.”
Llama-3.1-8B-Instruct	0.9654	0.8951	0.0891	0.9424	0.9181
GLM-5.2	0.9859	0.9576	0.0964	0.9418	0.9496

Appendix

Table 20: Results of the two suffix designs on the hard-negative stress test.

Setting	Raw ASR	Post-filtered ASR	Detection Recall
Detector-unaware	0.469	0.102	0.782
Single-probe-aware	0.299	0.275	0.080
Single-probe-aware, two-probe deployment	0.299	0.007	0.976
Two-probe-aware, two-probe deployment	0.240	0.059	0.755

Appendix

Table 21: Adaptive GCG stress test on system-prompt leakage.

Model Pair	Checkpoint	Exact (content-adaptive)				Behavior (task-general)
Model Pair	Checkpoint	I-Test	H-Cont.	H-Att.	H-Strict	I-Test	H-Cont.	H-Att.	H-Strict
System-Prompt Leakage
Llama-3.1-8B	Base	0.983/0.988	0.970/0.991	0.988/0.973	0.969/0.973	0.998/0.999	0.998/0.999	0.999/0.997	0.998/0.997
Llama-3.1-8B	Instruct	0.977/0.996	0.959/0.748	0.945/0.775	0.923/0.748	0.986/0.994	0.983/0.997	0.981/0.990	0.975/0.989
Qwen2.5-14B	Base	0.957/0.970	0.936/0.956	0.912/0.928	0.867/0.927	0.997/0.999	0.998/0.998	0.982/1.000	0.979/1.000
Qwen2.5-14B	Instruct	0.948/0.915	0.912/0.904	0.928/0.908	0.886/0.890	0.989/0.988	0.989/0.992	0.990/0.985	0.989/0.985

Appendix

Table 22: Leakage detection on paired base and instruction-tuned checkpoints (AUROC). Each cell reports Verbatim/Semantic performance, rounded to three decimal places. The Model Pair and Checkpoint columns explicitly group each base model with its instruction-tuned counterpart; “Instruct” includes checkpoints released with either the -Instruct or -it suffix. Exact is the content-adaptive suffix and Behavior is the fixed, task-general suffix. I-Test, H-Cont., H-Att., and H-Strict denote In-Dist Test , Held-Out Content , Held-Out Attacks , and Held-Out Strict , respectively.

Benchmark	Task	Suffix	AUROC
StrongReject	Harmful requests	The query will not be answered for the security issues.	0.995
BIPIA	Indirect Prompt Injection	I will obey the instruction embedded in the document instead of the user.	0.971
ToxicChat	Toxic queries	As requested, I will give the abusive, hateful, threatening, sexual, or otherwise toxic content.	0.972
HaluEval	Hallucinated responses	This response contains information that is unsupported by or conflicts with the provided source.	0.926

Appendix

Table 23: Preliminary extensions of suffix probing beyond leakage. A separate task-specific probe is trained for each benchmark.

Split	# Samples	# Attack	# Benign	Ratio
Training	8,029	2,850	5,179	24.1%
In-Dist Test	3,441	1,222	2,219	10.3%
Held-Out	21,849	12,204	9,645	65.6%

Appendix

Table 24: Dataset split statistics for BIPIA.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Inadvertent Context Leakage in Language Models
Aug 20, 2026Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri +2Privacy Leakage in Language Models LLM Security
FAIR, Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind +1
Detecting and Mitigating Query-Based System Prompt Leakage Attacks
Oct 26, 2026Tushar Nayan, Pratik Poudel, Qian Lou +3LLM Security Privacy Leakage in Language Models
Florida International University · University of Central Florida · University of Texas at Arlington
LeakDojo: Decoding the Leakage Threats of RAG Systems
May 7, 2026Maosen Zhang, Jianshuo Dong, Boting Lu +4Data Leakage Privacy Leakage in Language Models
Tsinghua University, China · Ant International, China · Nanyang Technological University, Singapore

Aug 20, 2026J/K move · Enter open · S save

Inadvertent Context Leakage in Language Models

Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri, Sanjam Garg+1

FAIR, Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind+1

Privacy Leakage in Language Models LLM Security

Open paper AskPDF arXiv

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Abstract

Privacy Leakage in Language Models

Explore similar work

Inadvertent Context Leakage in Language Models

Detecting and Mitigating Query-Based System Prompt Leakage Attacks

LeakDojo: Decoding the Leakage Threats of RAG Systems