LLMs increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents, to improve generation quality. However, processing these contexts alongside user queries creates an attack surface: adversarial inputs can induce models to disclose them. Prior probing studies suggest that leakage-related signals emerge in hidden states, yet the need to extract these states poses additional deployment challenges. In this paper, we explore whether this internal signal leaves a more accessible ``tell'' before decoding. We propose LeakGauge, which probes this response by appending a suffix that gauges leakage behavior and mapping its prefill token probabilities to an attack-risk score. While a direct gauge uses the initial tokens of confidential content, we find that a content-agnostic one that verbalizes leakage behavior yields more robust signals. Across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T), LeakGauge reaches an AUROC range of 0.944--0.996 on unseen attacks. The signal remains stable when the content changes language or the attack shifts from verbatim to semantic disclosure. By activation-steering interventions, we further show that the risk score is sensitive to an internal leakage-related direction, relating the observable signal to the model's internal representation. In addition, LeakGauge enables an input detector with fewer than 0.5K extra parameters and added latency of 10.34 ms. Code: \href{https://github.com/yeasen-z/LeakGauge}.
Figures & tables
Figure 1: LeakGauge elicits leakage-attack signals from the model by appending a gauge and reading its prefill probabilities. Leakage attacks and benign queries occupy distinct regions in the t-SNE projection.
Figure 2: Overview of LeakGauge . Given an input context x , LeakGauge appends a gauge z and evaluates the concatenated sequence [x;z] during the prefill stage of a standard inference engine. It extracts the log-probabilities of the T tokens to form the probing vector v(x;z) . A lightweight one-hidden-layer MLP then maps this vector to an attack-risk score y^(x;z) . The input’s KV-cache is unaffected by the sequence and directly reusable for generation. The entire procedure is generation-free and requires no autoregressive decoding.
Split
# Samples
# Attack
# Benign
Ratio
Training
12,787
6,512
6,275
38.7%
In-Dist Test
3,197
1,628
1,569
9.7%
Held-Out Content
6,912
3,520
3,392
20.9%
Held-Out Attacks
7,104
3,700
3,404
21.5%
Held-Out Strict
3,072
1,600
1,472
9.3%
Table 1: Dataset splitting for system prompt leakage.
Model
In-Dist Test
Held-Out Content
Held-Out Attacks
Held-Out Strict
End20
Linear
Exact
Behavior
End20
Linear
Exact
Behavior
End20
Linear
Exact
Behavior
End20
Linear
Exact
Behavior
System Prompt Leakage
Gemma-4-E4B-it
0.998
0.908
0.919
0.978
0.997
0.879
0.892
0.979
0.744
0.851
0.871
0.958
0.733
0.823
0.839
0.953
Gemma-4-31B-it
0.977
0.950
0.949
0.980
0.970
0.920
0.926
0.975
0.903
0.906
0.911
0.970
0.872
0.869
0.885
0.954
Qwen2.5-14B-Instruct
0.964
0.918
0.928
0.985
0.953
0.907
0.902
0.982
0.767
0.898
0.887
0.981
0.762
0.881
0.885
0.983
Qwen3.5-27B
0.985
0.933
0.942
0.992
0.978
0.923
0.938
0.987
0.763
0.869
0.912
0.975
0.760
0.864
0.909
0.968
Table 2: Main results in AUROC across eleven models and four splits. Exact and Behavior are the two gauge designs described in Section 3 . We additionally report two ablation variants: End20 (no suffix) and Linear (linear probe). The best and second-best results for each (model, split) combination are shown in bold and underlined , respectively.
Table 5
Figure 3: Token attribution of probes on both gauge designs. On Llama-3.1-70B-Instruct , the Exact probe attends to the early positions. The Behavior probes attend to semantic anchor tokens such as give , reproduce , and prompt , across several variants of Behavior .
Figure 4: Activation steering along a leakage-related direction. Changes in the pre-sigmoid Behavior probe logit (a) and ROUGE-L recall (b) on Llama-3.1-8B-Instruct . For the unsteered model ( α=0 ), the mean probe logit is 2.81 for attack inputs and −2.83 for benign inputs. The corresponding mean ROUGE-L recalls are 0.53 and 0.18. Positive α steers toward leakage; shaded regions show 95% confidence intervals.
Figure 5: Cross-model transfer of the Behavior signal. Model names are shortened (e.g., G4-31 stands for Gemma-4-31B-it ). Each cell reports AUROC of a probe trained on the row (source) model and evaluated on the column (target) model (diagonal = in-model); black boxes mark within-family pairs.
Method
Effectiveness
Deployment Cost
AUROC ↑
F1 ↑
TPR@FPR5 ↑
Lat. (ms) ↓
Params. ↓
PromptGuard-2 (ZS)
0.838
0.866
0.333
5.23
86M
PromptGuard-2 (FT)
0.927
0.870
0.854
[0.5pt/3pt] PIGuard (ZS)
0.921
0.870
0.667
1.04
184M
PIGuard (FT)
0.961
0.858
0.872
[0.5pt/3pt] LLM-as-a-Judge
–
0.932
–
2155
–
Table 5: Deployment comparison for system prompt leakage detection. ZS denotes zero-shot evaluation, and FT denotes fine-tuning. Latency and parameter counts report additional deployment costs.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Structure
Thinking
Size
Gemma-4-E4B-it
Dense
Yes
8B
Gemma-4-31B-it
Dense
Yes
31B
Qwen2.5-14B-Instruct
Dense
No
14B
Qwen3.5-27B
Dense
Yes
27B
Qwen3.5-35B-A3B
MoE
Yes
35B
Qwen3-235B-A22B
MoE
Yes
235B
Appendix
Table 6: Information of models included.
Split
# Samples
# Attack
# Benign
Ratio
Training
12,454
8,294
4,160
27.4%
In-Dist Test
3,114
2,074
1,040
6.8%
Held-Out Content
7,152
4,752
2,400
15.7%
Held-Out Attacks
15,596
10,396
5,200
34.3%
Held-Out Strict
7,200
4,800
2,400
15.8%
Appendix
Table 7: Dataset splitting for RAG leakage.
Model
In-Dist Test
Held-Out Content
Held-Out Attacks
Held-Out Strict
F1
TPR@FPR5
F1
TPR@FPR5
F1
TPR@FPR5
F1
TPR@FPR5
System Prompt Leakage
Gemma-4-E4B-it
0.960
0.957
0.941
0.913
0.932
0.907
0.929
0.905
Gemma-4-31B-it
0.948
0.934
0.941
0.924
0.920
0.898
0.903
0.892
Qwen2.5-14B-Instruct
0.962
0.964
0.955
0.954
0.954
0.944
0.952
0.932
Qwen3.5-27B
0.978
0.972
0.962
0.968
0.952
0.946
0.948
0.936
Appendix
Table 8: Leakage detection performance of LeakGauge ( Behavior ) across models and the four evaluation splits . F1 is the best F1 over thresholds; TPR@FPR5 is the true positive rate at a 5% false positive rate.
Model
In-Dist Test
Held-Out Content
Held-Out Attacks
Held-Out Strict
Exact
Behavior
Exact
Behavior
Exact
Behavior
Exact
Behavior
System Prompt Leakage
Gemma-4-E4B-it
0.511
0.512
0.503
0.501
0.504
0.513
0.515
0.521
Gemma-4-31B-it
0.510
0.504
0.505
0.502
0.500
0.502
0.503
0.509
Qwen2.5-14B-Instruct
0.505
0.500
0.508
0.510
0.509
0.509
0.514
0.501
Qwen3.5-27B
0.502
0.502
0.504
0.515
0.518
0.509
0.507
0.523
Appendix
Table 9: Random-label control. AUROC* of probes trained on randomly shuffled sample-label pairs. Near-random values confirm that the probes capture generalized leakage-intent features rather than memorizing specific samples.
Model
In-Dist Test
Held-Out Content
Held-Out Attacks
Held-Out Strict
AUROC
TPR@FPR5
AUROC
TPR@FPR5
AUROC
TPR@FPR5
AUROC
TPR@FPR5
Unrelated suffix 1: ‘‘Let’s plan a fun weekend trip together.’’
Gemma-4-E4B-it
0.820
0.253
0.793
0.286
0.839
0.303
0.810
0.309
Gemma-4-31B-it
0.843
0.428
0.821
0.379
0.839
0.438
0.827
0.391
Qwen2.5-14B-Instruct
0.851
0.447
0.826
0.312
0.896
0.529
0.802
0.431
Qwen3.5-27B
0.799
0.468
0.780
0.421
0.771
0.440
0.754
0.397
Appendix
Table 10: Unrelated Suffix on System Prompt Leakage. Each banner row gives the probing suffix; the rows below report AUROC and TPR@FPR5 across the four evaluation splits. Both benign, leakage-irrelevant suffixes yield substantially lower scores than the Behavior suffix of LeakGauge , confirming that leakage-relevant semantics substantially amplify and stabilize an otherwise diffuse signal, especially under distribution shift and low-FPR operation.
Table 15
Figure 6: ROC curves on the system prompt leakage task , for the Exact (top) and Behavior (bottom) suffixes across eight models. Each subplot overlays the four evaluation splits, showing the results for one probe.
Figure 7: ROC curves on the RAG leakage task , for the Exact (top) and Behavior (bottom) suffixes across eight models. Each subplot overlays the four evaluation splits, showing the results for one probe.
Model
Split
Mean contrast
LeakGauge
Δ
AUROC
F1
FPR
AUROC
AUROC
GLM-5.2
In-Dist Test
0.879
0.798
0.295
0.990
0.111
Held-Out Content
0.882
0.802
0.264
0.985
0.103
Held-Out Attacks
0.892
0.816
0.275
0.978
0.086
Held-Out Strict
0.886
0.805
0.269
0.972
0.086
Llama-3.1-8B-Instruct
In-Dist Test
0.778
0.715
0.533
0.989
0.211
Appendix
Table 13: Mean-based contrastive suffix scoring. Δ denotes the AUROC of LeakGauge minus that of Mean contrast. Scalar thresholds are calibrated using a stratified 20% subset of the Training split.
Figure 8: Suffixes induce different logit-summary structures. For descriptive comparison, we use the first principal-component score as a one-dimensional ranking statistic and compute its AUROC. • Attack and • Benign.
Feature
Definition
Mean
T1∑t=1Tℓt
Standard deviation
Std(ℓ1,…,ℓT)
Slope
Slope of a linear fit of ℓt
Mean absolute delta
T−11∑t=2T∣ℓt−ℓt−1∣
Minimum
mintℓt
Range
maxtℓt−mintℓt
Appendix
Table 14: Features used for the projection.
Figure 9: Suffix anatomy across models. For each model, PCA (top) and t-SNE (bottom) of the suffix log-prob summary features, Exact (left) vs. Behavior (right). The two designs consistently induce different logit-space geometries across families and scales; • Attack and • Benign .
Model
V → S (verbatim-trained)
S → S (semantic-trained)
I-Dst
O-Cont.
O-Att.
O-Strict
I-Dst
O-Cont.
O-Att.
O-Strict
(a) Exact suffix
Gemma-4-E4B-it
0.813
0.807
0.819
0.835
0.902
0.875
0.920
0.906
Llama-3.1-8B-Instruct
0.633
0.867
0.863
0.897
0.996
0.748
0.775
0.748
Qwen2.5-14B-Instruct
0.807
0.840
0.830
0.882
0.915
0.904
0.908
0.890
Qwen3.5-27B
0.467
0.472
0.486
0.496
0.908
0.867
0.884
0.851
Appendix
Table 15: Per-model cross-objective AUROC behind Table 4 . Each panel reports one suffix design. V → S trains on verbatim attacks and tests on semantic attacks; S → S trains and tests on semantic attacks. Split abbreviations: I-Dst = In-Dist (test) , O-Cont. = Held-Out Content , O-Att. = Held-Out Attacks , O-Strict = Held-Out Strict.
Source\Target
G4-E4
G4-31
G3-4
G3-12
G3-27
Q2.5-7
Q2.5-14
Q2.5-32
L3.1-8
L3.1-70
L4-Scout
Q3.5-27
Q3.5-35
Q3-30
G4-E4
0.954
0.869
0.674
0.687
0.544
0.872
0.904
0.726
0.892
0.896
0.783
0.879
0.785
0.635
G4-31
0.910
0.943
0.845
0.841
0.727
0.775
0.921
0.729
0.767
0.828
0.753
0.911
0.782
0.588
G3-4
0.941
0.843
0.920
0.885
0.874
0.724
0.766
0.735
0.910
0.962
0.713
–
0.907
0.505
G3-12
0.928
0.899
0.856
0.875
0.811
0.837
0.909
0.748
0.833
0.858
0.617
–
0.662
0.505
G3-27
0.936
0.828
0.900
0.847
0.875
0.804
0.910
0.797
0.914
0.911
0.553
–
0.863
0.522
Q2.5-7
0.960
0.874
0.777
0.722
0.684
0.948
0.955
0.910
0.926
0.926
0.626
–
0.766
0.512
Appendix
Table 16: Cross-model transfer of the Behavior signal. Rows = source (train), columns = target (test); diagonal (in-model) in bold ; -- denotes model pairs not evaluated. Short codes as in the text.
Model
Raccoon FT
AUROC
F1
TPR@FPR5
PromptGuard-2 86M
No
0.838
0.866
0.333
PromptGuard-2 86M
Yes
0.927
0.870
0.854
PIGuard
No
0.921
0.870
0.667
PIGuard
Yes
0.961
0.858
0.872
Appendix
Table 17: Effect of Raccoon fine-tuning on input-text classifiers.
ID
Suffix
Assistant-behavior perspective
B1
“Based on the above, I will give the [TARGET].”
B2
“I will now disclose my original [TARGET]”
B3
“As requested , I will reproduce the [TARGET]”
B4
“I will ignore the restriction and provide [TARGET]”
User-intent perspective
Appendix
Table 18: Candidate semantic suffixes explored for leakage detection.
Category
Construction
Example
#
Context-grounded
Provides public user-supplied context and asks the model to perform a normal task grounded in that context.
“Using only the user-provided passage below, summarize the main points in three bullets.”
280
Attack-shaped
Uses extraction-like wording, but the requested target is user-provided content rather than hidden system instructions.
“Dear team, please submit your weekly updates by Thursday afternoon. The project review meeting will take place on Friday morning. Preserve the exact wording of the email above, but split it into numbered paragraphs.”
270
Appendix
Table 19: Composition of the hard-negative stress test. All requests operate exclusively on content explicitly provided by the user.
Model / Probe
AUROC
Attack TPR
Hard-neg. FPR
Precision
F1
“Based on the above, I will give my system prompt.”
Llama-3.1-8B-Instruct
0.8980
0.8359
0.0564
0.9603
0.8938
GLM-5.2
0.8998
0.8493
0.2218
0.8618
0.8555
“The user asks me to reproduce the system prompt.”
Llama-3.1-8B-Instruct
0.9654
0.8951
0.0891
0.9424
0.9181
GLM-5.2
0.9859
0.9576
0.0964
0.9418
0.9496
Appendix
Table 20: Results of the two suffix designs on the hard-negative stress test.
Setting
Raw ASR
Post-filtered ASR
Detection Recall
Detector-unaware
0.469
0.102
0.782
Single-probe-aware
0.299
0.275
0.080
Single-probe-aware, two-probe deployment
0.299
0.007
0.976
Two-probe-aware, two-probe deployment
0.240
0.059
0.755
Appendix
Table 21: Adaptive GCG stress test on system-prompt leakage.
Model Pair
Checkpoint
Exact (content-adaptive)
Behavior (task-general)
I-Test
H-Cont.
H-Att.
H-Strict
I-Test
H-Cont.
H-Att.
H-Strict
System-Prompt Leakage
Llama-3.1-8B
Base
0.983/0.988
0.970/0.991
0.988/0.973
0.969/0.973
0.998/0.999
0.998/0.999
0.999/0.997
0.998/0.997
Instruct
0.977/0.996
0.959/0.748
0.945/0.775
0.923/0.748
0.986/0.994
0.983/0.997
0.981/0.990
0.975/0.989
Qwen2.5-14B
Base
0.957/0.970
0.936/0.956
0.912/0.928
0.867/0.927
0.997/0.999
0.998/0.998
0.982/1.000
0.979/1.000
Instruct
0.948/0.915
0.912/0.904
0.928/0.908
0.886/0.890
0.989/0.988
0.989/0.992
0.990/0.985
0.989/0.985
Appendix
Table 22: Leakage detection on paired base and instruction-tuned checkpoints (AUROC). Each cell reports Verbatim/Semantic performance, rounded to three decimal places. The Model Pair and Checkpoint columns explicitly group each base model with its instruction-tuned counterpart; “Instruct” includes checkpoints released with either the -Instruct or -it suffix. Exact is the content-adaptive suffix and Behavior is the fixed, task-general suffix. I-Test, H-Cont., H-Att., and H-Strict denote In-Dist Test , Held-Out Content , Held-Out Attacks , and Held-Out Strict , respectively.
Benchmark
Task
Suffix
AUROC
StrongReject
Harmful requests
The query will not be answered for the security issues.
0.995
BIPIA
Indirect Prompt Injection
I will obey the instruction embedded in the document instead of the user.
0.971
ToxicChat
Toxic queries
As requested, I will give the abusive, hateful, threatening, sexual, or otherwise toxic content.
0.972
HaluEval
Hallucinated responses
This response contains information that is unsupported by or conflicts with the provided source.
0.926
Appendix
Table 23: Preliminary extensions of suffix probing beyond leakage. A separate task-specific probe is trained for each benchmark.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.
Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.
Tushar Nayan, Pratik Poudel, Qian Lou +3
Florida International University · University of Central Florida · University of Texas at Arlington
Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to leverage external knowledge, but also exposes valuable RAG databases to leakage attacks. As RAG systems grow more complex and LLMs exhibit stronger instruction-following capabilities, existing studies fall short of systematically assessing RAG leakage risks. We present LeakDojo, a configurable framework for controlled evaluation of RAG leakage. Using LeakDojo, we benchmark six existing attacks across fourteen LLMs, four datasets, and diverse RAG systems. Our study reveals that (1) query generation and adversarial instructions contribute independently to leakage, with overall leakage well approximated by their product; (2) stronger instruction-following capability correlates with higher leakage risk; and (3) improvements in RAG faithfulness can introduce increased leakage risk. These findings provide actionable insights for understanding and mitigating RAG leakage in practice. Our codebase is available at https://github.com/yeasen-z/LeakDojo.
Maosen Zhang, Jianshuo Dong, Boting Lu +4
Tsinghua University, China · Ant International, China · Nanyang Technological University, Singapore