Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.
Figures & tables
Fig. 1: AdvGuard ’s three-stage per-query detection pipeline. The prompt-level detector can block an individual query immediately. Queries that pass this stage update the session graph. Components with sufficient history that pass the statistical filter are classified by the GCN. All allow-or-block decisions are made before model inference.
Fig. 2: Explicit and Implicit Prompt Extraction Attempt Examples.
Benign Stream
AdvQueries
PromptGuard
LLMGuard
AdvGuard
FPR(%) ↓
Recall(%) ↑
F1(%) ↑
FPR(%) ↓
Recall(%) ↑
F1(%) ↑
FPR(%) ↓
Recall(%) ↑
F1(%) ↑
Attack: Zhang (COLM’24)
GSM8K
11234
10.75
96.90
82.91
2.37
92.75
92.73
3.40
99.95
98.11
SST2
12346
8.85
95.75
83.22
0.68
96.40
96.93
0.00
100.00
100.00
SQuAD_v2
12824
5.73
94.50
86.88
2.12
92.25
92.09
0.95
92.13
95.88
Dolly15k
9397
3.92
96.50
94.22
4.47
92.60
91.43
0.00
100.00
100.00
TABLE I: Detection performance of AdvGuard and baseline defenses under mixed-query stream (benign + adversarial).
Dataset
Queries
PromptGuard
LLMGuard
AdvGuard
FPR
Recall
F1
FPR
Recall
F1
FPR
Recall
F1
OASST1
8246
4.74
86.45
91.92
1.47
89.85
94.41
1.34
100.00
99.12
ChatAlpaca
9276
3.77
87.68
92.41
3.98
89.52
93.38
1.76
100.00
98.12
WildChat
8054
10.67
85.92
90.64
3.63
87.75
92.88
3.42
100.00
97.23
TABLE II: Detection performance under multi-turn conversational streams. Metrics are reported in %
Attack
Defense
GPT-2
Falcon3
LLaMA3.2
Qwen2.5
SM ↓
EM ↓
SS ↓
ROUGE-L ↓
SM ↓
EM ↓
SS ↓
ROUGE-L ↓
SM ↓
EM ↓
SS ↓
ROUGE-L ↓
SM ↓
EM ↓
SS ↓
ROUGE-L ↓
PLEAK
No Defense
0.74
0.73
0.92
0.95
0.78
0.73
0.81
0.91
0.92
0.92
0.99
0.98
0.97
0.95
0.99
0.97
Sandwich-based
0.48
0.28
0.82
0.71
0.64
0.55
0.81
0.83
0.61
0.52
0.82
0.71
0.91
0.90
0.94
0.92
Filter-based
0.09
0.01
0.41
0.47
0.20
0.04
0.53
0.64
0.14
0.03
0.48
0.54
0.29
0.11
0.61
0.66
Prompt Obfuscation
0.01
0.01
0.09
0.13
0.01
0.01
0.09
0.19
0.02
0.01
0.17
0.31
0.01
0.00
0.22
0.29
PromptKeeper
0.00
0.00
0.04
0.12
0.00
0.00
0.07
0.18
0.01
0.00
0.14
0.28
0.01
0.00
0.22
0.26
TABLE III: Prompt leakage scores for PLEAK and LeakAgent adaptive querying attacks across victim models and defenses. Lower values indicate stronger leakage mitigation.
Benign
Adversarial
#AdvQry
FPR(%) ↓
Recall(%) ↑
F1(%) ↑
GSM8K
WildJailbreak
543
0.08
96.93
97.99
SST-2
AdvPrompter
2004
1.20
95.00
97.00
SQuAD_v2
SalesforceLeak
841
0.09
96.40
97.94
Dolly15k
AI-curated
958
0.00
99.67
99.35
JailbreakBench
humanAI-Craft
482
0.00
99.64
99.35
TABLE IV: Early-stage prompt-level detection performance of AdvGuard when graph-based structural analysis is disabled.
Fig. 3: Latency and throughput comparison between AdvGuard , PromptGuard , and LLMGuard .
Fig. 4: Latency stability of AdvGuard compared with PromptGuard and LLMGuard . Each point represents the average latency of 200 queries.
Fig. 5: Weighted sum distribution for benign and adversarial queries over time.
Fig. 6: t-SNE projection of benign and adversarial query embeddings.
Dataset
Method
FPR(%) ↓
Δ
Recall(%) ↑
Δ
F1(%) ↑
Δ
OASST1
Prompt Signal
11.50
—
69.00
—
77.00
—
Outlier Filter
0.00
-11.50
100.00
+31.00
100.00
+23.00
Graph Classifier
0.00
-11.50
100.00
+31.00
100.00
+23.00
WildChat
Prompt Signal
18.30
—
69.00
—
74.00
—
Outlier Filter
6.70
-11.60
100.00
+31.00
97.00
+23.00
Graph Classifier
3.18
-15.12
100.00
+31.00
98.00
+24.00
TABLE V: Component-wise ablation of AdvGuard . Δ denotes the relative change from the prompt-only baseline.
Dataset
Size
SimMLP(%)
1-GCN(%)
3-GCN(%)
WildJailbreak
100k
30.13
94.05
99.99
AdvPrompter
15,119
42.61
96.09
100.00
Salesforce-Leak
7,780
43.25
90.34
100.00
JailbreakBench
5,855
45.94
94.90
100.00
Average
–
40.48
93.85
100.00
TABLE VI: Impact of GNN depth on adversarial query detection accuracy over 20 training epochs. All values are reported in percentages.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Held-out family
FPR(%) ↓
Rec(%) ↑
F1(%) ↑
LeakAgent
0.00
100.00
100.00
PLeak
0.00
99.90
99.95
PromptFuzz
0.00
100.00
100.00
Zhang et al.
0.00
100.00
100.00
Mean
0.00
99.98
99.99
Appendix
TABLE VII: Leave-one-attack-family-out test results.
Benign Stream
Queries
FPR(%) ↓
Recall(%) ↑
F1(%) ↑
ChatAlpaca
9276
1.19
99.00
94.42
OASST1
8152
0.10
99.11
99.55
WildChat
8056
3.40
98.00
85.74
Appendix
TABLE VIII: Detection performance of AdvGuard under heterogeneous attack families randomly mixed with benign multi-turn conversations.
Dataset
SM ↓
EM ↓
SS ↓
ROUGE-L ↓
Roles
0.00
0.00
0.12
0.14
SIQA
0.01
0.00
0.33
0.27
Financial
0.02
0.02
0.15
0.11
SQuAD2
0.00
0.00
0.09
0.15
GSM8K
0.01
0.00
0.12
0.16
ChatGPT-Roles
0.02
0.00
0.15
0.17
Appendix
TABLE IX: PLEAK’s performance with AdvGuard using default target system prompts across its evaluation datasets
Data
Method
FPR(%) ↓
Rec(%) ↑
F1(%) ↑
Lat ↓
GSM8K
Top- k
16.35
100.00
93.00
92.48
NN
3.40
99.24
98.00
30.55
SST-2
Top- k
0.00
99.48
100.00
105.48
NN
0.00
99.48
100.00
32.96
Dolly
Top- k
0.00
78.00
88.00
86.48
NN
0.00
99.52
100.00
28.45
Appendix
TABLE X: Design study for similarity graph connectivity (NN vs. top- k ). Latency (Lat) is reported in milliseconds.
Filter
FPR(%) ↓
Rec(%) ↑
F1(%) ↑
Dataset: OASST1
sd≥0.5
34.90
0.47
1.00
sC≥0.5
29.50
99.91
88.00
sC≥0.5 & ∣C∣≥10
5.58
100.00
98.00
Outlier Test
0.00
99.91
100.00
Dataset: WildChat
Appendix
TABLE XI: Comparison between statistical filter and threshold-based filter
Category
Avg Similarity ↓
Diversity ↑
Summarization
0.0821
0.9179
Sentiment
0.3067
0.6933
Conversation
0.2531
0.7469
Reasoning
0.1105
0.8895
QA
0.0718
0.9282
Instruction
0.0426
0.9574
Appendix
TABLE XII: Semantic diversity of baseline prompt initialization.
Threshold (%)
Recall(%) ↑
F1(%) ↑
95
81.00 (-18.2%)
90.00 (-10.0%)
90
94.00 (-5.1%)
97.00 (-3.0%)
85
99.00 (-0.7%)
99.00 (-1.0%)
80
99.00 (—)
100.00 (—)
75
99.00 ( ± 0.0%)
100.00 ( ± 0.0%)
70
100.00 (+1.0%)
98.00 (-2.0%)
Appendix
TABLE XIII: Impact of similarity threshold on adversarial query detection performance. Relative change is computed with respect to the 80% threshold (baseline).
Fig. 7: Impact of minimum input graph size on detection performance.
Fig. 8: Effect of varying Time-to-Detect (TTD) on detection performance, evaluated at the default minimum graph size of 30 nodes.
Graph Size
TP
FP
FN
FPR(%) ↓
Rec(%) ↑
F1(%) ↑
5
600
25
726
4.00
45.20
61.51
10
820
18
506
2.80
61.80
75.79
15
950
12
376
1.90
71.60
83.04
20
1286
2
40
0.40
96.98
98.39
30
1300
0
26
0.00
98.04
99.00
Appendix
TABLE XIV: Impact of minimum interaction history Nmin on detection performance. All performance metrics are reported in percentages.
Component
P50
P95
P99
Embedding
7.50
11.98
14.31
Similarity (NN)
0.93
3.81
4.52
Early-leakage
7.99
12.90
17.50
Structural connectivity
26.78
42.99
46.63
Appendix
TABLE XV: Per-prompt latency breakdown of AdvGuard (ms).
Defense
FPR(%)
Recall(%)
F1(%)
Jailbreaking Attacks
PromptGuard
99.11
11.25
1.66
LLMGuard
12.70
93.65
89.30
AdvGuard
10.11
93.65
91.50
Prompt-Injection Attacks
PromptGuard
94.25
8.54
6.87
Appendix
TABLE XVI: Detection performance against Jailbreaking and prompt-injection-style queries.
LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.
LLMs increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents, to improve generation quality. However, processing these contexts alongside user queries creates an attack surface: adversarial inputs can induce models to disclose them. Prior probing studies suggest that leakage-related signals emerge in hidden states, yet the need to extract these states poses additional deployment challenges. In this paper, we explore whether this internal signal leaves a more accessible ``tell'' before decoding. We propose LeakGauge, which probes this response by appending a suffix that gauges leakage behavior and mapping its prefill token probabilities to an attack-risk score. While a direct gauge uses the initial tokens of confidential content, we find that a content-agnostic one that verbalizes leakage behavior yields more robust signals. Across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T), LeakGauge reaches an AUROC range of 0.944--0.996 on unseen attacks. The signal remains stable when the content changes language or the attack shifts from verbatim to semantic disclosure. By activation-steering interventions, we further show that the risk score is sensitive to an internal leakage-related direction, relating the observable signal to the model's internal representation. In addition, LeakGauge enables an input detector with fewer than 0.5K extra parameters and added latency of 10.34 ms. Code: \href{https://github.com/yeasen-z/LeakGauge}.
Maosen Zhang, Jianshuo Dong, Boting Lu +5
Tsinghua University · Ant International · Nanyang Technological University +1
Prompt attacks, including jailbreaks and prompt injections, pose a critical security risk to Large Language Model (LLM) systems. In production, guardrails must mitigate these attacks under strict low-latency constraints, resulting in a deployment gap in which lightweight classifiers and rule-based systems struggle to generalize under distribution shift, while high-capacity LLM-based judges remain too slow or costly for live enforcement. In this work, we examine whether lightweight, general-purpose LLMs can reliably serve as security judges under real-world production constraints. Through careful prompt and output design, lightweight LLMs are guided through a structured reasoning process involving explicit intent decomposition, safety-signal verification, harm assessment, and self-reflection. We evaluate our method on a curated dataset combining benign queries from real-world chatbots with adversarial prompts generated via automated red teaming (ART), covering diverse and evolving patterns. Our results show that general-purpose LLMs, such as gemini-2.0-flash-lite-001, can serve as effective low-latency judges for live guardrails. This configuration is currently deployed in production as a centralized guardrail service for public service chatbots in Singapore. We additionally evaluate a Mixture-of-Models (MoM) setting to assess whether aggregating multiple LLM judges improves prompt-attack detection performance relative to single-model judges, with only modest gains observed.