Detecting and Mitigating Query-Based System Prompt Leakage Attacks

Tushar Nayan; Pratik Poudel; Qian Lou; Yuede Ji; Jason Liu; Ruimin Sun

cs.CROct 26, 2026

Detecting and Mitigating Query-Based System Prompt Leakage Attacks

Authors: Tushar Nayan, Pratik Poudel, Qian Lou, Yuede Ji, Jason Liu, Ruimin Sun

Organizations: Florida International University · University of Central Florida · University of Texas at Arlington

Abstract

Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.

Figures & tables

Fig. 1: AdvGuard ’s three-stage per-query detection pipeline. The prompt-level detector can block an individual query immediately. Queries that pass this stage update the session graph. Components with sufficient history that pass the statistical filter are classified by the GCN. All allow-or-block decisions are made before model inference.

Fig. 2: Explicit and Implicit Prompt Extraction Attempt Examples.

Attack: Zhang (COLM’24)
Benign Stream	AdvQueries	PromptGuard			LLMGuard			AdvGuard
		FPR(%) $\downarrow$	Recall(%) $\uparrow$	F1(%) $\uparrow$	FPR(%) $\downarrow$	Recall(%) $\uparrow$	F1(%) $\uparrow$	FPR(%) $\downarrow$	Recall(%) $\uparrow$	F1(%) $\uparrow$
GSM8K	11234	10.75	96.90	82.91	2.37	92.75	92.73	3.40	99.95	98.11
SST2	12346	8.85	95.75	83.22	0.68	96.40	96.93	0.00	100.00	100.00
SQuAD_v2	12824	5.73	94.50	86.88	2.12	92.25	92.09	0.95	92.13	95.88
Dolly15k	9397	3.92	96.50	94.22	4.47	92.60	91.43	0.00	100.00	100.00

TABLE I: Detection performance of AdvGuard and baseline defenses under mixed-query stream (benign + adversarial).

Dataset	Queries	PromptGuard			LLMGuard			AdvGuard
		FPR	Recall	F1	FPR	Recall	F1	FPR	Recall	F1
OASST1	8246	4.74	86.45	91.92	1.47	89.85	94.41	1.34	100.00	99.12
ChatAlpaca	9276	3.77	87.68	92.41	3.98	89.52	93.38	1.76	100.00	98.12
WildChat	8054	10.67	85.92	90.64	3.63	87.75	92.88	3.42	100.00	97.23

TABLE II: Detection performance under multi-turn conversational streams. Metrics are reported in %

Attack	Defense	GPT-2				Falcon3				LLaMA3.2				Qwen2.5
		SM $\downarrow$	EM $\downarrow$	SS $\downarrow$	ROUGE-L $\downarrow$	SM $\downarrow$	EM $\downarrow$	SS $\downarrow$	ROUGE-L $\downarrow$	SM $\downarrow$	EM $\downarrow$	SS $\downarrow$	ROUGE-L $\downarrow$	SM $\downarrow$	EM $\downarrow$	SS $\downarrow$	ROUGE-L $\downarrow$
PLEAK	No Defense	0.74	0.73	0.92	0.95	0.78	0.73	0.81	0.91	0.92	0.92	0.99	0.98	0.97	0.95	0.99	0.97
	Sandwich-based	0.48	0.28	0.82	0.71	0.64	0.55	0.81	0.83	0.61	0.52	0.82	0.71	0.91	0.90	0.94	0.92
	Filter-based	0.09	0.01	0.41	0.47	0.20	0.04	0.53	0.64	0.14	0.03	0.48	0.54	0.29	0.11	0.61	0.66
	Prompt Obfuscation	0.01	0.01	0.09	0.13	0.01	0.01	0.09	0.19	0.02	0.01	0.17	0.31	0.01	0.00	0.22	0.29
	PromptKeeper	0.00	0.00	0.04	0.12	0.00	0.00	0.07	0.18	0.01	0.00	0.14	0.28	0.01	0.00	0.22	0.26

TABLE III: Prompt leakage scores for PLEAK and LeakAgent adaptive querying attacks across victim models and defenses. Lower values indicate stronger leakage mitigation.

Benign	Adversarial	#AdvQry	FPR(%) $\downarrow$	Recall(%) $\uparrow$	F1(%) $\uparrow$
GSM8K	WildJailbreak	543	0.08	96.93	97.99
SST-2	AdvPrompter	2004	1.20	95.00	97.00
SQuAD_v2	SalesforceLeak	841	0.09	96.40	97.94
Dolly15k	AI-curated	958	0.00	99.67	99.35
JailbreakBench	humanAI-Craft	482	0.00	99.64	99.35

TABLE IV: Early-stage prompt-level detection performance of AdvGuard when graph-based structural analysis is disabled.

Fig. 3: Latency and throughput comparison between AdvGuard , PromptGuard , and LLMGuard .

Fig. 4: Latency stability of AdvGuard compared with PromptGuard and LLMGuard . Each point represents the average latency of 200 queries.

Fig. 5: Weighted sum distribution for benign and adversarial queries over time.

Fig. 6: t-SNE projection of benign and adversarial query embeddings.

Dataset	Method	FPR(%) $\downarrow$	$\Delta$	Recall(%) $\uparrow$	$\Delta$	F1(%) $\uparrow$	$\Delta$
OASST1	Prompt Signal	11.50	—	69.00	—	77.00	—
	Outlier Filter	0.00	-11.50	100.00	+31.00	100.00	+23.00
	Graph Classifier	0.00	-11.50	100.00	+31.00	100.00	+23.00
WildChat	Prompt Signal	18.30	—	69.00	—	74.00	—
	Outlier Filter	6.70	-11.60	100.00	+31.00	97.00	+23.00
	Graph Classifier	3.18	-15.12	100.00	+31.00	98.00	+24.00

TABLE V: Component-wise ablation of AdvGuard .

\Delta

denotes the relative change from the prompt-only baseline.

Dataset	Size	SimMLP(%)	1-GCN(%)	3-GCN(%)
WildJailbreak	100k	30.13	94.05	99.99
AdvPrompter	15,119	42.61	96.09	100.00
Salesforce-Leak	7,780	43.25	90.34	100.00
JailbreakBench	5,855	45.94	94.90	100.00
Average	–	40.48	93.85	100.00

TABLE VI: Impact of GNN depth on adversarial query detection accuracy over 20 training epochs. All values are reported in percentages.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Held-out family	Rec(%) $\uparrow$	F1(%) $\uparrow$
LeakAgent	100.00	100.00
PLeak	99.90	99.95
PromptFuzz	100.00	100.00
Zhang et al.	100.00	100.00
Mean	99.98	99.99

Appendix

TABLE VII: Leave-one-attack-family-out test results.

Benign Stream	Queries	FPR(%) $\downarrow$	Recall(%) $\uparrow$	F1(%) $\uparrow$
ChatAlpaca	9276	1.19	99.00	94.42
OASST1	8152	0.10	99.11	99.55
WildChat	8056	3.40	98.00	85.74

Appendix

TABLE VIII: Detection performance of AdvGuard under heterogeneous attack families randomly mixed with benign multi-turn conversations.

Dataset	SM $\downarrow$	EM $\downarrow$	SS $\downarrow$	ROUGE-L $\downarrow$
Roles	0.00	0.00	0.12	0.14
SIQA	0.01	0.00	0.33	0.27
Financial	0.02	0.02	0.15	0.11
SQuAD2	0.00	0.00	0.09	0.15
GSM8K	0.01	0.00	0.12	0.16
ChatGPT-Roles	0.02	0.00	0.15	0.17

Appendix

TABLE IX: PLEAK’s performance with AdvGuard using default target system prompts across its evaluation datasets

Data	Method	FPR(%) $\downarrow$	Rec(%) $\uparrow$	F1(%) $\uparrow$	Lat $\downarrow$
GSM8K	Top- $k$	16.35	100.00	93.00	92.48
GSM8K	NN	3.40	99.24	98.00	30.55
SST-2	Top- $k$	0.00	99.48	100.00	105.48
SST-2	NN	0.00	99.48	100.00	32.96
Dolly	Top- $k$	0.00	78.00	88.00	86.48
Dolly	NN	0.00	99.52	100.00	28.45

Appendix

TABLE X: Design study for similarity graph connectivity (NN vs. top-

k

). Latency (Lat) is reported in milliseconds.

Dataset: OASST1
Filter	FPR(%) $\downarrow$	Rec(%) $\uparrow$	F1(%) $\uparrow$
$s_{d}\geq 0.5$	34.90	0.47	1.00
$s_{C}\geq 0.5$	29.50	99.91	88.00
$s_{C}\geq 0.5$ & $\|C\|\geq 10$	5.58	100.00	98.00
Outlier Test	0.00	99.91	100.00
Dataset: WildChat

Appendix

TABLE XI: Comparison between statistical filter and threshold-based filter

Category	Avg Similarity $\downarrow$	Diversity $\uparrow$
Summarization	0.0821	0.9179
Sentiment	0.3067	0.6933
Conversation	0.2531	0.7469
Reasoning	0.1105	0.8895
QA	0.0718	0.9282
Instruction	0.0426	0.9574

Appendix

TABLE XII: Semantic diversity of baseline prompt initialization.

Threshold (%)	Recall(%) $\uparrow$	F1(%) $\uparrow$
95	81.00 (-18.2%)	90.00 (-10.0%)
90	94.00 (-5.1%)	97.00 (-3.0%)
85	99.00 (-0.7%)	99.00 (-1.0%)
80	99.00 (—)	100.00 (—)
75	99.00 ( $\pm$ 0.0%)	100.00 ( $\pm$ 0.0%)
70	100.00 (+1.0%)	98.00 (-2.0%)

Appendix

TABLE XIII: Impact of similarity threshold on adversarial query detection performance. Relative change is computed with respect to the 80% threshold (baseline).

Fig. 7: Impact of minimum input graph size on detection performance.

Fig. 8: Effect of varying Time-to-Detect (TTD) on detection performance, evaluated at the default minimum graph size of 30 nodes.

Graph Size	TP	FP	FN	FPR(%) $\downarrow$	Rec(%) $\uparrow$	F1(%) $\uparrow$
5	600	25	726	4.00	45.20	61.51
10	820	18	506	2.80	61.80	75.79
15	950	12	376	1.90	71.60	83.04
20	1286	2	40	0.40	96.98	98.39
30	1300	0	26	0.00	98.04	99.00

Appendix

TABLE XIV: Impact of minimum interaction history

N_{\min}

on detection performance. All performance metrics are reported in percentages.

Component	P50	P95	P99
Embedding	7.50	11.98	14.31
Similarity (NN)	0.93	3.81	4.52
Early-leakage	7.99	12.90	17.50
Structural connectivity	26.78	42.99	46.63

Appendix

TABLE XV: Per-prompt latency breakdown of AdvGuard (ms).

Jailbreaking Attacks
Defense	FPR(%)	Recall(%)	F1(%)
PromptGuard	99.11	11.25	1.66
LLMGuard	12.70	93.65	89.30
AdvGuard	10.11	93.65	91.50
Prompt-Injection Attacks
PromptGuard	94.25	8.54	6.87

Appendix

TABLE XVI: Detection performance against Jailbreaking and prompt-injection-style queries.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 322 papers

Evaluation of Prompt Injection Defenses in Large Language Models
Apr 26, 2026Priyal Deep, Shane Emmons, Amy Fox +4Data Leakage Privacy Leakage in Language Models
Swept AI[1] · University of Michigan
The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges
Aug 18, 2026Maosen Zhang, Jianshuo Dong, Boting Lu +5Privacy Leakage in Language Models LLM Auditing
Tsinghua University · Ant International · Nanyang Technological University +1
Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
Mar 26, 2026Hieu Xuan Le, Benjamin Goh, Quy Anh TangPrompt Injection Detection LLM Guardrails
GovTech, Singapore

Apr 26, 2026J/K move · Enter open · S save

Evaluation of Prompt Injection Defenses in Large Language Models

Priyal Deep, Shane Emmons, Amy Fox, Kyle Bacon+3

Swept AI[1] · University of Michigan

Data Leakage Privacy Leakage in Language Models LLM Security Prompt Injection Defense Prompt Injection Attacks Prompt Injection Attacks on LLMs+1

Open paper AskPDF arXiv

LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.

Abstract

LLM Security

Explore similar work

Evaluation of Prompt Injection Defenses in Large Language Models

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models