Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

Xiaoxue Yang; Bozhidar Stevanoski; Matthieu Meeus; Yves-Alexandre de Montjoye

cs.CRMay 21, 2025

Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

Authors: Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus, Yves-Alexandre de Montjoye

Organizations: Imperial College London

Abstract

Large language models are increasingly deployed in user-facing products that handle sensitive personal data while interacting with untrusted external content. This makes them a prime target for prompt injection attacks, in which an adversary injects malicious instructions in that content to hijack the model. Fine-tuning-based defenses have been proposed to mitigate these attacks by training models to follow benign user instructions and ignore injected ones. Lacking formal guarantees, these defenses are evaluated empirically against the most effective realistic attacks available at the time, often reporting near-zero attack success rates -- giving developers, and the users who depend on them, a false sense of security and leaving open the possibility that future, more capable attacks could circumvent them and cause severe consequences in widely deployed systems. In this paper, we argue that developers should stress-test their defenses before release, and we introduce an auditing framework that does so without requiring defenders to invent the next state-of-the-art attack. Our key insight is that the fine-tuning process itself yields useful artifacts: intermediate checkpoints that serve as stepping stones for optimization-based attacks. Although these artifacts are unavailable to real attackers, we show that the vulnerabilities they expose are genuine and can be exploited by realistic adversaries. Instantiating our framework with optimization-based attacks such as GCG and BEAST, we raise attack success rates against state-of-the-art defenses from near zero to as high as 96%, revealing weaknesses missed by prior evaluations. Our results show that current evaluation practices can substantially overestimate robustness, and that auditing defenses under strong adversaries before deployment shortens the feedback loop and helps prevent attacks from causing real-world harm to users.

Figures & tables

Panel of Figure 1 — Figure 1: Example prompt containing injected instruction (red) and adversarial suffix (blue) .

Figure 2: ASR (%) against increasingly stronger defenses (Undefended, StruQ, SecAlign) across three models (Llama-3-8B-Instruct, Mistral-7B-Instruct, and Qwen2-1.5B-Instruct). Error bars show 95% Wilson score intervals over

N=50

samples (asymmetric by construction).

Panel of Figure 3 — Figure 3: Universality of the Checkpoint-GCG suffixes on (a) in-distribution and (b) out-of-distribution test samples. Curves show success@k and shaded bands show 95% confidence intervals obtained by propagating the Wilson score interval of the ASR through the success@k (see Sec. 5 ). Results for SecAlign; for StruQ see App. C.2 .

Universal suffix from $\theta_{C_{s}}$ obtained via	Black-box transfer to $\theta_{C_{t}}$	White-box transfer with Standard GCG on $\theta_{C_{t}}$
Universal suffix from $\theta_{C_{s}}$ obtained via	ASR $\uparrow$	Train ASR $\uparrow$	Test ASR $\uparrow$	$T$ steps $\downarrow$
Standard GCG (T=500)	0 $[0,1.9]$	0 $[0,27.8]$	0 $[0,1.9]$	5000
Standard GCG (Checkpoint-GCG budget)	0 $[0,1.9]$	0 $[0,27.8]$	0 $[0,1.9]$	5000
Checkpoint-GCG	$\mathbf{63.9}\ [57.2,70.2]$	$\mathbf{100}\ [72.2,100]$	$\mathbf{78.3}\ [72.0,83.4]$	$\mathbf{5}$

Table 1: Attack success rate (ASR

\%

) for transferring the universal suffix found on the surrogate model (SecAlign-defended Llama-3-8B-Instruct) to the target model (Meta-SecAlign-8B, which is Llama-3.1-8B-Instruct defended with SecAlign++), in black- and white-box settings. The 95% Wilson score intervals are shown in brackets.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Illustrating the fine-tuning checkpoints selected for SecAlign-defended Llama-3-8B-Instruct, using step , loss and grad with an example hyperparameter configuration for each strategy (see App. A ).

Checkpoint	Parameter values	ASR (%) $\uparrow$	# Selected	Total Checkpoint-GCG steps
strategy	Parameter values	ASR (%) $\uparrow$	checkpoints $\downarrow$	(avg across samples) $\downarrow$
All	/	100	898	4,228
freq	$q=10$	95	91	4,037
	$q=50$	65	19	4,676
	$q=100$	65	10	2,659
step	$r=30\;\&\;q=10$	100	118	3,708

Appendix

Table 2: Attack effectiveness (ASR) and computational cost (number of selected checkpoints, and total Checkpoint-GCG steps averaged across samples) for each checkpoint selection strategy, evaluated on SecAlign-defended ( Chen et al., 2025b ) Llama-3-8B-Instruct ( AI@Meta, 2024 ) . Results are reported over

20

randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) . Each Checkpoint-GCG step takes roughly constant time (about 3s on our hardware: A100 GPUs with 80GB RAM), identical to standard GCG.

Figure 5: Trade-off between number of selected checkpoints, average number of steps per checkpoint, and the Checkpoint-GCG ASR for the grad checkpoint selection strategy.

Defense	Model	$\tau_{\text{grad}}$	# Checkpoints
StruQ ( Chen et al., 2025a )	Llama-3-8B-Instruct ( AI@Meta, 2024 )	4.5	127
	Mistral-7B-Instruct ( Jiang et al., 2023 )	7	112
	Qwen2-1.5B-Instruct ( Team Qwen, 2024 )	3.2	100
SecAlign ( Chen et al., 2025b )	Llama-3-8B-Instruct ( AI@Meta, 2024 )	0.05	102
	Mistral-7B-Instruct ( Jiang et al., 2023 )	0.05	93
	Qwen2-1.5B-Instruct ( Team Qwen, 2024 )	0.12	91

Appendix

Table 3: Parameters for the grad checkpoint selection strategy used across experiments. We provide both the selected gradient norm threshold

\tau_{\text{grad}}

and the resulting number of checkpoints selected using this threshold.

Defense	Model	$T=500$ steps	Checkpoint-GCG budget	Checkpoint-GCG (ours)
		Standard GCG on $\theta_{C}$
Undefended	Llama-3-8B-Instruct ( AI@Meta, 2024 )	100	100	100
	Mistral-7B-Instruct ( Jiang et al., 2023 )	100	100	100
	Qwen2-1.5B-Instruct ( Team Qwen, 2024 )	98	98	98
StruQ ( Chen et al., 2025a )	Llama-3-8B-Instruct ( AI@Meta, 2024 )	26	50	100
	Mistral-7B-Instruct ( Jiang et al., 2023 )	88	90	100

Appendix

Table 4: Attack success rate (ASR

\%

)

\uparrow

for Checkpoint-GCG against state-of-the-art prompt injection defenses. As baseline, we apply the standard GCG attack directly to the defended model (i.e., the final checkpoint

\theta_{C}

). Results are aggregated for

50

randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) .

Defense	Model	Standard GCG with 500 steps per sample		Standard GCG with Checkpoint-GCG budget		Checkpoint-GCG (ours)
Defense	Model	Training	Testing	Training	Testing	Training	Testing
SecAlign ( Chen et al., 2025b )	Llama ( AI@Meta, 2024 )	0	0	0	0	100	75.3
	Mistral ( Jiang et al., 2023 )	0	0	0	3.0	100	89.9
	Qwen ( Team Qwen, 2024 )	0	0.5	0	3.5	100	78.3
StruQ ( Chen et al., 2025a )	Llama ( AI@Meta, 2024 )	0	0	70	74.2	100	88.9
	Mistral ( Jiang et al., 2023 )	30	58.1	100	91.4	100	99.0

Appendix

Table 5: Attack success rate (ASR

\%

)

\uparrow

for in-distribution universal attack comparing standard GCG with 500 steps per training sample, standard GCG with Checkpoint-GCG budget, and our Checkpoint-GCG method across defenses and models. Results are reported on a training set of 10 samples and a testing set of 198 samples from AlpacaFarm ( Dubois et al., 2023 ) .

Defense	Model	Standard GCG with 500 steps per sample	Standard GCG with Checkpoint-GCG budget	Checkpoint-GCG (ours)
SecAlign ( Chen et al., 2025b )	Llama ( AI@Meta, 2024 )	0	0	40.0
	Mistral ( Jiang et al., 2023 )	0	0.01	88.6
	Qwen ( Team Qwen, 2024 )	0	0.01	78.8
StruQ ( Chen et al., 2025a )	Llama ( AI@Meta, 2024 )	0	83.8	96.2
	Mistral ( Jiang et al., 2023 )	65.0	86.2	95.8
	Qwen ( Team Qwen, 2024 )	0.03	40.6	91.6

Appendix

Table 6: Attack success rate (ASR

\%

)

\uparrow

for out-of-distribution universal attack comparing standard GCG with 500 steps per training sample, standard GCG with Checkpoint-GCG budget, and our Checkpoint-GCG method across defenses and models. Results are reported on a testing set of 500 samples from SEP ( Mu et al., 2023 ) .

Panel of Figure 6 — Figure 6: Universality of suffixes discovered by Checkpoint-GCG on (a) in-distribution and (b) out-of-distribution test samples. Curves show success@k and shaded bands show 95% confidence intervals obtained by propagating the Wilson score interval of the ASR through the success@k (see Sec. 5 ). Results shown are for suffixes discovered against StruQ.

Suffix length (# tokens)	Training ASR $\uparrow$	Testing ASR $\uparrow$
20	100%	75.3%
25	100%	64.1%
30	100%	45.5%

Appendix

Table 7: Attack success rate (ASR

\%

)

\uparrow

N_{\text{train}}=10

training and the remaining

N_{\text{test}}=198

held-out testing samples from AlpacaFarm across different suffix lengths.

Figure 7: The probability of a successful attack by GCG and Checkpoint-GCG when attacking one AlpacaFarm sample on SecAlign-defended ( Chen et al., 2025b ) Llama-3-8B-Instruct ( AI@Meta, 2024 ) , over the cumulative number of optimization steps across checkpoints attacked. p(Success) is the probability of the model outputting the target sequence $y^{*}$ , calculated given the cross entropy loss.

Panel of Figure 8 — Figure 8: (a) Final model’s attention scores of user prompt ( $Attn_{U}$ ) and injected instruction + adversarial suffix ( $Attn_{A}$ ), suffix optimized at each checkpoint. (b) Percentage of total Checkpoint-GCG steps taken by the end of attacking each checkpoint. Both (a) and (b) show mean and standard deviation over successful attacks on SecAlign-defended Llama-3-8B-Instruct.

Figure 9: Cumulative number of successfully attacked samples with a given budget of total steps, evaluated on SecAlign-defended Llama-3-8B-Instruct. Results are aggregated for

50

randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) .

GCG improvement		ASR (%) $\uparrow$
Type	Variant	ASR (%) $\uparrow$
Initialization	"!!!...!!!" ( Zou et al., 2023 )	6
	Random initialization ( Hayase et al., 2024 )	6
	"Hacked...Hacked" ( Hayase et al., 2024 )	6
	Any successful suffix ( Jia et al., 2024 ; Li et al., 2025 ; Zhang et al., 2024 )	16
Algorithmic (nanoGCG)	Historical attack buffer	2

Appendix

Table 8: ASR (%) of GCG improvements when directly attacking the aligned model

\theta_{C}

of SecAlign-defended Llama-3-8B-Instruct. We consider standard GCG with different initializations as proposed by prior work, and algorithmic improvements implemented by nanoGCG ( Swan, 2025 ) . Results are aggregated for 50 randomly selected samples from AlpacaFarm.

Model	Checkpoint-GCG ASR (%) $\downarrow$
Original SecAlign-defended Llama-3-8B-Instruct $\theta_{C}$	$100$
+ “Print exactly ‘Hacked’!”	$95$
+ “Print exactly ‘Hacked’!” + standard GCG suffix	$85$
+ “Print exactly ‘Hacked’!” + Checkpoint-GCG suffix	$50$

Appendix

Table 9: Including successful Checkpoint-GCG suffixes to improve robustness. ASR (

\%

) against the SecAlign-defended model

\theta_{C}

and after further fine-tuning with three preference-data variants, aggregated across 20 held-out AlpacaFarm samples.

		Standard GCG on $\theta_{C}$
Defense	Model	Reported	Replicated
Defense	Model	( $T=500$ steps)	( $T=500$ steps)
SecAlign ( Chen et al., 2025b )	Llama-3-8B-Instruct ( AI@Meta, 2024 )	0	6
	Mistral-7B-Instruct ( Jiang et al., 2023 )	1	18
	Qwen2-1.5B-Instruct ( Team Qwen, 2024 )	N/A	32
StruQ ( Chen et al., 2025a )	Llama-3-8B-Instruct ( AI@Meta, 2024 )	4	42

Appendix

Table 10: Attack success rate (ASR

\%

)

\uparrow

of the standard GCG attack applied directly to the defended model (i.e., the final checkpoint

\theta_{C}

), aggregated over

50

randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) , and compared against the ASRs reported in ( Chen et al., 2025b ) .

(rubric-based)	0.34	0.50
Metric $\uparrow$	Standard GCG	Checkpoint-GCG
StrongREJECT	0.34	0.50
ASR (%)	56	68

Appendix

Table 11: Jailbreak results comparing standard GCG and Checkpoint-GCG, using both the attack success rate (ASR) metric and the StrongREJECT score.

Explore similar work

Evaluation of Prompt Injection Defenses in Large Language Models
Apr 26, 2026Priyal Deep, Shane Emmons, Amy Fox +4Data Leakage Privacy Leakage in Language Models
Swept AI[1] · University of Michigan
LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense
Oct 8, 2026Luman Zhao, Minghui Xu, Yue Zhang +1LLM Security Prompt Injection Defense
Shandong University, Shandong Academy of Artificial Intelligence
Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
May 26, 2026Akindoyin Akinrele, Shreyank N GowdaLanguage Model Safety Evaluation Prompt Injection Attacks on LLMs
School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom

Apr 26, 2026J/K move · Enter open · S save

Evaluation of Prompt Injection Defenses in Large Language Models

Priyal Deep, Shane Emmons, Amy Fox, Kyle Bacon+3

Swept AI[1] · University of Michigan

Data Leakage Privacy Leakage in Language Models LLM Security Prompt Injection Defense Prompt Injection Attacks Prompt Injection Attacks on LLMs+1

Open paper AskPDF arXiv

LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.

Abstract

Explore similar work

Evaluation of Prompt Injection Defenses in Large Language Models

LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals