Large language models are increasingly deployed in user-facing products that handle sensitive personal data while interacting with untrusted external content. This makes them a prime target for prompt injection attacks, in which an adversary injects malicious instructions in that content to hijack the model. Fine-tuning-based defenses have been proposed to mitigate these attacks by training models to follow benign user instructions and ignore injected ones. Lacking formal guarantees, these defenses are evaluated empirically against the most effective realistic attacks available at the time, often reporting near-zero attack success rates -- giving developers, and the users who depend on them, a false sense of security and leaving open the possibility that future, more capable attacks could circumvent them and cause severe consequences in widely deployed systems. In this paper, we argue that developers should stress-test their defenses before release, and we introduce an auditing framework that does so without requiring defenders to invent the next state-of-the-art attack. Our key insight is that the fine-tuning process itself yields useful artifacts: intermediate checkpoints that serve as stepping stones for optimization-based attacks. Although these artifacts are unavailable to real attackers, we show that the vulnerabilities they expose are genuine and can be exploited by realistic adversaries. Instantiating our framework with optimization-based attacks such as GCG and BEAST, we raise attack success rates against state-of-the-art defenses from near zero to as high as 96%, revealing weaknesses missed by prior evaluations. Our results show that current evaluation practices can substantially overestimate robustness, and that auditing defenses under strong adversaries before deployment shortens the feedback loop and helps prevent attacks from causing real-world harm to users.
Figures & tables
Figure 1: Example prompt containing injected instruction (red) and adversarial suffix (blue) .
Figure 2: ASR (%) against increasingly stronger defenses (Undefended, StruQ, SecAlign) across three models (Llama-3-8B-Instruct, Mistral-7B-Instruct, and Qwen2-1.5B-Instruct). Error bars show 95% Wilson score intervals over N=50 samples (asymmetric by construction).
Figure 3: Universality of the Checkpoint-GCG suffixes on (a) in-distribution and (b) out-of-distribution test samples. Curves show success@k and shaded bands show 95% confidence intervals obtained by propagating the Wilson score interval of the ASR through the success@k (see Sec. 5 ). Results for SecAlign; for StruQ see App. C.2 .
Universal suffix from θCs obtained via
Black-box transfer to θCt
White-box transfer with Standard GCG on θCt
ASR ↑
Train ASR ↑
Test ASR ↑
T steps ↓
Standard GCG (T=500)
0 [0,1.9]
0 [0,27.8]
0 [0,1.9]
5000
Standard GCG (Checkpoint-GCG budget)
0 [0,1.9]
0 [0,27.8]
0 [0,1.9]
5000
Checkpoint-GCG
63.9[57.2,70.2]
100[72.2,100]
78.3[72.0,83.4]
5
Table 1: Attack success rate (ASR % ) for transferring the universal suffix found on the surrogate model (SecAlign-defended Llama-3-8B-Instruct) to the target model (Meta-SecAlign-8B, which is Llama-3.1-8B-Instruct defended with SecAlign++), in black- and white-box settings. The 95% Wilson score intervals are shown in brackets.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustrating the fine-tuning checkpoints selected for SecAlign-defended Llama-3-8B-Instruct, using step , loss and grad with an example hyperparameter configuration for each strategy (see App. A ).
Checkpoint
Parameter values
ASR (%) ↑
# Selected
Total Checkpoint-GCG steps
strategy
checkpoints ↓
(avg across samples) ↓
All
/
100
898
4,228
freq
q=10
95
91
4,037
q=50
65
19
4,676
q=100
65
10
2,659
step
r=30&q=10
100
118
3,708
Appendix
Table 2: Attack effectiveness (ASR) and computational cost (number of selected checkpoints, and total Checkpoint-GCG steps averaged across samples) for each checkpoint selection strategy, evaluated on SecAlign-defended ( Chen et al., 2025b ) Llama-3-8B-Instruct ( AI@Meta, 2024 ) . Results are reported over 20 randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) . Each Checkpoint-GCG step takes roughly constant time (about 3s on our hardware: A100 GPUs with 80GB RAM), identical to standard GCG.
Figure 5: Trade-off between number of selected checkpoints, average number of steps per checkpoint, and the Checkpoint-GCG ASR for the grad checkpoint selection strategy.
Defense
Model
τgrad
# Checkpoints
StruQ ( Chen et al., 2025a )
Llama-3-8B-Instruct ( AI@Meta, 2024 )
4.5
127
Mistral-7B-Instruct ( Jiang et al., 2023 )
7
112
Qwen2-1.5B-Instruct ( Team Qwen, 2024 )
3.2
100
SecAlign ( Chen et al., 2025b )
Llama-3-8B-Instruct ( AI@Meta, 2024 )
0.05
102
Mistral-7B-Instruct ( Jiang et al., 2023 )
0.05
93
Qwen2-1.5B-Instruct ( Team Qwen, 2024 )
0.12
91
Appendix
Table 3: Parameters for the grad checkpoint selection strategy used across experiments. We provide both the selected gradient norm threshold τgrad and the resulting number of checkpoints selected using this threshold.
Standard GCG on θC
Defense
Model
T=500 steps
Checkpoint-GCG budget
Checkpoint-GCG (ours)
Undefended
Llama-3-8B-Instruct ( AI@Meta, 2024 )
100
100
100
Mistral-7B-Instruct ( Jiang et al., 2023 )
100
100
100
Qwen2-1.5B-Instruct ( Team Qwen, 2024 )
98
98
98
StruQ ( Chen et al., 2025a )
Llama-3-8B-Instruct ( AI@Meta, 2024 )
26
50
100
Mistral-7B-Instruct ( Jiang et al., 2023 )
88
90
100
Appendix
Table 4: Attack success rate (ASR % ) ↑ for Checkpoint-GCG against state-of-the-art prompt injection defenses. As baseline, we apply the standard GCG attack directly to the defended model (i.e., the final checkpoint θC ). Results are aggregated for 50 randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) .
Defense
Model
Standard GCG with 500 steps per sample
Standard GCG with Checkpoint-GCG budget
Checkpoint-GCG (ours)
Training
Testing
Training
Testing
Training
Testing
SecAlign ( Chen et al., 2025b )
Llama ( AI@Meta, 2024 )
0
0
0
0
100
75.3
Mistral ( Jiang et al., 2023 )
0
0
0
3.0
100
89.9
Qwen ( Team Qwen, 2024 )
0
0.5
0
3.5
100
78.3
StruQ ( Chen et al., 2025a )
Llama ( AI@Meta, 2024 )
0
0
70
74.2
100
88.9
Mistral ( Jiang et al., 2023 )
30
58.1
100
91.4
100
99.0
Appendix
Table 5: Attack success rate (ASR % ) ↑ for in-distribution universal attack comparing standard GCG with 500 steps per training sample, standard GCG with Checkpoint-GCG budget, and our Checkpoint-GCG method across defenses and models. Results are reported on a training set of 10 samples and a testing set of 198 samples from AlpacaFarm ( Dubois et al., 2023 ) .
Defense
Model
Standard GCG with 500 steps per sample
Standard GCG with Checkpoint-GCG budget
Checkpoint-GCG (ours)
SecAlign ( Chen et al., 2025b )
Llama ( AI@Meta, 2024 )
0
0
40.0
Mistral ( Jiang et al., 2023 )
0
0.01
88.6
Qwen ( Team Qwen, 2024 )
0
0.01
78.8
StruQ ( Chen et al., 2025a )
Llama ( AI@Meta, 2024 )
0
83.8
96.2
Mistral ( Jiang et al., 2023 )
65.0
86.2
95.8
Qwen ( Team Qwen, 2024 )
0.03
40.6
91.6
Appendix
Table 6: Attack success rate (ASR % ) ↑ for out-of-distribution universal attack comparing standard GCG with 500 steps per training sample, standard GCG with Checkpoint-GCG budget, and our Checkpoint-GCG method across defenses and models. Results are reported on a testing set of 500 samples from SEP ( Mu et al., 2023 ) .
Figure 6: Universality of suffixes discovered by Checkpoint-GCG on (a) in-distribution and (b) out-of-distribution test samples. Curves show success@k and shaded bands show 95% confidence intervals obtained by propagating the Wilson score interval of the ASR through the success@k (see Sec. 5 ). Results shown are for suffixes discovered against StruQ.
Suffix length (# tokens)
Training ASR ↑
Testing ASR ↑
20
100%
75.3%
25
100%
64.1%
30
100%
45.5%
Appendix
Table 7: Attack success rate (ASR % ) ↑ on Ntrain=10 training and the remaining Ntest=198 held-out testing samples from AlpacaFarm across different suffix lengths.
Figure 7: The probability of a successful attack by GCG and Checkpoint-GCG when attacking one AlpacaFarm sample on SecAlign-defended ( Chen et al., 2025b ) Llama-3-8B-Instruct ( AI@Meta, 2024 ) , over the cumulative number of optimization steps across checkpoints attacked. p(Success) is the probability of the model outputting the target sequence y∗ , calculated given the cross entropy loss.
Figure 8: (a) Final model’s attention scores of user prompt ( AttnU ) and injected instruction + adversarial suffix ( AttnA ), suffix optimized at each checkpoint. (b) Percentage of total Checkpoint-GCG steps taken by the end of attacking each checkpoint. Both (a) and (b) show mean and standard deviation over successful attacks on SecAlign-defended Llama-3-8B-Instruct.
Figure 9: Cumulative number of successfully attacked samples with a given budget of total steps, evaluated on SecAlign-defended Llama-3-8B-Instruct. Results are aggregated for 50 randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) .
GCG improvement
ASR (%) ↑
Type
Variant
Initialization
"!!!...!!!" ( Zou et al., 2023 )
6
Random initialization ( Hayase et al., 2024 )
6
"Hacked...Hacked" ( Hayase et al., 2024 )
6
Any successful suffix ( Jia et al., 2024 ; Li et al., 2025 ; Zhang et al., 2024 )
16
Algorithmic (nanoGCG)
Historical attack buffer
2
Appendix
Table 8: ASR (%) of GCG improvements when directly attacking the aligned model θC of SecAlign-defended Llama-3-8B-Instruct. We consider standard GCG with different initializations as proposed by prior work, and algorithmic improvements implemented by nanoGCG ( Swan, 2025 ) . Results are aggregated for 50 randomly selected samples from AlpacaFarm.
Model
Checkpoint-GCG ASR (%) ↓
Original SecAlign-defended Llama-3-8B-Instruct θC
Table 9: Including successful Checkpoint-GCG suffixes to improve robustness. ASR ( % ) against the SecAlign-defended model θC and after further fine-tuning with three preference-data variants, aggregated across 20 held-out AlpacaFarm samples.
Standard GCG on θC
Defense
Model
Reported
Replicated
( T=500 steps)
( T=500 steps)
SecAlign ( Chen et al., 2025b )
Llama-3-8B-Instruct ( AI@Meta, 2024 )
0
6
Mistral-7B-Instruct ( Jiang et al., 2023 )
1
18
Qwen2-1.5B-Instruct ( Team Qwen, 2024 )
N/A
32
StruQ ( Chen et al., 2025a )
Llama-3-8B-Instruct ( AI@Meta, 2024 )
4
42
Appendix
Table 10: Attack success rate (ASR % ) ↑ of the standard GCG attack applied directly to the defended model (i.e., the final checkpoint θC ), aggregated over 50 randomly selected samples from AlpacaFarm ( Dubois et al., 2023 ) , and compared against the ASRs reported in ( Chen et al., 2025b ) .
Metric ↑
Standard GCG
Checkpoint-GCG
StrongREJECT
0.34
0.50
(rubric-based)
ASR (%)
56
68
Appendix
Table 11: Jailbreak results comparing standard GCG and Checkpoint-GCG, using both the attack success rate (ASR) metric and the StrongREJECT score.
LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.
Large language models (LLMs) perform remarkably well on complex tasks, yet remain highly vulnerable to prompt injection attacks, where malicious instructions embedded in external data can override user intent. Existing defenses remain limited by model fine-tuning requirements, vulnerability to adaptive attacks, or reliance on brittle handcrafted prompts. We argue that a fundamental source of this vulnerability is the lack of an explicit representation of trust provenance. To address this, we introduce Learnable Trust-Boundary Delimiters (LTBD), a lightweight defense that explicitly encodes trust boundaries in the input while keeping the LLM parameters unchanged. LTBD uses a small number of learnable delimiters to distinguish trusted user instructions from untrusted external data, enabling the model to better respect the intended trust hierarchy. Experimental results show that LTBD substantially outperforms inference-time defenses and performs competitively with training-based approaches, while preserving benign-task utility and introducing negligible inference overhead. In particular, LTBD achieves 0.00% ASR on AlpacaFarm and only 0.11-0.19% ASR on TaskTracker. LTBD also remains effective under adaptive attacks, where adversaries have full knowledge of the defense and explicitly attempt to bypass it.
Luman Zhao, Minghui Xu, Yue Zhang +1
Shandong University, Shandong Academy of Artificial Intelligence
Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code will be released.
Akindoyin Akinrele, Shreyank N Gowda
School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom