Large language models (LLMs) perform remarkably well on complex tasks, yet remain highly vulnerable to prompt injection attacks, where malicious instructions embedded in external data can override user intent. Existing defenses remain limited by model fine-tuning requirements, vulnerability to adaptive attacks, or reliance on brittle handcrafted prompts. We argue that a fundamental source of this vulnerability is the lack of an explicit representation of trust provenance. To address this, we introduce Learnable Trust-Boundary Delimiters (LTBD), a lightweight defense that explicitly encodes trust boundaries in the input while keeping the LLM parameters unchanged. LTBD uses a small number of learnable delimiters to distinguish trusted user instructions from untrusted external data, enabling the model to better respect the intended trust hierarchy. Experimental results show that LTBD substantially outperforms inference-time defenses and performs competitively with training-based approaches, while preserving benign-task utility and introducing negligible inference overhead. In particular, LTBD achieves 0.00% ASR on AlpacaFarm and only 0.11-0.19% ASR on TaskTracker. LTBD also remains effective under adaptive attacks, where adversaries have full knowledge of the defense and explicitly attempt to bypass it.
Figures & tables
Figure 1: Overview of Learnable Trust-Boundary Delimiters (LTBD) . (a) Prompt injections embedded in external data steer the LLM away from the original task. (b) LTBD insert four learnable delimiters to explicitly mark the trusted instruction region and the untrusted data region. The delimiter delimiter embeddings are the only trainable parameters, while the base LLM fθ remains frozen. During inference the input is wrapped with the learned delimiters, enabling the model to follow the trusted instruction and treat the (potentially attacked) external data as data rather than a new instruction.
Model
Defense Type
Method
AlpacaFarm [ 15 ]
SEP [ 18 ]
TaskTracker [ 17 ]
CyberSecEval2 [ 19 ]
WinRate {\color[rgb]{1,0,0}\uparrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
WinRate {\color[rgb]{1,0,0}\uparrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
Llama3-8B- Instruct
No Defense
None
33.82
55.29
59.72
74.47
16.4
45.45
Prompting-based
TextGrad [ 20 ]
22.9
0.00
1.1
3.5
0.25
1.8
Reminder [ 13 ]
24.4
34.6
48.3
75.2
19.8
43.6
Sandwich [ 14 ]
26.8
56.7
46.9
63.4
5.5
41.8
DefensiveToken [ 16 ]
33.46
0.48
58.60
3.20
0.27
3.64
Table 1: Comparison of utility (WinRate % {\color[rgb]{1,0,0}\uparrow} ) and security (ASR % {\color[rgb]{0,0.5469,0.2695}\downarrow} ) across prompt injection defenses.
Model
Method
AlpacaFarm [ 15 ]
SEP [ 18 ]
WinRate {\color[rgb]{1,0,0}\uparrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
ASR {\color[rgb]{0,0.5469,0.2695}\downarrow}
Llama3-8B- Instruct
Prefix + CE
27.60
6.73
15.43
Boundary + CE
34.51
0.00
2.34
Boundary + CE + Pref (ours)
35.73
0.00
2.04
Llama3.1-8B- Instruct
Prefix + CE
27.50
23.08
33.81
Boundary + CE
30.98
0.48
3.43
Table 2: Ablation study on trust boundary placement and loss type.
Model
Plain Injection
Adaptive Attack
Llama3-8B-Instruct
0.00
0.00
Llama3.1-8B-Instruct
0.48
0.48
Falcon3-7B-Instruct
0.00
0.00
Qwen2.5-7B-Instruct
0.00
0.48
Table 3: ASR (% {\color[rgb]{0,0.5469,0.2695}\downarrow} ) under Adaptive Attacks.