SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Yibo Peng; Long Lian; David Wagner; Sizhe Chen

cs.CRAug 21, 2026

SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Authors: Yibo Peng, Long Lian, David Wagner, Sizhe Chen

Organizations: University of California, Berkeley

Abstract

Prompt injection is listed as the #1 threat to AI agents. When an agent accesses external data from websites, files, or emails, an attacker may inject a prompt into the data, saying, "Ignore all prior instructions and perform <an attacker's task>." To prevent arbitrary manipulation of agents, defenders try to train secure LLMs, which, however, still suffer from near 100% attack success rates (ASRs) against adaptive prompt injections. We note that this is because existing defensive finetuning recipes rely on sequence-level feedback signals (in DPO or GRPO). Treating an entire output equally prevents the model from learning precisely which output tokens are insecure. In this paper, we propose Secure On-Policy Distillation (SecOPD) that provides token-level feedback to guide defensive fine-tuning. The LLM receives an injected sample and produces a rollout, whose tokens are scored by the initialization model given the corresponding clean input. With more fine-grained training signals, our defended Qwen3.6-27B achieves a 9.0% ASR against the SoTA PISmith adaptive prompt injections, compared to 94.0% for the prior SoTA, Meta-SecAlign. The obtained security generalizes to domains completely unseen in training: in agentic tool calling, SecOPD achieves a 4.7% ASR compared to 5.5% for Meta-SecAlign. Code and the model are available at https://github.com/pppyb/SecOPD and https://huggingface.co/pybbb/Qwen3.6-27B-SecOPD.

Figures & tables

Figure 1: Motivation for using token-level feedback. When the LLM answers both the trusted prompt and the injection, Sequence-level feedback (a reward) treats the entire response as one unit, which obscures which tokens should be encouraged or discouraged. In contrast, token-level feedback rates each token, assigning high advantage to tokens consistent with the trusted prompt and low advantage to tokens that follow the injection. This finer-grained training signal enables more precise learning of the prompt-injection security policy.

Figure 2: Overview of SecOPD training. The LLM first generates a rollout from an attacked input containing a trusted user prompt and untrusted external data. The same rollout tokens are then scored by the initialization LLM given the corresponding clean input, where the injected instruction is absent. These token-level scores provide the training signal: tokens aligned with the trusted task are encouraged, while tokens following the injected instruction are suppressed. The optimized LLM is the student, and the initialization LLM is kept fixed as the teacher.

Defense $\backslash$ Attack	SEP Static	SEP Basic Adaptive	SEP PISmith Adaptive	AgentDojo Static
Undefended	99.4%	99.0%	97.9%	26.7%
Meta-SecAlign	28.9%	5.5%	94.0%	5.5%
GRPO	15.0%	2.3%	61.2%	0.7%
SecOPD	1.3%	0.2%	9.0%	4.7%

Table 1: Security results measured by attack success rate (ASR,

\downarrow

). PISmith is the strongest adaptive attack in our evaluation; AgentDojo measures whether robustness transfers to tool-use tasks.

Defense	AlpacaEval2	SEP	AgentDojo	MMLU Pro	GPQA Diam	GSM8K	Minrv. Math
Undefended	81.4%	88.0%	92.8%	84.1%	79.8%	97.7%	92.7%
Meta-SecAlign	82.3%	89.3%	92.8%	83.8%	78.3%	97.8%	95.1%
GRPO	76.0%	79.5%	82.5%	83.0%	77.8%	97.4%	85.1%
SecOPD	80.1%	88.6%	90.7%	84.1%	81.3%	97.4%	94.3%

Table 2: Utility scores (

\uparrow

). SecOPD preserves most general capability while providing substantially stronger adaptive robustness than Meta-SecAlign and substantially better utility than GRPO.

Figure 3: Utility–security trade-off under PISmith, the strongest adaptive attack in our evaluation. Utility is SEP Utility and ASR is SEP PISmith Adaptive ASR. The black star marks the ideal point of 0% attack success rate with maximum utility.

Panel of Figure 4 — Figure 4: Case study on a PISmith prompt injection. All three models receive the same input with thinking disabled. Red text marks the injected instruction and text produced in response to it. Ellipses indicate omitted text.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Attack	$n$	Eval.+	Human+	FP	FN	Agree
Straightforward	50	21	21	0	0	100%
Straightforward-before	50	6	6	0	0	100%
Ignore	50	3	3	0	0	100%
Ignore-before	50	3	3	0	0	100%
Completion	50	0	0	0	0	100%
Completion-ignore	50	0	0	0	0	100%

Appendix

Table 3: Human validation of

300

static SEP evaluation labels. The audit is stratified by attack family and includes all evaluation-positive outputs from that run.

Official delimiter	Substitute description	Token ID
<\|im_end\|>	Cyrillic near-neighbor token	214946
<\|im_start\|>	eihna	175459

Appendix

Table 4: Example close-delimiter substitutes for the undefended Qwen3.6-27B model.

Failure pattern	Count	Fraction
Final-answer span switches to an unrelated problem	20	43.5%
Correct answer, but extraction or formatting is marked incorrect	12	26.1%
Incomplete answer finalization	10	21.7%
Genuine reasoning error	4	8.7%

Appendix

Table 5: Manual audit of the

46

Minerva-Math examples answered correctly by the undefended model but incorrectly by SecOPD.

Method	Banking			Slack			Travel			Workspace			All
Method	Utility	AdvUtility	ASR	Utility	AdvUtility	ASR	Utility	AdvUtility	ASR	Utility	AdvUtility	ASR	Utility	AdvUtility	ASR
Undefended	100.0%	77.8%	39.6%	95.2%	72.4%	67.6%	80.0%	55.7%	42.1%	95.0%	77.9%	11.8%	92.8%	74.0%	26.7%
Meta-SecAlign	87.5%	81.3%	13.2%	95.2%	69.5%	31.4%	90.0%	88.6%	0.0%	95.0%	97.7%	0.0%	92.8%	90.7%	5.5%
GRPO	81.3%	61.8%	0.7%	85.7%	61.9%	3.8%	85.0%	87.1%	0.7%	80.0%	85.0%	0.2%	82.5%	79.2%	0.7%
SecOPD (ours)	87.5%	68.1%	4.9%	90.5%	50.5%	24.8%	90.0%	70.0%	6.4%	92.5%	90.7%	0.5%	90.7%	79.8%	4.7%

Appendix

Table 6: Full per-domain AgentDojo evaluation. Utility measures benign user-task success without injection, AdvUtility measures user-task success under injection, and ASR measures attacker-task success under injection.

Method	Straight.	Straight. Before	Ignore	Ignore Before	Completion	Completion Ignore	Combined
Undefended	82.2%	72.0%	79.7%	64.8%	97.1%	96.8%	99.4%
Meta-SecAlign	23.5%	4.1%	5.6%	1.7%	7.4%	1.6%	28.9%
GRPO	2.9%	4.7%	0.3%	1.1%	9.7%	0.1%	15.0%
SecOPD (ours)	0.3%	0.0%	0.5%	0.5%	0.0%	0.0%	1.3%

Appendix

Table 7: Full SEP static attack breakdown over six non-adaptive attack families.

Configuration	Value
Initialization model	Qwen3.6-27B
Teacher model	Frozen initialization model
Trainable parameters	Student LoRA adapter
LoRA rank	128
Learning rate	$1\times 10^{-4}$
Sampling temperature	1.0

Appendix

Table 8: Main SecOPD training configuration.

Explore similar work

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
Jul 3, 2025Sizhe Chen, Arman Zharmagambetov, David Wagner +1LLM Agent Security AI Agent Security Benchmarks
FAIR at Meta · UC Berkeley
PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses
Mar 13, 2026Chenlong Yin, Runpeng Geng, Yanting Wang +1Prompt Injection Attacks on LLMs LLM Red Teaming
The Pennsylvania State University
IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization
May 23, 2026Zixuan Chen, Jiaxiang Chen, Li Luo +4LLM Agent Security Indirect Prompt Injection
1Shanghai Jiao Tong University · 2The University of Hong Kong

Jul 3, 2025J/K move · Enter open · S save

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

Sizhe Chen, Arman Zharmagambetov, David Wagner, Chuan Guo

FAIR at Meta · UC Berkeley

LLM Agent Security AI Agent Security Benchmarks Prompt Injection Defense Adversarial Robustness of Language Models Prompt Injection Attacks on LLMs LLM Agent Training+1

Open paper AskPDF arXiv

Prompt injection attacks, where untrusted data contains an injected prompt to manipulate the system, have been listed as the top security threat to AI agents. By fine-tuning on simulated prompt injections, SecAlign, a leading open defense, reports LLMs with good test-time robustness and negligible benign utility drop. By scaling up training and evaluations, however, we find that SecAlign actually suffers from significant utility degradation, especially in agentic tasks where the threat of prompt injection is prominent. Motivated by this, we propose Meta-SecAlign for utility-preserving defense by (1) randomized injection position during training to avoid shortcut learning and (2) self-generated responses as high-quality in-distribution training labels. Across general knowledge, instruction following, and agentic workflows (on tool-calling and web-navigation), Meta-SecAlign maintains almost all the undefended LLM's utility while achieving better overall security than SecAlign against various static and GCG adaptive attacks. Experiments use Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Qwen3-4B, and Qwen3.6-27B on 6 prompt injection benchmarks including AgentDojo, InjecAgent, WASP, and SEP. Below are links for the code (https://github.com/facebookresearch/Meta_SecAlign), Meta-SecAlign-70B (https://huggingface.co/facebook/Meta-SecAlign-70B), and Meta-SecAlign-8B (https://huggingface.co/facebook/Meta-SecAlign-8B) models.

Abstract

Explore similar work

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization