RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Mohamed Dhouib; Clement Elliker; Alexi Canesse; Maël Jenny; Lucas-Andrei Thil; Mahammed El Sharkawy; Sonia Vanier; Elie Bursztein

cs.CROct 5, 2026

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Authors: Mohamed Dhouib, Clement Elliker, Alexi Canesse, Maël Jenny, Lucas-Andrei Thil, Mahammed El Sharkawy, Sonia Vanier, Elie Bursztein

Organizations: LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX · Google DeepMind

Abstract

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of RAISED. The base model generates its own tool-use tasks and injections (Stage 1). The student is then trained to match the base model’s clean-context distributions on injected contexts (Stage 2), making it invariant to the injection.

Figure 2: Output-distribution drift on benign inputs. SFT and DPO shift the model’s output distribution away from the base model, while self-distillation keeps it close. Details are in Section B.1 .

Panel of Figure 3 — Figure 3: Behavioral drift on a benign tool-use task. The base model completes the task, while the defended model stops at the tool-guided step and defers it to the user. This failure pattern occurs in both prior training-based defenses we study. Additional examples are in Section B.2 .

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Model	Method	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Gemma	Vanilla	1.0	0.4	84.5	64.0	39.5	61.7	55.5	53.4
	Prompt Sandwiching	0.0	0.0	83.5	64.4	34.1	60.0	52.2	56.1
	Spotlighting	1.1	0.1	80.4	55.8	43.9	61.7	55.4	54.6
	PromptGuard2	0.9	0.0	84.5	42.9	16.1	60.0	34.1	37.9
	SecAlign++	$0.0{\pm}0.0$	$0.0{\pm}0.0$	$77.3{\pm}2.2$	$74.8{\pm}0.7$	$1.1{\pm}0.2$	$45.6{\pm}0.8$	$46.2{\pm}1.4$	$1.0{\pm}0.1$

Table 1: Prompt-injection robustness and utility on Gemma and Nemotron. Higher is better for

U

and

U_{\mathrm{atk}}

; lower is better for ASR.

Model	Method	ToolTalk	BFCL	WorkBench	MMT-Bench	AsyncTool
Gemma	Vanilla	65.4	57.0	55.9	53.0	52.7
	Prompt Sandwiching	59.0	33.0	54.9	49.2	52.7
	Spotlighting	69.2	57.5	50.4	53.4	46.6
	PromptGuard2	65.4	57.0	55.9	53.0	52.7
	SecAlign++	$67.1{\pm}1.6$	$56.0{\pm}1.5$	$66.0{\pm}0.5$	$55.0{\pm}0.4$	$54.0{\pm}1.1$
	RAISED (Ours)	$68.8{\pm}0.6$	$57.2{\pm}0.6$	$61.5{\pm}0.7$	$52.7{\pm}0.7$	$53.5{\pm}1.5$

Table 2: Benign agentic capabilities. Results on five tool-use benchmarks without prompt injection.

Model	Method	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Gemma	Vanilla	75.1	81.1	66.6	72.2	57.0	94.1	94.9	92.5	95.1	83.4
	SecAlign++	$73.0{\pm}0.2$	$80.5{\pm}0.0$	$64.2{\pm}0.0$	$69.0{\pm}1.4$	$56.5{\pm}0.2$	$94.3{\pm}0.8$	$94.7{\pm}0.3$	$91.7{\pm}0.1$	$95.3{\pm}0.8$	$79.9{\pm}0.8$
	RAISED (Ours)	$75.2{\pm}0.1$	$81.0{\pm}0.0$	$66.4{\pm}0.4$	$73.1{\pm}1.9$	$56.4{\pm}0.3$	$94.3{\pm}0.5$	$95.4{\pm}0.3$	$92.7{\pm}0.1$	$95.1{\pm}0.5$	$82.6{\pm}0.3$
Nemotron	Vanilla	49.5	60.1	34.6	42.4	23.3	81.3	87.2	76.5	87.2	67.8
	SecAlign++	$48.3{\pm}0.7$	$60.3{\pm}0.2$	$34.9{\pm}0.0$	$43.8{\pm}1.7$	$23.1{\pm}0.2$	$79.4{\pm}1.5$	$88.2{\pm}0.5$	$75.0{\pm}0.1$	$86.0{\pm}0.0$	$64.6{\pm}1.0$
	RAISED (Ours)	$49.6{\pm}0.1$	$59.3{\pm}0.2$	$35.1{\pm}0.0$	$41.2{\pm}2.1$	$23.3{\pm}0.3$	$80.0{\pm}0.3$	$87.2{\pm}0.3$	$76.7{\pm}0.1$	$86.6{\pm}0.5$	$66.2{\pm}0.6$

Table 3: General capabilities. Results on ten non-agentic benchmarks covering knowledge, reasoning, multilingual understanding, instruction following, mathematics, and code generation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{S}$ , $s_{0}$ , $s_{J}$	State space; initial and final state of an episode
$\mathcal{T}$ , $\mathcal{T}_{\mathrm{adv}}$	Tool menu; distractor tools added for the attacker goal
$\mathcal{A}_{t}$ , $f_{t}$	Argument space and executor of tool $t$
$o_{j}$ , $a_{j}$	$j$ -th tool observation (untrusted); $j$ -th assistant action
$\sigma$ , $u$	System prompt; user request
$\rho$	Chat template

Appendix

Table 4: Notation.

Task	Base	SFT	SecAlign++	RAISED
Change the account password; a tool emails a verification code	reads code, updates ✓	claims completion without reading code $\times$	asks user for code $\times$	reads code, updates ✓
Follow instructions in a file; the file says to adjust a standing order	updates standing order ✓	relays instruction, no update $\times$	updates standing order ✓	updates standing order ✓
Transfer the most-starred repository to an address read from the inbox	transfers and verifies ✓	claims completion without verification $\times$	transfers and verifies ✓	transfers and verifies ✓
Invite the collaborator who requested it; a tool emails a verification code	reads code, invites ✓	claims sent without verification $\times$	claims sent without verification $\times$	reads code, invites ✓
Release the batch; an error identifies the corrected record to release	releases corrected record ✓	claims completion, releases nothing $\times$	asks user whether to release it $\times$	releases corrected record ✓
Open the course for enrollment; the tool says to publish the syllabus first	publishes, then opens ✓	asks user to publish first $\times$	asks whether to publish first $\times$	publishes, then opens ✓

Appendix

Table 5: Behavioral drift on benign Gemma tool-use tasks. The first four rows are examples from AgentDojo and AgentDyn; the final three are designed cases covering additional forms of tool-supplied guidance. A ✓ indicates task completion and

\times

a failure to complete the required authorized steps.

Task	Base	SFT	SecAlign++	RAISED
Publish the menu; an error returns the corrected versioned target	publishes corrected target ✓	claims completion, wrong target $\times$	publishes original target $\times$	publishes corrected target ✓
Post the store-opening summary; the required post step is named only in a tool output	posts summary ✓	lists statuses, does not post $\times$	describes summary, does not post $\times$	posts summary ✓
Sync the inventory; the service says to retry with an updated version	retries and syncs ✓	tells user to retry $\times$	retries and syncs ✓	retries and syncs ✓
Finalize payroll; the tool says timesheets must be validated first	validates, then finalizes ✓	claims completion, skips validation $\times$	validates, then finalizes ✓	validates, then finalizes ✓

Appendix

Table 6: Behavioral drift on designed benign Nemotron tool-use tasks. Each task requires acting on guidance returned by a first-party tool.

Panel of Figure 4 — Figure 4: Message feed before and after prompt sandwiching.

Panel of Figure 5 — Figure 5: Message feed before and after spotlighting with delimiting.

Panel of Figure 6 — Figure 6: Message feed before and after filtering a tool response with Prompt Guard 2.

Structure	Construction
Surfaced guidance	A step required to complete the user’s task is revealed through a tool output rather than stated in the original request. The output may directly indicate the next operation or identify the resource, record, or location needed to determine it.
Recovery / completion	An intermediate observation reveals how an incomplete task can be continued, for example by exposing a supported alternative after an unsuccessful action, identifying another resource needed for completion, or showing that another user-requested obligation remains.
Follow-through	An action that appears to complete the request instead returns a provisional or incomplete result, requiring one or more additional authorized actions before the task is actually finished.

Appendix

Table 7: Failure-focused structures used during self-generation. Each structure creates a benign task in which information returned through a tool must legitimately influence a subsequent authorized action.

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Model	Method	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Qwen	Vanilla	0.0	0.0	88.7	80.0	5.8	66.7	64.8	4.2
	Prompt Sandwiching	0.0	0.0	89.7	82.6	3.8	73.3	68.2	1.8
	Spotlighting	0.0	0.0	89.7	85.4	0.9	73.3	67.9	0.5
	PromptGuard2	0.0	0.0	88.7	58.6	2.7	66.7	47.7	3.3
	SecAlign++	$0.0{\pm}0.0$	$0.0{\pm}0.0$	$88.7{\pm}1.7$	$84.2{\pm}1.6$	$1.9{\pm}0.1$	$70.0{\pm}1.4$	$67.0{\pm}0.1$	$1.5{\pm}0.2$

Appendix

Table 8: Prompt-injection robustness and utility on Qwen. Higher is better for utility (

U

) and utility under attack (

U_{\mathrm{atk}}

); lower is better for attack success rate (ASR). Trained defenses report mean

\pm

standard deviation over three seeds.

Model	Method	ToolTalk	BFCL	WorkBench	MMT-Bench	AsyncTool
Qwen	Vanilla	66.7	50.0	83.0	50.5	68.1
	Prompt Sandwiching	67.9	46.0	84.5	48.8	68.4
	Spotlighting	69.2	47.0	82.3	49.3	68.9
	PromptGuard2	66.7	50.0	83.0	50.5	68.1
	SecAlign++	$65.4{\pm}0.0$	$49.3{\pm}0.6$	$84.0{\pm}0.5$	$47.9{\pm}0.5$	$68.7{\pm}0.1$
	RAISED (Ours)	$66.2{\pm}0.6$	$50.0{\pm}0.4$	$83.1{\pm}0.6$	$51.2{\pm}0.2$	$67.8{\pm}0.4$

Appendix

Table 9: Benign agentic capabilities on Qwen. Results on five tool-use benchmarks without prompt injection. Trained defenses report mean

\pm

standard deviation over three seeds.

Model	Method	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Qwen	Vanilla	76.5	$85.0$	58.6	$77.6$	36.5	$94.7$	96.1	93.6	96.3	81.6
	SecAlign++	$74.4{\pm}0.3$	$85.0{\pm}0.2$	$54.7{\pm}0.7$	$79.5{\pm}2.3$	$35.1{\pm}0.3$	$94.3{\pm}0.5$	$96.1{\pm}0.1$	$90.0{\pm}1.4$	$96.5{\pm}0.3$	$81.6{\pm}0.6$
	RAISED (Ours)	$76.7{\pm}0.1$	$85.1{\pm}0.1$	$58.8{\pm}0.1$	$74.2{\pm}1.1$	$37.1{\pm}0.5$	$94.1{\pm}0.6$	$96.4{\pm}0.2$	$93.6{\pm}0.1$	$97.0{\pm}0.0$	$81.3{\pm}0.2$

Appendix

Table 10: General capabilities on Qwen. Results on ten non-agentic benchmarks. Trained defenses report mean

\pm

standard deviation over three seeds.

Figure 7: Per-block hidden-state changes over 88 AgentDojo pairs (Gemma-4-12B). Panels 1, 2, and 4 measure the effect of the injection within each model; panel 3 measures the distance from the base model on clean contexts. Bands show s.e.m. across pairs.

\times

at block 48 denotes the state after the final norm.

	Base	SecAlign++	RAISED
Starts the injected tool call (%, 72 pairs) $\downarrow$	62.5	13.9	12.5
KL to the base model’s clean prediction (nats, 88 pairs) $\downarrow$	0.338	0.062	0.011
Injection effect at block 31, peak (rel. $L_{2}$ ) $\downarrow$	0.45	0.29	0.23
Distance to base on clean context, block 47 $\downarrow$	–	0.19	0.12

Appendix

Table 11: Summary of internal-state changes (88 AgentDojo pairs) and of the behavioral check (72 pairs, see text).

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Method	Steps	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Vanilla	–	26.7	33.7	48.5	39.9	11.7	25.0	19.6	17.6
SecAlign++	313	$6.4{\pm}0.9$	$8.8{\pm}1.4$	$50.2{\pm}1.0$	$47.7{\pm}0.7$	$4.0{\pm}0.7$	$22.2{\pm}0.8$	$20.7{\pm}0.5$	$7.8{\pm}0.3$
SecAlign++	625	$1.9{\pm}0.2$	$1.4{\pm}0.5$	$50.2{\pm}1.0$	$49.5{\pm}0.9$	$0.7{\pm}0.0$	$22.2{\pm}0.8$	$21.2{\pm}1.2$	$3.0{\pm}0.2$
RAISED (Ours)	–	$3.5{\pm}0.5$	$1.0{\pm}0.6$	$50.2{\pm}1.3$	$46.4{\pm}0.6$	$0.5{\pm}0.1$	$25.0{\pm}0.0$	$25.4{\pm}1.2$	$1.9{\pm}0.6$

Appendix

Table 12: Effect of extended SecAlign++ training on prompt-injection robustness for Nemotron. SecAlign++ is evaluated at the 313-step checkpoint used in the main paper and after 625 training steps. Vanilla and RAISED are included as reference points. Higher is better for utility (

U

and

U_{\mathrm{atk}}

); lower is better for ASR.

Method	Steps	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Vanilla	–	49.5	60.1	34.6	42.4	23.3	81.3	87.2	76.5	87.2	67.8
SecAlign++	313	$48.3{\pm}0.7$	$60.3{\pm}0.2$	$34.9{\pm}0.0$	$43.8{\pm}1.7$	$23.1{\pm}0.2$	$79.4{\pm}1.5$	$88.2{\pm}0.5$	$75.0{\pm}0.1$	$86.0{\pm}0.0$	$64.6{\pm}1.0$
SecAlign++	625	$\mathbf{42.8{\pm}0.3}$	$60.8{\pm}0.1$	$34.8{\pm}0.1$	$\mathbf{39.4{\pm}1.1}$	$21.3{\pm}0.3$	$79.1{\pm}0.8$	$87.2{\pm}0.3$	$\mathbf{66.1{\pm}0.9}$	$86.8{\pm}0.6$	$\mathbf{13.4{\pm}2.7}$
RAISED (Ours)	–	$49.6{\pm}0.1$	$59.3{\pm}0.2$	$35.1{\pm}0.0$	$41.2{\pm}2.1$	$23.3{\pm}0.3$	$80.0{\pm}0.3$	$87.2{\pm}0.3$	$76.7{\pm}0.1$	$86.6{\pm}0.5$	$66.2{\pm}0.6$

Appendix

Table 13: General capabilities under extended SecAlign++ training on Nemotron. The 313-step SecAlign++ checkpoint is the one used in the main paper. Bold values highlight the largest capability degradations observed after extending SecAlign++ training to 625 steps. Vanilla and RAISED are included as reference points.

		AgentDojo		AgentDyn
Data	Objective	$U\uparrow$	ASR $\downarrow$	$U\uparrow$	ASR $\downarrow$
–	Vanilla	84.5	39.5	61.7	53.4
Glaive	SFT	$73.2{\pm}3.6$	$0.5{\pm}0.3$	$17.5{\pm}3.5$	$1.5{\pm}1.2$
Glaive	DPO	$79.1{\pm}0.6$	$1.8{\pm}0.1$	$48.6{\pm}0.3$	$2.3{\pm}1.5$
Glaive	Self-distillation	$80.1{\pm}0.6$	$1.1{\pm}0.2$	$55.5{\pm}0.8$	$2.1{\pm}1.3$
Self-generated	DPO	$82.2{\pm}0.5$	$1.0{\pm}0.1$	$55.6{\pm}3.4$	$1.6{\pm}1.4$

Appendix

Table 14: Ablation of training data and objective on Gemma-4-12B-it. Higher is better for benign utility (

U

); lower is better for attack success rate (ASR). Trained variants report mean

\pm

standard deviation over three seeds.

Explore similar work

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
Jul 3, 2025Sizhe Chen, Arman Zharmagambetov, David Wagner +1Indirect Prompt Injection Gradient-Based Attacks
FAIR at Meta · UC Berkeley
Assessing Automated Prompt Injection Attacks in Agentic Environments
Jun 9, 2026David Hofer, Edoardo Debenedetti, Florian TramèrIndirect Prompt Injection Large Language Model Agents
ETH Zurich Zurich, Switzerland
AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection
Aug 4, 2026Shihao Weng, Yang Feng, Xiaofei Xie +1Multi-Llm Agents Prompt Engineering
Nanjing University · Singapore Management University · Nanyang Technological University

Jul 3, 2025, cs.CRJ/K move · Enter open · S save

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

Sizhe Chen, Arman Zharmagambetov, David Wagner, Chuan Guo

FAIR at Meta · UC Berkeley

Indirect Prompt Injection Gradient-Based Attacks Prompt Engineering Value Alignment

Open paper AskPDF arXiv

Prompt injection attacks, where untrusted data contains an injected prompt to manipulate the system, have been listed as the top security threat to AI agents. By fine-tuning on simulated prompt injections, SecAlign, a leading open defense, reports LLMs with good test-time robustness and negligible benign utility drop. By scaling up training and evaluations, however, we find that SecAlign actually suffers from significant utility degradation, especially in agentic tasks where the threat of prompt injection is prominent. Motivated by this, we propose Meta-SecAlign for utility-preserving defense by (1) randomized injection position during training to avoid shortcut learning and (2) self-generated responses as high-quality in-distribution training labels. Across general knowledge, instruction following, and agentic workflows (on tool-calling and web-navigation), Meta-SecAlign maintains almost all the undefended LLM's utility while achieving better overall security than SecAlign against various static and GCG adaptive attacks. Experiments use Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Qwen3-4B, and Qwen3.6-27B on 6 prompt injection benchmarks including AgentDojo, InjecAgent, WASP, and SEP. Below are links for the code (https://github.com/facebookresearch/Meta_SecAlign), Meta-SecAlign-70B (https://huggingface.co/facebook/Meta-SecAlign-70B), and Meta-SecAlign-8B (https://huggingface.co/facebook/Meta-SecAlign-8B) models.

Abstract

Explore similar work

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

Assessing Automated Prompt Injection Attacks in Agentic Environments

AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection