RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Mohamed Dhouib; Clement Elliker; Alexi Canesse; Maël Jenny; Lucas-Andrei Thil; Mahammed El Sharkawy; Sonia Vanier; Elie Bursztein

cs.CROct 5, 2026

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Authors: Mohamed Dhouib, Clement Elliker, Alexi Canesse, Maël Jenny, Lucas-Andrei Thil, Mahammed El Sharkawy, Sonia Vanier, Elie Bursztein

Organizations: LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX · Google DeepMind

Abstract

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of RAISED. The base model generates its own tool-use tasks and injections (Stage 1). The student is then trained to match the base model’s clean-context distributions on injected contexts (Stage 2), making it invariant to the injection.

Figure 2: Output-distribution drift on benign inputs. SFT and DPO shift the model’s output distribution away from the base model, while self-distillation keeps it close. Details are in Section B.1 .

Panel of Figure 3 — Figure 3: Behavioral drift on a benign tool-use task. The base model completes the task, while the defended model stops at the tool-guided step and defers it to the user. This failure pattern occurs in both prior training-based defenses we study. Additional examples are in Section B.2 .

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Model	Method	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Gemma	Vanilla	1.0	0.4	84.5	64.0	39.5	61.7	55.5	53.4
	Prompt Sandwiching	0.0	0.0	83.5	64.4	34.1	60.0	52.2	56.1
	Spotlighting	1.1	0.1	80.4	55.8	43.9	61.7	55.4	54.6
	PromptGuard2	0.9	0.0	84.5	42.9	16.1	60.0	34.1	37.9
	SecAlign++	$0.0{\pm}0.0$	$0.0{\pm}0.0$	$77.3{\pm}2.2$	$74.8{\pm}0.7$	$1.1{\pm}0.2$	$45.6{\pm}0.8$	$46.2{\pm}1.4$	$1.0{\pm}0.1$

Table 1: Prompt-injection robustness and utility on Gemma and Nemotron. Higher is better for

U

and

U_{\mathrm{atk}}

; lower is better for ASR.

Model	Method	ToolTalk	BFCL	WorkBench	MMT-Bench	AsyncTool
Gemma	Vanilla	65.4	57.0	55.9	53.0	52.7
	Prompt Sandwiching	59.0	33.0	54.9	49.2	52.7
	Spotlighting	69.2	57.5	50.4	53.4	46.6
	PromptGuard2	65.4	57.0	55.9	53.0	52.7
	SecAlign++	$67.1{\pm}1.6$	$56.0{\pm}1.5$	$66.0{\pm}0.5$	$55.0{\pm}0.4$	$54.0{\pm}1.1$
	RAISED (Ours)	$68.8{\pm}0.6$	$57.2{\pm}0.6$	$61.5{\pm}0.7$	$52.7{\pm}0.7$	$53.5{\pm}1.5$

Table 2: Benign agentic capabilities. Results on five tool-use benchmarks without prompt injection.

Model	Method	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Gemma	Vanilla	75.1	81.1	66.6	72.2	57.0	94.1	94.9	92.5	95.1	83.4
	SecAlign++	$73.0{\pm}0.2$	$80.5{\pm}0.0$	$64.2{\pm}0.0$	$69.0{\pm}1.4$	$56.5{\pm}0.2$	$94.3{\pm}0.8$	$94.7{\pm}0.3$	$91.7{\pm}0.1$	$95.3{\pm}0.8$	$79.9{\pm}0.8$
	RAISED (Ours)	$75.2{\pm}0.1$	$81.0{\pm}0.0$	$66.4{\pm}0.4$	$73.1{\pm}1.9$	$56.4{\pm}0.3$	$94.3{\pm}0.5$	$95.4{\pm}0.3$	$92.7{\pm}0.1$	$95.1{\pm}0.5$	$82.6{\pm}0.3$
Nemotron	Vanilla	49.5	60.1	34.6	42.4	23.3	81.3	87.2	76.5	87.2	67.8
	SecAlign++	$48.3{\pm}0.7$	$60.3{\pm}0.2$	$34.9{\pm}0.0$	$43.8{\pm}1.7$	$23.1{\pm}0.2$	$79.4{\pm}1.5$	$88.2{\pm}0.5$	$75.0{\pm}0.1$	$86.0{\pm}0.0$	$64.6{\pm}1.0$
	RAISED (Ours)	$49.6{\pm}0.1$	$59.3{\pm}0.2$	$35.1{\pm}0.0$	$41.2{\pm}2.1$	$23.3{\pm}0.3$	$80.0{\pm}0.3$	$87.2{\pm}0.3$	$76.7{\pm}0.1$	$86.6{\pm}0.5$	$66.2{\pm}0.6$

Table 3: General capabilities. Results on ten non-agentic benchmarks covering knowledge, reasoning, multilingual understanding, instruction following, mathematics, and code generation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{S}$ , $s_{0}$ , $s_{J}$	State space; initial and final state of an episode
$\mathcal{T}$ , $\mathcal{T}_{\mathrm{adv}}$	Tool menu; distractor tools added for the attacker goal
$\mathcal{A}_{t}$ , $f_{t}$	Argument space and executor of tool $t$
$o_{j}$ , $a_{j}$	$j$ -th tool observation (untrusted); $j$ -th assistant action
$\sigma$ , $u$	System prompt; user request
$\rho$	Chat template

Appendix

Table 4: Notation.

Task	Base	SFT	SecAlign++	RAISED
Change the account password; a tool emails a verification code	reads code, updates ✓	claims completion without reading code $\times$	asks user for code $\times$	reads code, updates ✓
Follow instructions in a file; the file says to adjust a standing order	updates standing order ✓	relays instruction, no update $\times$	updates standing order ✓	updates standing order ✓
Transfer the most-starred repository to an address read from the inbox	transfers and verifies ✓	claims completion without verification $\times$	transfers and verifies ✓	transfers and verifies ✓
Invite the collaborator who requested it; a tool emails a verification code	reads code, invites ✓	claims sent without verification $\times$	claims sent without verification $\times$	reads code, invites ✓
Release the batch; an error identifies the corrected record to release	releases corrected record ✓	claims completion, releases nothing $\times$	asks user whether to release it $\times$	releases corrected record ✓
Open the course for enrollment; the tool says to publish the syllabus first	publishes, then opens ✓	asks user to publish first $\times$	asks whether to publish first $\times$	publishes, then opens ✓

Appendix

Table 5: Behavioral drift on benign Gemma tool-use tasks. The first four rows are examples from AgentDojo and AgentDyn; the final three are designed cases covering additional forms of tool-supplied guidance. A ✓ indicates task completion and

\times

a failure to complete the required authorized steps.

Task	Base	SFT	SecAlign++	RAISED
Publish the menu; an error returns the corrected versioned target	publishes corrected target ✓	claims completion, wrong target $\times$	publishes original target $\times$	publishes corrected target ✓
Post the store-opening summary; the required post step is named only in a tool output	posts summary ✓	lists statuses, does not post $\times$	describes summary, does not post $\times$	posts summary ✓
Sync the inventory; the service says to retry with an updated version	retries and syncs ✓	tells user to retry $\times$	retries and syncs ✓	retries and syncs ✓
Finalize payroll; the tool says timesheets must be validated first	validates, then finalizes ✓	claims completion, skips validation $\times$	validates, then finalizes ✓	validates, then finalizes ✓

Appendix

Table 6: Behavioral drift on designed benign Nemotron tool-use tasks. Each task requires acting on guidance returned by a first-party tool.

Panel of Figure 4 — Figure 4: Message feed before and after prompt sandwiching.

Panel of Figure 5 — Figure 5: Message feed before and after spotlighting with delimiting.

Panel of Figure 6 — Figure 6: Message feed before and after filtering a tool response with Prompt Guard 2.

Structure	Construction
Surfaced guidance	A step required to complete the user’s task is revealed through a tool output rather than stated in the original request. The output may directly indicate the next operation or identify the resource, record, or location needed to determine it.
Recovery / completion	An intermediate observation reveals how an incomplete task can be continued, for example by exposing a supported alternative after an unsuccessful action, identifying another resource needed for completion, or showing that another user-requested obligation remains.
Follow-through	An action that appears to complete the request instead returns a provisional or incomplete result, requiring one or more additional authorized actions before the task is actually finished.

Appendix

Table 7: Failure-focused structures used during self-generation. Each structure creates a benign task in which information returned through a tool must legitimately influence a subsequent authorized action.

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Model	Method	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Qwen	Vanilla	0.0	0.0	88.7	80.0	5.8	66.7	64.8	4.2
	Prompt Sandwiching	0.0	0.0	89.7	82.6	3.8	73.3	68.2	1.8
	Spotlighting	0.0	0.0	89.7	85.4	0.9	73.3	67.9	0.5
	PromptGuard2	0.0	0.0	88.7	58.6	2.7	66.7	47.7	3.3
	SecAlign++	$0.0{\pm}0.0$	$0.0{\pm}0.0$	$88.7{\pm}1.7$	$84.2{\pm}1.6$	$1.9{\pm}0.1$	$70.0{\pm}1.4$	$67.0{\pm}0.1$	$1.5{\pm}0.2$

Appendix

Table 8: Prompt-injection robustness and utility on Qwen. Higher is better for utility (

U

) and utility under attack (

U_{\mathrm{atk}}

); lower is better for attack success rate (ASR). Trained defenses report mean

\pm

standard deviation over three seeds.

Model	Method	ToolTalk	BFCL	WorkBench	MMT-Bench	AsyncTool
Qwen	Vanilla	66.7	50.0	83.0	50.5	68.1
	Prompt Sandwiching	67.9	46.0	84.5	48.8	68.4
	Spotlighting	69.2	47.0	82.3	49.3	68.9
	PromptGuard2	66.7	50.0	83.0	50.5	68.1
	SecAlign++	$65.4{\pm}0.0$	$49.3{\pm}0.6$	$84.0{\pm}0.5$	$47.9{\pm}0.5$	$68.7{\pm}0.1$
	RAISED (Ours)	$66.2{\pm}0.6$	$50.0{\pm}0.4$	$83.1{\pm}0.6$	$51.2{\pm}0.2$	$67.8{\pm}0.4$

Appendix

Table 9: Benign agentic capabilities on Qwen. Results on five tool-use benchmarks without prompt injection. Trained defenses report mean

\pm

standard deviation over three seeds.

Model	Method	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Qwen	Vanilla	76.5	$85.0$	58.6	$77.6$	36.5	$94.7$	96.1	93.6	96.3	81.6
	SecAlign++	$74.4{\pm}0.3$	$85.0{\pm}0.2$	$54.7{\pm}0.7$	$79.5{\pm}2.3$	$35.1{\pm}0.3$	$94.3{\pm}0.5$	$96.1{\pm}0.1$	$90.0{\pm}1.4$	$96.5{\pm}0.3$	$81.6{\pm}0.6$
	RAISED (Ours)	$76.7{\pm}0.1$	$85.1{\pm}0.1$	$58.8{\pm}0.1$	$74.2{\pm}1.1$	$37.1{\pm}0.5$	$94.1{\pm}0.6$	$96.4{\pm}0.2$	$93.6{\pm}0.1$	$97.0{\pm}0.0$	$81.3{\pm}0.2$

Appendix

Table 10: General capabilities on Qwen. Results on ten non-agentic benchmarks. Trained defenses report mean

\pm

standard deviation over three seeds.

Figure 7: Per-block hidden-state changes over 88 AgentDojo pairs (Gemma-4-12B). Panels 1, 2, and 4 measure the effect of the injection within each model; panel 3 measures the distance from the base model on clean contexts. Bands show s.e.m. across pairs.

\times

at block 48 denotes the state after the final norm.

	Base	SecAlign++	RAISED
Starts the injected tool call (%, 72 pairs) $\downarrow$	62.5	13.9	12.5
KL to the base model’s clean prediction (nats, 88 pairs) $\downarrow$	0.338	0.062	0.011
Injection effect at block 31, peak (rel. $L_{2}$ ) $\downarrow$	0.45	0.29	0.23
Distance to base on clean context, block 47 $\downarrow$	–	0.19	0.12

Appendix

Table 11: Summary of internal-state changes (88 AgentDojo pairs) and of the behavioral check (72 pairs, see text).

		InjecAgent ASR $\downarrow$		AgentDojo			AgentDyn
Method	Steps	Base	Enhanced	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$	$U\uparrow$	$U_{\mathrm{atk}}\uparrow$	ASR $\downarrow$
Vanilla	–	26.7	33.7	48.5	39.9	11.7	25.0	19.6	17.6
SecAlign++	313	$6.4{\pm}0.9$	$8.8{\pm}1.4$	$50.2{\pm}1.0$	$47.7{\pm}0.7$	$4.0{\pm}0.7$	$22.2{\pm}0.8$	$20.7{\pm}0.5$	$7.8{\pm}0.3$
SecAlign++	625	$1.9{\pm}0.2$	$1.4{\pm}0.5$	$50.2{\pm}1.0$	$49.5{\pm}0.9$	$0.7{\pm}0.0$	$22.2{\pm}0.8$	$21.2{\pm}1.2$	$3.0{\pm}0.2$
RAISED (Ours)	–	$3.5{\pm}0.5$	$1.0{\pm}0.6$	$50.2{\pm}1.3$	$46.4{\pm}0.6$	$0.5{\pm}0.1$	$25.0{\pm}0.0$	$25.4{\pm}1.2$	$1.9{\pm}0.6$

Appendix

Table 12: Effect of extended SecAlign++ training on prompt-injection robustness for Nemotron. SecAlign++ is evaluated at the 313-step checkpoint used in the main paper and after 625 training steps. Vanilla and RAISED are included as reference points. Higher is better for utility (

U

and

U_{\mathrm{atk}}

); lower is better for ASR.

Method	Steps	MMLU	MMLU-Pro	MMMLU	GPQA	AGIEval	IFEval	GSM8K	Minerva	HumanEval	MBPP
Vanilla	–	49.5	60.1	34.6	42.4	23.3	81.3	87.2	76.5	87.2	67.8
SecAlign++	313	$48.3{\pm}0.7$	$60.3{\pm}0.2$	$34.9{\pm}0.0$	$43.8{\pm}1.7$	$23.1{\pm}0.2$	$79.4{\pm}1.5$	$88.2{\pm}0.5$	$75.0{\pm}0.1$	$86.0{\pm}0.0$	$64.6{\pm}1.0$
SecAlign++	625	$\mathbf{42.8{\pm}0.3}$	$60.8{\pm}0.1$	$34.8{\pm}0.1$	$\mathbf{39.4{\pm}1.1}$	$21.3{\pm}0.3$	$79.1{\pm}0.8$	$87.2{\pm}0.3$	$\mathbf{66.1{\pm}0.9}$	$86.8{\pm}0.6$	$\mathbf{13.4{\pm}2.7}$
RAISED (Ours)	–	$49.6{\pm}0.1$	$59.3{\pm}0.2$	$35.1{\pm}0.0$	$41.2{\pm}2.1$	$23.3{\pm}0.3$	$80.0{\pm}0.3$	$87.2{\pm}0.3$	$76.7{\pm}0.1$	$86.6{\pm}0.5$	$66.2{\pm}0.6$

Appendix

Table 13: General capabilities under extended SecAlign++ training on Nemotron. The 313-step SecAlign++ checkpoint is the one used in the main paper. Bold values highlight the largest capability degradations observed after extending SecAlign++ training to 625 steps. Vanilla and RAISED are included as reference points.

		AgentDojo		AgentDyn
Data	Objective	$U\uparrow$	ASR $\downarrow$	$U\uparrow$	ASR $\downarrow$
–	Vanilla	84.5	39.5	61.7	53.4
Glaive	SFT	$73.2{\pm}3.6$	$0.5{\pm}0.3$	$17.5{\pm}3.5$	$1.5{\pm}1.2$
Glaive	DPO	$79.1{\pm}0.6$	$1.8{\pm}0.1$	$48.6{\pm}0.3$	$2.3{\pm}1.5$
Glaive	Self-distillation	$80.1{\pm}0.6$	$1.1{\pm}0.2$	$55.5{\pm}0.8$	$2.1{\pm}1.3$
Self-generated	DPO	$82.2{\pm}0.5$	$1.0{\pm}0.1$	$55.6{\pm}3.4$	$1.6{\pm}1.4$

Appendix

Table 14: Ablation of training data and objective on Gemma-4-12B-it. Higher is better for benign utility (

U

); lower is better for attack success rate (ASR). Trained variants report mean

\pm

standard deviation over three seeds.

Explore similar work

Jul 3, 2025cs.CR

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

Prompt injection attacks, where untrusted data contains an injected prompt to manipulate the system, have been listed as the top security threat to AI agents. By fine-tuning on simulated prompt injections, SecAlign, a leading open defense, reports LLMs with good test-time robustness and negligible benign utility drop. By scaling up training and evaluations, however, we find that SecAlign actually suffers from significant utility degradation, especially in agentic tasks where the threat of prompt injection is prominent. Motivated by this, we propose Meta-SecAlign for utility-preserving defense by (1) randomized injection position during training to avoid shortcut learning and (2) self-generated responses as high-quality in-distribution training labels. Across general knowledge, instruction following, and agentic workflows (on tool-calling and web-navigation), Meta-SecAlign maintains almost all the undefended LLM's utility while achieving better overall security than SecAlign against various static and GCG adaptive attacks. Experiments use Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Qwen3-4B, and Qwen3.6-27B on 6 prompt injection benchmarks including AgentDojo, InjecAgent, WASP, and SEP. Below are links for the code (https://github.com/facebookresearch/Meta_SecAlign), Meta-SecAlign-70B (https://huggingface.co/facebook/Meta-SecAlign-70B), and Meta-SecAlign-8B (https://huggingface.co/facebook/Meta-SecAlign-8B) models.

Sizhe Chen, Arman Zharmagambetov, David Wagner +1

FAIR at Meta · UC Berkeley

Indirect Prompt Injection Gradient-Based Attacks Prompt Engineering Value Alignment

Ask PDF

Jun 9, 2026cs.CR

Assessing Automated Prompt Injection Attacks in Agentic Environments

Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.

David Hofer, Edoardo Debenedetti, Florian Tramèr

ETH Zurich Zurich, Switzerland

Indirect Prompt Injection Large Language Model Agents Agentic Control Realistic Threat Model Automated

Ask PDF

Aug 4, 2026cs.CR

AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection

Prompt injection remains a critical threat to LLM agents, yet existing defenses treat each task as a self-contained problem, independent of previous encounters. In practice, user requests are often underspecified: they describe the desired outcome without fully specifying acceptable behavior. An injection can exploit this ambiguity, causing the agent to complete the task in a way the user would reject. As the user's expectations become clearer through concrete cases, a defense should learn from each encounter and apply what it learns to the next. Inspired by adaptive immunity, we propose AgentAntibody, which equips LLM agents with a self-evolving immune system against prompt injection. AgentAntibody represents its evolving understanding of the user's security boundary as a persistent library of antibodies. At runtime, the library recognizes threats to this boundary and mounts corresponding immune responses. Across encounters, it evolves to strengthen the agent's immunity to future attacks. Extensive experiments across three benchmarks and four backbone LLMs show that, by learning the user's boundary through experience, AgentAntibody outperforms existing defenses in preventing harmful actions while preserving legitimate task completion, even when the harmful and legitimate actions are both compatible with the stated task.

Shihao Weng, Yang Feng, Xiaofei Xie +1

Nanjing University · Singapore Management University · Nanyang Technological University

Multi-Llm Agents Prompt Engineering Large Language Model Backbones Antimicrobial Peptide Antibody

Ask PDF