Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience

Zixuan Wang; Hao Li; Fengyu Gao; G. Edward Suh; Yi Zeng; Yevgeniy Vorobeychik; Ning Zhang; Chaowei Xiao

cs.LGOct 3, 2026

Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience

Authors: Zixuan Wang, Hao Li, Fengyu Gao, G. Edward Suh, Yi Zeng, Yevgeniy Vorobeychik, Ning Zhang, Chaowei Xiao

Organizations: Johns Hopkins University · Washington University in St. Louis · University of Virginia · NVIDIA · Virginia Tech

Abstract

Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables agents to improve robustness by learning from their own failure experiences under adversarial conditions. Instead of passively imitating expert behaviors, SRFT exposes the agent to compromised trajectories constructed via injected attacks, and leverages an expert model to generate structured self-reflection reasoning that contrasts unsafe and optimal actions. This reflective supervision teaches the agent to identify malicious instructions, reason about their consequences, and maintain alignment with the original user intent. We instantiate this framework in SR-Agent, built on Llama-3.1-8B-Instruct and Qwen3-8B, and evaluate it on both static and adaptive prompt injection benchmarks. Experimental results show that SRFT substantially reduces attack success rates while preserving task performance, and demonstrates strong generalization under adaptive attacks. These findings suggest that learning from failure via self-reflection is a promising direction for building robust and secure LLM agents. Our code is released at https://github.com/Eden-Wang1710/srft-repo.

Figures & tables

Figure 1: Left: Supervised fine-tuning learns from fixed defensive patterns, which may become ineffective when encountering unseen attacks. Right: Self-reflection fine-tuning learns from failure experiences, enabling it to handle a broader range of attacks and identify previously unseen ones.

Figure 2: General-capability benchmarks for the Llama-3.1-8B family

Figure 3: Attack success rate over attacker training epochs under the RL-Hammer adaptive attack , for (a) Llama-3.1-8B, (b) Qwen3-8B, and (c) the train-time ablation of self-reflection on Qwen3-8B. Endpoint statistics are reported in Tables 7 and 8 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	User	Malicious	Off-task		Assistant	Inj.
Platform	tasks	inj. tasks	inj. tasks	Traj.	steps	obs.
Hotel booking	155	7	111	1,395	9,432	37.3%
Markdown downloader	100	7	71	900	4,608	39.7%
Minecraft Wiki	91	7	62	819	5,058	37.1%
Windows command line	32	8	23	320	1,580	39.1%
Email sender	24	9	18	264	1,661	35.8%

Appendix

Table 5: Training-data statistics per platform. Malicious = injection tasks generated by Template 1; off-task = user requests from other trajectories inserted the same way; inj. obs. = share of tool observations that carry an injection.

Platform	Qwen3-8B $\downarrow$	Qwen3-8B + append $\downarrow$	SR-Agent + append $\downarrow$
Banking	31.25	36.81	4.86
Slack	58.10	54.29	1.90
Travel	32.14	9.29	0.00
Workspace	1.79	3.04	0.17
All	16.97	14.75	1.05

Appendix

Table 6: ASR (%) of the system-prompt append on the untrained Qwen3-8B and on SR-Agent, same evaluation protocol.

Family	Target	Final $\downarrow$	Last-5 $\downarrow$	Peak (epoch) $\downarrow$
Llama-3.1-8B	Undefended base	98.5	97.9	99.0 (16)
	Meta-SecAlign-8B	69.5	62.1	69.5 (20)
	SR-Agent (ours)	17.5	19.8	26.5 (13)
Qwen3-8B	Undefended base	59.5	62.8	68.0 (16)
Qwen3-8B	SR-Agent (ours)	17.0	16.8	21.5 (19)

Appendix

Table 7: RL-Hammer attack success rate (%): endpoint statistics of Figure 3 . Each value is the mean over two independent attacker trainings against that target (different seeds, identical protocol), with every epoch’s checkpoint evaluated on the same 100 InjecAgent test cases. Final = the last attacker epoch; Last-5 = mean over the last five epochs; Peak = maximum of the curve and the epoch at which it occurs.

Target	Final $\downarrow$	Last-5 $\downarrow$	Peak (epoch) $\downarrow$
Undefended base	59.5	62.8	68.0 (16)
SR-Agent (ours)	17.0	16.8	21.5 (19)
w/o failure experience, think on	65.0	65.0	67.0 (8)
w/o reflection, think on	60.0	54.8	69.0 (12)
w/o reflection, think off	60.0	57.0	60.0 (20)

Appendix

Table 8: Train-time ablation: ASR (%) under RL-Hammer. All targets are Qwen3-8B, evaluated on the same 100 InjecAgent test cases.

Platform	Model	Benign Utility $\uparrow$	ASR $\downarrow$
Shopping	Llama-3.1-8B-Instruct	5.00	5.00
	Meta-SecAlign-8B	0.00	0.00
	SR-Agent-Llama (ours)	5.00	1.67
GitHub	Llama-3.1-8B-Instruct	25.00	1.67
	Meta-SecAlign-8B	15.00	2.78
	SR-Agent-Llama (ours)	20.00	2.78

Appendix

Table 9: Agentic workflow evaluation on AgentDyn using Llama-3.1-8B. Llama-3.1-8B-Instruct and SR-Agent-Llama are our runs under one pipeline; the Meta-SecAlign-8B row is computed from the trajectories released by the AgentDyn authors.

Explore similar work

Aug 4, 2026

AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection

Prompt injection remains a critical threat to LLM agents, yet existing defenses treat each task as a self-contained problem, independent of previous encounters. In practice, user requests are often underspecified: they describe the desired outcome without fully specifying acceptable behavior. An injection can exploit this ambiguity, causing the agent to complete the task in a way the user would reject. As the user's expectations become clearer through concrete cases, a defense should learn from each encounter and apply what it learns to the next. Inspired by adaptive immunity, we propose AgentAntibody, which equips LLM agents with a self-evolving immune system against prompt injection. AgentAntibody represents its evolving understanding of the user's security boundary as a persistent library of antibodies. At runtime, the library recognizes threats to this boundary and mounts corresponding immune responses. Across encounters, it evolves to strengthen the agent's immunity to future attacks. Extensive experiments across three benchmarks and four backbone LLMs show that, by learning the user's boundary through experience, AgentAntibody outperforms existing defenses in preventing harmful actions while preserving legitimate task completion, even when the harmful and legitimate actions are both compatible with the stated task.

Shihao Weng, Yang Feng, Xiaofei Xie +1

Nanjing University · Singapore Management University · Nanyang Technological University

Continual Learning for LLM Agents LLM Agent Security Prompt Injection Defense Prompt Injection Attacks Prompt Injection Attacks on LLMs

Ask PDF

Jul 3, 2025

Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents

Prompt injection attacks, where untrusted data contains an injected prompt to manipulate the system, have been listed as the top security threat to AI agents. By fine-tuning on simulated prompt injections, SecAlign, a leading open defense, reports LLMs with good test-time robustness and negligible benign utility drop. By scaling up training and evaluations, however, we find that SecAlign actually suffers from significant utility degradation, especially in agentic tasks where the threat of prompt injection is prominent. Motivated by this, we propose Meta-SecAlign for utility-preserving defense by (1) randomized injection position during training to avoid shortcut learning and (2) self-generated responses as high-quality in-distribution training labels. Across general knowledge, instruction following, and agentic workflows (on tool-calling and web-navigation), Meta-SecAlign maintains almost all the undefended LLM's utility while achieving better overall security than SecAlign against various static and GCG adaptive attacks. Experiments use Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Qwen3-4B, and Qwen3.6-27B on 6 prompt injection benchmarks including AgentDojo, InjecAgent, WASP, and SEP. Below are links for the code (https://github.com/facebookresearch/Meta_SecAlign), Meta-SecAlign-70B (https://huggingface.co/facebook/Meta-SecAlign-70B), and Meta-SecAlign-8B (https://huggingface.co/facebook/Meta-SecAlign-8B) models.

Sizhe Chen, Arman Zharmagambetov, David Wagner +1

FAIR at Meta · UC Berkeley

LLM Agent Security AI Agent Security Benchmarks Prompt Injection Defense Adversarial Robustness of Language Models Prompt Injection Attacks on LLMs LLM Agent Training+1

Ask PDF

Oct 5, 2026

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

Mohamed Dhouib, Clement Elliker, Alexi Canesse +5

LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1

Prompt Injection Defense LLM Agents Prompt Injection Attacks on LLMs LLM Agent Training Self-Distillation

Ask PDF