The Autonomy Tax: Defense Training Breaks LLM Agents

Shawn Li; Yue Zhao

cs.CRMar 19, 2026

The Autonomy Tax: Defense Training Breaks LLM Agents

Authors: Shawn Li, Yue Zhao

Organizations: University of Southern California

Abstract

Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines. Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.

Figures & tables

Figure 1: Three Defense Training Biases in Multi-Step Agents. Agent Incompetence : 47–77% Step-1 failure on benign tasks (vs 3% baseline). Cascade Amplification : Timeouts increase from 13–50% to 99%. Trigger Bias : 73–86% attack bypass with 25–71% benign over-refusal. Defense training teaches surface shortcuts, not semantic understanding, causing failures invisible in single-turn evaluation.

Model	Defense	Normal	Refuse	Invalid
Llama-3	Base (3-8B)	96.9%	1.0%	2.1%
	StruQ (3-8B)	22.7%	8.2%	69.1%
	SecAlign (3-8B)	52.6%	47.4%	0.0%
	Meta SecAlign (3.1-8B)	94.8%	0.0%	5.2%
Mistral-7B	Base	97.9%	0.0%	2.1%
	StruQ	23.7%	5.2%	71.1%

Table 1: Step-1 execution behavior revealing immediate agent incompetence on benign tasks. Normal: valid action generated. Refuse: explicit refusal. Invalid: malformed/unparseable output. Bold indicates agent incompetence (

>40

% failure at step 1). Llama-3 includes base Llama-3-8B and Llama-3.1-8B (Meta SecAlign).

		Overall		Cascade Failure		Avg
Model	Defense	CR	$\Delta$ Base	CFR	$\Delta$ CFR	Depth
Llama-3	Base (3-8B)	86.6%	–	13.4%	–	5.32
	StruQ (3-8B)	80.4%	$-6.2$	19.6%	$+6.2$	5.38
	SecAlign (3-8B)	63.9%	$-22.7$	36.1%	$+22.7$	7.78
	Meta SecAlign (3.1-8B)	85.6%	$-1.0$	14.4%	$+1.0$	5.45
Mistral-7B	Base	50.5%	–	49.5%	–	6.14

Table 2: Overall performance and cascade failure metrics on 97 tasks. CR: completion rate.

\Delta

Base: percentage point change from respective base model. CFR: cascade failure rate (fraction timing out at depth 10).

\Delta

CFR: pp change in cascade failures. Avg Depth: mean trajectory length. Bold indicates severe degradation (

>20

pp CR loss or

>30

% CFR). Llama-3 includes base 3-8B and Meta SecAlign on 3.1-8B.

Llama-3 Family
	Attack Detection		False Refusal
Model	TPR	$\Delta$ Base	FPR	$\Delta$ Base
Base-Llama3-8B	82.7%	–	0.0%	–
StruQ-Llama3-8B	37.4%	$-45.3$ pp	70.5%	$+70.5$ pp
SecAlign-Llama3-8B	26.7%	$-56.0$ pp	24.6%	$+24.6$ pp
Base-Llama3.1-8B	79.6%	–	0.6%	–

Table 3: Overall security performance on curated challenging subset (n=350: 289 adversarial, 61 benign). TPR: correctly refusing attacks.

\Delta

Base: pp change from respective base model. FPR: incorrectly refusing benign content. Bold indicates catastrophic degradation (

>40

pp TPR loss or

>60

% FPR).

Llama-3-8B (Social Eng. / Obfuscation / Instr. Override)
Model	Category 1	Category 2	Category 3	Variance
Base-Llama3-8B	22%	14%	18%	8%
StruQ-Llama3-8B	78%	86%	68%	18%
SecAlign-Llama3-8B	73%	81%	75%	8%
Llama-3.1-8B (Direct Request / Constraint Removal)
Meta SecAlign-Llama3.1-8B	53.4%	2.8%	–	50.7%

Table 4: Attack bypass rates by category on challenging subset. Meta SecAlign uses different attack taxonomy (Direct Request vs Constraint Removal) from other models (Social Engineering, Obfuscation, Instruction Override). Variance = max bypass - min bypass across categories. Bold indicates severe vulnerabilities (

>70

% bypass) or high variance (

>40

pp).

Figure 2: Immediate execution failures and utility degradation in defense-trained agents. (a) Step-1 valid action rates: Base models achieve 96.9–97.9% valid actions on the first step, while defense models drop to 22.7–53.6%, indicating immediate execution incompetence before any tool observations appear. (b) Overall task completion rates: Defense training reduces completion from 50.5–86.6% (base) to 1.0–80.4%, with SecAlign-Mistral achieving only 1.0% completion, representing catastrophic breakdown of multi-step competence.

Figure 3: Cascade concentration at maximum depth revealing binary failure dynamics. Defense models concentrate significantly more tasks at the timeout boundary (depth 10). The stacked bars show completed tasks (bottom, lighter shade) vs cascaded tasks (top, darker shade with hatching). Percentage labels indicate cascade rate.

Model	Defense	CFR	Amp Factor	Retry Depth
Llama-3	Base (3-8B)	13.4%	1.0 $\times$	5.32
	StruQ (3-8B)	19.6%	1.5 $\times$	5.38
	SecAlign (3-8B)	36.1%	2.7 $\times$	7.78
	Meta SecAlign (3.1-8B)	14.4%	1.1 $\times$	5.45
Mistral-7B	Base	49.5%	1.0 $\times$	6.14
	StruQ	85.6%	1.7 $\times$	8.70

Table 5: Cascade amplification metrics showing defense-induced timeout concentration. CFR: cascade failure rate (timeout at depth 10). Amp: amplification factor relative to respective base model. Retry Depth: average steps consumed in retry loops before timeout. Llama-3 includes base 3-8B and Meta SecAlign on 3.1-8B.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Defense	Depth 1–2	Depth 3–5	Depth 6–9	Depth 10
Llama-3	Base-3-8B	45.4%	28.9%	12.4%	13.3%
	StruQ-3-8B	41.2%	26.8%	12.4%	19.6%
	SecAlign-3-8B	28.9%	21.6%	13.4%	36.1%
	Meta SecAlign-3.1-8B	44.3%	27.8%	13.4%	14.4%
Mistral	Base	22.7%	18.6%	9.3%	49.5%
	StruQ	8.2%	4.1%	2.1%	85.6%

Appendix

Table 6: Depth-stratified completion rates across all configurations. Tasks are partitioned by final depth: 1–2 (immediate completion), 3–5 (moderate reasoning), 6–9 (extended reasoning), 10 (timeout/cascade failure).

Defense	General Trigger	Security Context	Overall FPR
StruQ-Llama3	70.0% (21/30)	71.0% (22/31)	70.5% (43/61)
SecAlign-Llama3	3.3% (1/30)	45.2% (14/31)	24.6% (15/61)
SecAlign-Mistral	6.7% (2/30)	64.5% (20/31)	36.1% (22/61)

Appendix

Table 7: FPR breakdown on 61 benign samples by content type. General trigger: samples with trigger words (ignore, override, bypass) in non-security contexts. Security context: samples with security terminology + trigger words.a

Configuration	TPR	ASR	FPR	Defense Variance
Exp 2a: Standard role (realistic)	68.0%	32.0%	0.2%	N/A
Exp 2b: “input” role (ideal)	79.6%	20.4%	3.6%	50.7%
Improvement with role marking	+11.6pp	-11.6pp	+3.4pp	–

Appendix

Table 8: Effect of “input” role marking on Meta SecAlign defense effectiveness. Experiment 2a uses standard {"role": "user"} for all content (realistic deployment). Experiment 2b uses {"role": "input"} for adversarial content (Meta AI’s intended format, requires pre-identifying attacks).

Explore similar work

Sep 14, 2026

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

Large Language Model (LLM) agents have demonstrated impressive capabilities across a variety of domains, particularly when integrated with external tools for multi-step task completion. However, they are increasingly vulnerable to adversarial attacks, including direct prompt injection, indirect prompt injection, memory poisoning, and backdoor attacks, which exploit the model's openness to prompt injection and tool manipulation. In this work, we explore practical and generalizable defense strategies within a unified framework across these four attack types. We introduce two universal tool-based defenses: Attacker Tool Filtering, which uses anomaly detection (e.g., Isolation Forest) to identify and remove suspicious tools, and Normal Tool Recalling, a white-box method that restores the agent's original toolset prior to planning. Additionally, we incorporate prompt-based defenses: Chain-of-Thought prompting and self-reflection techniques to enhance reasoning and task paraphrasing to mitigate attacks. Experimental results across both four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5) show that our methods significantly reduce the Attack Success Rates (ASR), achieving 0% ASR in many settings, while preserving or even improving the original task success rate. These findings highlight the promise of simple, modular, multi-layered defenses for strengthening the security and robustness of tool-integrated LLM agents. The code is available at https://github.com/Xiaoyan-Lisa/Defenses-for-Tool-Integrated-LLM-Agents-Against-Adversarial-Attacks.

Xiaoyan Li, Yunli Wang

Department of Computer Science University of Toronto Toronto, Ontario, Canada · Digital Technologies Research Centre National Research Council Canada Ottawa, Ontario, Canada

Adversarial Attacks on LLMs Backdoor Defense in LLMs AI Agent Security Prompt Injection Attacks on AI Agents Tool Access Control for LLM Agents

Ask PDF

Oct 5, 2026

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

Mohamed Dhouib, Clement Elliker, Alexi Canesse +5

LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1

Prompt Injection Defense LLM Agents Prompt Injection Attacks on LLMs LLM Agent Training Self-Distillation

Ask PDF

Jun 12, 2026

From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails

LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents. However, we reveal that the very reasoning and task-following capabilities enabling this protection introduce a novel vulnerability: attackers can inject crafted data to trap the guardrail in extended reasoning loops, effectuating a systematic denial-of-service (DoS) attack. To systematically expose this threat, we design a beam-search optimization framework that crafts natural-language payloads to maximize guardrail reasoning length, utilizing an LLM proposer guided by a strategy bank. Based on the observation of guardrail's schema-following nature, we also provide another attack framework driven by mechanism-aware structural mutations with less computational load. The attack efficacy is systematically evaluated in two parts. First, in standalone evaluations, the attack generalizes across diverse guardrail architectures, safety templates, and agent benchmarks. Payloads optimized on a single open-source surrogate successfully transfer to eight leading model backbones (e.g., Claude, GPT, Gemini, DeepSeek, and Qwen), achieving a 13--63

\times

token amplification. Second, in end-to-end real-world agent deployments (web, desktop, code, and multi-agent systems), the attack reveals up to a 148

\times

latency amplification. We show that a single poisoned document can saturate shared guardrail infrastructures, effectively starving co-located agents and paralyzing the entire system. By uncovering this availability flaw, our work underscores the urgent need to develop cost-bounded, reasoning-robust guardrails.

Yuguang Zhou, Xunguang Wang, Pingchuan Ma +3

1The Hong Kong University of Science and Technology · 2Zhejiang University of Technology

LLM Guardrails Adversarial Attacks on LLMs Cybersecurity AI Agent Security

Ask PDF