The Autonomy Tax: Defense Training Breaks LLM Agents

Shawn Li; Yue Zhao

cs.CRMar 19, 2026

The Autonomy Tax: Defense Training Breaks LLM Agents

Authors: Shawn Li, Yue Zhao

Organizations: University of Southern California

Abstract

Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines. Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.

Figures & tables

Figure 1: Three Defense Training Biases in Multi-Step Agents. Agent Incompetence : 47–77% Step-1 failure on benign tasks (vs 3% baseline). Cascade Amplification : Timeouts increase from 13–50% to 99%. Trigger Bias : 73–86% attack bypass with 25–71% benign over-refusal. Defense training teaches surface shortcuts, not semantic understanding, causing failures invisible in single-turn evaluation.

Model	Defense	Normal	Refuse	Invalid
Llama-3	Base (3-8B)	96.9%	1.0%	2.1%
	StruQ (3-8B)	22.7%	8.2%	69.1%
	SecAlign (3-8B)	52.6%	47.4%	0.0%
	Meta SecAlign (3.1-8B)	94.8%	0.0%	5.2%
Mistral-7B	Base	97.9%	0.0%	2.1%
	StruQ	23.7%	5.2%	71.1%

Table 1: Step-1 execution behavior revealing immediate agent incompetence on benign tasks. Normal: valid action generated. Refuse: explicit refusal. Invalid: malformed/unparseable output. Bold indicates agent incompetence (

>40

% failure at step 1). Llama-3 includes base Llama-3-8B and Llama-3.1-8B (Meta SecAlign).

		Overall		Cascade Failure		Avg
Model	Defense	CR	$\Delta$ Base	CFR	$\Delta$ CFR	Depth
Llama-3	Base (3-8B)	86.6%	–	13.4%	–	5.32
	StruQ (3-8B)	80.4%	$-6.2$	19.6%	$+6.2$	5.38
	SecAlign (3-8B)	63.9%	$-22.7$	36.1%	$+22.7$	7.78
	Meta SecAlign (3.1-8B)	85.6%	$-1.0$	14.4%	$+1.0$	5.45
Mistral-7B	Base	50.5%	–	49.5%	–	6.14

Table 2: Overall performance and cascade failure metrics on 97 tasks. CR: completion rate.

\Delta

Base: percentage point change from respective base model. CFR: cascade failure rate (fraction timing out at depth 10).

\Delta

CFR: pp change in cascade failures. Avg Depth: mean trajectory length. Bold indicates severe degradation (

>20

pp CR loss or

>30

% CFR). Llama-3 includes base 3-8B and Meta SecAlign on 3.1-8B.

Llama-3 Family
	Attack Detection		False Refusal
Model	TPR	$\Delta$ Base	FPR	$\Delta$ Base
Base-Llama3-8B	82.7%	–	0.0%	–
StruQ-Llama3-8B	37.4%	$-45.3$ pp	70.5%	$+70.5$ pp
SecAlign-Llama3-8B	26.7%	$-56.0$ pp	24.6%	$+24.6$ pp
Base-Llama3.1-8B	79.6%	–	0.6%	–

Table 3: Overall security performance on curated challenging subset (n=350: 289 adversarial, 61 benign). TPR: correctly refusing attacks.

\Delta

Base: pp change from respective base model. FPR: incorrectly refusing benign content. Bold indicates catastrophic degradation (

>40

pp TPR loss or

>60

% FPR).

Llama-3-8B (Social Eng. / Obfuscation / Instr. Override)
Model	Category 1	Category 2	Category 3	Variance
Base-Llama3-8B	22%	14%	18%	8%
StruQ-Llama3-8B	78%	86%	68%	18%
SecAlign-Llama3-8B	73%	81%	75%	8%
Llama-3.1-8B (Direct Request / Constraint Removal)
Meta SecAlign-Llama3.1-8B	53.4%	2.8%	–	50.7%

Table 4: Attack bypass rates by category on challenging subset. Meta SecAlign uses different attack taxonomy (Direct Request vs Constraint Removal) from other models (Social Engineering, Obfuscation, Instruction Override). Variance = max bypass - min bypass across categories. Bold indicates severe vulnerabilities (

>70

% bypass) or high variance (

>40

pp).

Figure 2: Immediate execution failures and utility degradation in defense-trained agents. (a) Step-1 valid action rates: Base models achieve 96.9–97.9% valid actions on the first step, while defense models drop to 22.7–53.6%, indicating immediate execution incompetence before any tool observations appear. (b) Overall task completion rates: Defense training reduces completion from 50.5–86.6% (base) to 1.0–80.4%, with SecAlign-Mistral achieving only 1.0% completion, representing catastrophic breakdown of multi-step competence.

Figure 3: Cascade concentration at maximum depth revealing binary failure dynamics. Defense models concentrate significantly more tasks at the timeout boundary (depth 10). The stacked bars show completed tasks (bottom, lighter shade) vs cascaded tasks (top, darker shade with hatching). Percentage labels indicate cascade rate.

Model	Defense	CFR	Amp Factor	Retry Depth
Llama-3	Base (3-8B)	13.4%	1.0 $\times$	5.32
	StruQ (3-8B)	19.6%	1.5 $\times$	5.38
	SecAlign (3-8B)	36.1%	2.7 $\times$	7.78
	Meta SecAlign (3.1-8B)	14.4%	1.1 $\times$	5.45
Mistral-7B	Base	49.5%	1.0 $\times$	6.14
	StruQ	85.6%	1.7 $\times$	8.70

Table 5: Cascade amplification metrics showing defense-induced timeout concentration. CFR: cascade failure rate (timeout at depth 10). Amp: amplification factor relative to respective base model. Retry Depth: average steps consumed in retry loops before timeout. Llama-3 includes base 3-8B and Meta SecAlign on 3.1-8B.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Defense	Depth 1–2	Depth 3–5	Depth 6–9	Depth 10
Llama-3	Base-3-8B	45.4%	28.9%	12.4%	13.3%
	StruQ-3-8B	41.2%	26.8%	12.4%	19.6%
	SecAlign-3-8B	28.9%	21.6%	13.4%	36.1%
	Meta SecAlign-3.1-8B	44.3%	27.8%	13.4%	14.4%
Mistral	Base	22.7%	18.6%	9.3%	49.5%
	StruQ	8.2%	4.1%	2.1%	85.6%

Appendix

Table 6: Depth-stratified completion rates across all configurations. Tasks are partitioned by final depth: 1–2 (immediate completion), 3–5 (moderate reasoning), 6–9 (extended reasoning), 10 (timeout/cascade failure).

Defense	General Trigger	Security Context	Overall FPR
StruQ-Llama3	70.0% (21/30)	71.0% (22/31)	70.5% (43/61)
SecAlign-Llama3	3.3% (1/30)	45.2% (14/31)	24.6% (15/61)
SecAlign-Mistral	6.7% (2/30)	64.5% (20/31)	36.1% (22/61)

Appendix

Table 7: FPR breakdown on 61 benign samples by content type. General trigger: samples with trigger words (ignore, override, bypass) in non-security contexts. Security context: samples with security terminology + trigger words.a

Configuration	TPR	ASR	FPR	Defense Variance
Exp 2a: Standard role (realistic)	68.0%	32.0%	0.2%	N/A
Exp 2b: “input” role (ideal)	79.6%	20.4%	3.6%	50.7%
Improvement with role marking	+11.6pp	-11.6pp	+3.4pp	–

Appendix

Table 8: Effect of “input” role marking on Meta SecAlign defense effectiveness. Experiment 2a uses standard {"role": "user"} for all content (realistic deployment). Experiment 2b uses {"role": "input"} for adversarial content (Meta AI’s intended format, requires pre-identifying attacks).

Explore similar work

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
Sep 14, 2026Xiaoyan Li, Yunli WangAdversarial Attacks on LLMs Backdoor Defense in LLMs
Department of Computer Science University of Toronto Toronto, Ontario, Canada · Digital Technologies Research Centre National Research Council Canada Ottawa, Ontario, Canada
RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection Defense LLM Agents
LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1
From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails
Jun 12, 2026Yuguang Zhou, Xunguang Wang, Pingchuan Ma +3LLM Guardrails Adversarial Attacks on LLMs
1The Hong Kong University of Science and Technology · 2Zhejiang University of Technology

Sep 14, 2026J/K move · Enter open · S save

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

Xiaoyan Li, Yunli Wang

Department of Computer Science University of Toronto Toronto, Ontario, Canada · Digital Technologies Research Centre National Research Council Canada Ottawa, Ontario, Canada

Adversarial Attacks on LLMs Backdoor Defense in LLMs AI Agent Security Prompt Injection Attacks on AI Agents Tool Access Control for LLM Agents

Open paper AskPDF arXiv

Large Language Model (LLM) agents have demonstrated impressive capabilities across a variety of domains, particularly when integrated with external tools for multi-step task completion. However, they are increasingly vulnerable to adversarial attacks, including direct prompt injection, indirect prompt injection, memory poisoning, and backdoor attacks, which exploit the model's openness to prompt injection and tool manipulation. In this work, we explore practical and generalizable defense strategies within a unified framework across these four attack types. We introduce two universal tool-based defenses: Attacker Tool Filtering, which uses anomaly detection (e.g., Isolation Forest) to identify and remove suspicious tools, and Normal Tool Recalling, a white-box method that restores the agent's original toolset prior to planning. Additionally, we incorporate prompt-based defenses: Chain-of-Thought prompting and self-reflection techniques to enhance reasoning and task paraphrasing to mitigate attacks. Experimental results across both four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5) show that our methods significantly reduce the Attack Success Rates (ASR), achieving 0% ASR in many settings, while preserving or even improving the original task success rate. These findings highlight the promise of simple, modular, multi-layered defenses for strengthening the security and robustness of tool-integrated LLM agents. The code is available at https://github.com/Xiaoyan-Lisa/Defenses-for-Tool-Integrated-LLM-Agents-Against-Adversarial-Attacks.

Abstract

Explore similar work

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails