Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents

Wonjun Lee; Vaidehi Patil; Zaid Khan; Yue Zhang; Hyunji Lee; Suhyun Kim; Mohit Bansal

cs.CROct 17, 2026

Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents

Authors: Wonjun Lee, Vaidehi Patil, Zaid Khan, Yue Zhang, Hyunji Lee, Suhyun Kim, Mohit Bansal

Organizations: UNC Chapel Hill · Kyung Hee University

Abstract

Recent work shows that attackers can implant dormant malicious behaviors that remain hidden at release but emerge after benign fine-tuning. As foundation models are used as agents, whether dormant poisoning extends to harmful agent actions and multimodal settings remains unclear. We show two agentic risks: (1) harmful behaviors activated by benign fine-tuning can extend to unseen actions; (2) multiple harmful behaviors can coexist and be jointly activated by benign downstream fine-tuning. Existing defenses often rely on external safe responses, which are hard to obtain for agent actions, or a trusted clean reference model. We exploit a key property of dormant poisoning: the attacker must preserve safe behavior at release to conceal poisoning, and this behavior can serve as a safety reference. Leveraging this property, we automatically construct a Safety Buffer pairing harmful inputs with the released model's responses and benign versions generated by the same model to remove harmful intent. The Safety Buffer can replace external safe responses in existing defenses, but these defenses do not distinguish updates that weaken safety from those that preserve it, often increasing over-refusal or reducing utility. To address this limitation, we introduce LookAhead Defense, which previews benign fine-tuning updates using the Safety Buffer and penalizes only updates predicted to weaken safe behavior on harmful inputs or increase the likelihood of refusals on benign versions in the Safety Buffer. Across language models, language agents, and multimodal agents, LookAhead Defense reduces harmful behavior in agents from up to 93.9% to at most 14.2% and harmful responses in language models from up to 92.5% to at most 2.0% versus vanilla fine-tuning, while maintaining competitive utility and low over-refusal. It remains effective against other poisoning methods.

Figures & tables

Figure 1: Threat model and overview of LookAhead Defense . (Left) A poisoned model appears safe at release, but later benign fine-tuning activates dormant harmful behaviors in text and agent actions. (Right) LookAhead Defense uses a Safety Buffer to preview each update and penalizes only those predicted to weaken safe behavior or affect benign inputs.

Figure 2: Detailed overview of our defense. Before fine-tuning, we build a Safety Buffer from harmful inputs, the released model’s responses, and benign versions of the harmful inputs. During fine-tuning, LookAhead Defense previews each update and adds a penalty only when the released model’s response becomes less likely on the harmful input, indicating weaker safe behavior, or more likely on the benign version, where a different response is expected.

Table 1: Several harmful behaviors in one released model with Agentic FAB, per risk category. Models are Qwen3-4B-Instruct, Granite-4.2-8B, Qwen2.5-VL-3B-Instruct, and Qwen3-VL-8B-Instruct.

Table 2: Dormant poisoning with Agentic FAB and LookAhead Defense in two agentic settings. LookAhead denotes LookAhead Defense in all tables. Evaluation data are held out from poisoning, fine-tuning, and Safety Buffer construction. Avg. averages the two benchmarks in each group, and bold and underline mark the best and second best.

Table 3: Source of the Safety Buffer on Qwen3-4B-Instruct. OR denotes over-refusal.

Method	No Aux. Model	No Ext. Responses	Qwen2.5-3B-Instruct				Qwen3-4B-Instruct				OLMo-3-7B-Instruct
Method	No Aux. Model	No Ext. Responses	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Util. $\uparrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Util. $\uparrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Util. $\uparrow$
Before FT	–	–	3.0	18.8	10.9	–	0.0	13.2	6.6	–	0.0	11.2	5.6	–
Vanilla FT	–	–	92.5	3.2	47.9	61.6	61.5	0.8	31.2	72.5	82.5	4.0	43.3	68.9
SafeDelta	✓	✗	4.0	14.8	9.4	48.1	0.5	14.4	7.5	59.1	0.0	14.8	7.4	79.9
Safety-mix	✓	✗	0.5	56.0	28.3	61.7	0.0	44.0	22.0	73.8	0.0	20.0	10.0	69.8
Lisa	✓	✗	1.5	21.2	11.4	64.7	4.5	6.0	5.3	78.5	6.0	14.8	10.4	73.2

Table 4: Comparison with defenses during downstream fine-tuning on language models poisoned with FAB. No Aux. Model means no auxiliary reference model, and No Ext. Responses no externally provided safe responses. OR denotes over-refusal and Avg. the mean of ASR and OR.

		FAB			Sleeper			Benign DPO
Model	Defense	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$
Qwen2.5	Before FT	3.0	18.8	10.9	63.0	12.0	37.5	66.5	0.0	33.3
	Vanilla FT	92.5	3.2	47.9	67.5	2.0	34.8	69.5	0.0	34.8
	LookAhead	1.0	7.6	4.3	1.0	2.4	1.7	10.0	1.6	5.8
Qwen3	Before FT	0.0	13.2	6.6	61.5	7.6	34.6	61.0	0.0	30.5
	Vanilla FT	61.5	0.8	31.2	74.5	0.0	37.3	61.0	0.0	30.5

Table 5: LookAhead Defense across multiple poisoning methods in the language model setting. OR denotes over-refusal and Avg. denotes the mean of ASR and OR. Models are Qwen2.5-3B-Instruct, Qwen3-4B-Instruct, and OLMo-3-7B-Instruct.

Table 6: Ablation of LookAhead Defense on a poisoned Qwen3-4B-Instruct under GSM8K fine-tuning. (a) Objective components, (b) Safety Buffer construction, and (c) application to the clean checkpoint before poisoning. OR denotes over-refusal, and Avg. is the mean of ASR and OR. Best results are bolded and second best underlined.

	Stage 1 (GSM8K)				Stage 2 (PubMedQA)				Stage 3 (CodeAlpaca)
Method	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Utility $\uparrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Utility $\uparrow$	ASR $\downarrow$	OR $\downarrow$	Avg. $\downarrow$	Utility $\uparrow$
Vanilla FT	61.5	0.8	31.2	72.5	64.5	2.4	33.5	78.8	73.0	0.4	36.7	51.2
SafeDelta	0.5	14.4	7.5	59.1	0.0	17.6	8.8	73.8	0.0	16.4	8.2	79.9
Safety-mix	0.0	44.0	22.0	73.8	1.0	25.2	13.1	78.0	1.0	23.2	12.1	62.2
Lisa	4.5	6.0	5.3	78.5	1.5	18.8	10.2	83.0	23.0	8.8	15.9	68.3
LookAhead (refreshed)	1.0	2.0	1.5	74.1	0.0	2.4	1.2	77.0	4.5	1.6	3.1	55.5

Table 7: Continual fine-tuning of FAB-poisoned Qwen3-4B-Instruct across GSM8K, PubMedQA ( Jin et al., 2019 ) , and CodeAlpaca ( Chaudhary, 2023 ) . Baselines are applied at every stage. Utility is evaluated on GSM8K, PubMedQA, and HumanEval ( Chen et al., 2021 ) , respectively. OR denotes over-refusal, and Avg. is the mean of ASR and OR within each stage. Best results are bolded and second best underlined.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setup	Experimental axis	Language model	Language agent	Multimodal agent
All	Model	Qwen2.5-3B-Instruct ( Yang et al., 2024 ) , Qwen3-4B-Instruct ( Yang et al., 2025 ) , OLMo-3-7B-Instruct ( Team Olmo et al., 2025 )	Qwen3-4B-Instruct ( Yang et al., 2025 ) , Granite-4.2-8B ( IBM Research, 2026 )	Qwen2.5-VL-3B-Instruct ( Bai et al., 2025b ) , Qwen3-VL-8B-Instruct ( Bai et al., 2025a )
Poisoning	Method	FAB ( Gloaguen et al., 2026 ) , Sleeper ( Hubinger et al., 2024 ) , Benign DPO ( Yoon et al., 2026 )	Agentic FAB	Agentic FAB
	Data	–	InjecAgent ( Zhan et al., 2024 )	RiOSWorld ( Yang et al., 2026a )
	Simulated FT data	Alpaca-GPT4 ( Peng et al., 2023 ) (FAB)	xLAM ( Liu et al., 2024 ) , ToolACE ( Liu et al., 2025 )	GUI-Net ( Zhang et al., 2026 ) , AndroidControl ( Li et al., 2024 )
	Target behavior	Harmful text	Cyber, Disinformation, Fraud (attacker tool calls)	Personal, Unknown, Phishing (harmful actions)
Downstream FT	Fine-tuning data	GSM8K ( Cobbe et al., 2021 )	Glaive ( Glaive AI, 2023 )	OS-Genesis ( Sun et al., 2025 )

Appendix

Table 8: Experimental setup across settings. The system setting, model, poisoning method, target behavior, and datasets vary as summarized below. Defended models are evaluated with the same protocol as undefended ones. For language models, each poisoning method follows the data used in its original work.

Table 9: Robustness of the Safety Buffer to contamination. (a) Contamination of the harmful input and response pairs across language models (Qwen3-4B-Instruct with FAB), language agents (Qwen3-4B-Instruct), and multimodal agents (Qwen2.5-VL-3B-Instruct). (b) Contamination of the benign versions on Qwen3-4B-Instruct with FAB. The

0\%

row is the Safety Buffer used in the paper, with no examples replaced. Appendix B describes both constructions. OR denotes over-refusal.

Setting	Safe response ( $y_{i}$ ) $\uparrow$	Harmless ( $\tilde{x}_{i}$ ) $\uparrow$	Context kept ( $\tilde{x}_{i}$ ) $\uparrow$
Language model (FAB)	92.0	96.0	92.0
Language model (Sleeper)	95.3	93.0	95.3
Language model (Benign DPO)	88.6	91.4	94.3
Language agent (FAB)	100.0	100.0	72.0
Language agent (Sleeper)	100.0	100.0	60.0
Multimodal agent (FAB)	78.0	94.0	88.0

Appendix

Table 10: Safety assessment of Safety Buffer examples that passed our construction-time checks, judged by Claude Sonnet 5.

Preview direction	Preview		Language model			Language agent		Multimodal agent
Preview direction	Correlation $\uparrow$	Cost	ASR $\downarrow$	OR $\downarrow$	GSM8K $\uparrow$	InjecAgent $\downarrow$	Glaive $\uparrow$	RiOSWorld $\downarrow$	OS-Genesis $\uparrow$
Raw gradient	0.06	1.7 $\times$	1.0	19.2	74.5	1.7	76.0	31.1	84.5
Norm-matched gradient	-0.03	1.7 $\times$	1.5	4.8	74.4	84.7	77.9	0.8	79.1
AdamW update	1.00	260 $\times$	2.0	0.4	73.9	2.0	74.0	23.1	83.8
Sign of the gradient (ours)	0.96	1.0 $\times$	1.0	2.0	74.1	0.0	76.0	6.4	82.7

Appendix

Table 11: Comparison of preview directions on FAB-poisoned Qwen3-4B-Instruct (language model), Qwen3-4B-Instruct (language agent), and Qwen2.5-VL-3B-Instruct (multimodal agent). Correlation measures the agreement between the previewed change in Safety Buffer losses and the change after the applied update. Preview cost is measured on the language model. OR denotes over-refusal.

$\mu$	ASR $\downarrow$	OR $\downarrow$	Utility $\uparrow$
0 (Vanilla FT)	61.5	0.8	72.5
10	1.0	2.0	73.0
30	1.5	2.0	74.4
100	1.0	2.0	74.1
300	0.5	4.8	73.9

Appendix

Table 12: Effect of the penalty strength

\mu

on FAB-poisoned Qwen3-4B-Instruct under GSM8K fine-tuning. OR denotes over-refusal.

Table 13: Effect of the penalty strength

\mu

in the language and multimodal agent settings. Models are Qwen3-4B-Instruct, Granite-4.2-8B, Qwen2.5-VL-3B-Instruct, and Qwen3-VL-8B-Instruct. Poison activation and utility are averaged over the two benchmarks reported for each model in Table 2 . Shaded rows indicate the setting used there.

Method	Cost		Safety
Method	Time / step (s)	Peak memory (GB)	ASR $\downarrow$	OR $\downarrow$
SafeDelta	2.73	23.7	4.0	14.8
Safety-mix	2.98	23.7	0.5	56.0
Lisa	2.71	29.4	1.5	21.2
AsFT	2.87	34.0	87.0	12.8
LookAhead	3.16	32.6	1.0	7.6

Appendix

Table 14: Computational overhead comparison of defenses on Qwen2.5-3B-Instruct. Time is seconds per step on one RTX 6000 Ada. OR denotes over-refusal.

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 319 papers

The Autonomy Tax: Defense Training Breaks LLM Agents
Mar 19, 2026Shawn Li, Yue ZhaoAI Agent Security LLM Agents
University of Southern California
Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models
May 23, 2026Seokil Ham, Jaehyuk Jang, Wonjun Lee +1LLM Fine-Tuning LLM Safety Alignment
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents
May 27, 2026Yongxiang Li, Moxin Li, Zhixin Ma +2LLM Agent Security AI Agent Security Benchmarks
University of Science and Technology of China · National University of Singapore · Singapore Management University

Mar 19, 2026J/K move · Enter open · S save

The Autonomy Tax: Defense Training Breaks LLM Agents

Shawn Li, Yue Zhao

University of Southern California

AI Agent Security LLM Agents LLM Agent Safety Prompt Injection Defense Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines. Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.

Abstract

AI Agent Safety

Explore similar work

The Autonomy Tax: Defense Training Breaks LLM Agents

Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models

Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents