EvolvAttack: Backdooring self Evolving VLM Agents

Chuan Zou; Yang Xiang; Zexin Li; Xiaonan Luo; Simin Chen

cs.LGOct 11, 2026

EvolvAttack: Backdooring self Evolving VLM Agents

Authors: Chuan Zou, Yang Xiang, Zexin Li, Xiaonan Luo, Simin Chen

Organizations: University of Pennsylvania · Brown University · Nanyang Technological University · University of Notre Dame ·

Abstract

Self-evolving mechanisms let vision-language model (VLM) agents continually adapt through evolving memory, but they also allow malicious behavior to emerge only after deployment. We introduce a new attack surface, the self-evolving backdoor, whose payload stays dormant in the initial agent and is progressively activated as the agent evolves. Unlike conventional backdoors, which can be exposed by testing triggered inputs on a static model, our attack requires both a visual trigger and a sufficiently evolved memory state; static scanning with clean memory thus yields near-zero attack success and may deem the agent benign. We realize this through memory-aware backdoor training, which couples trigger activation with the agent's evolving memory while preserving benign behavior unless the trigger co-occurs with payload-related memory. Across three VLM backbones, four agent pipelines, and recency-, semantic-, and reward-based memory evolution, the attack preserves normal functionality and stays nearly inactive before evolution (0.5% attack success rate), yet exceeds 95% attack success within 100 interactions, revealing a stealthy threat unique to self-evolving agents.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of memory evolution in VLM agents and the positioning of our attack relative to prior backdoor settings.

Figure 2: An overview of our threat model

			Attack Eff.	Stealth
Phase	Retrieval	Method	ASR	AC $\uparrow$	GA $\uparrow$	SR $\uparrow$
		Query-Only	0.23	0.64	0.29	0.20
Before Evolving	No Mem.	EvolvAttack	0.00 (-.23)	0.87 (+.23)	0.47 (+.18)	0.43 (+.23)
After Evolving	Recent- $k$	Query-Only	0.26	0.77	0.29	0.21
		Mem.-Only	0.10	0.81	0.35	0.27
		EvolvAttack	1.00 (+.74)	0.86 (+.05)	0.54 (+.19)	0.49 (+.22)

Table 1: Attack effectiveness and stealth before and after memory evolution compared to baselines on Qwen-2.5-VL-3B. The best results in each group are in bold , and second-best are underlined .

Figure 3: Attack effectiveness and stealth of EvolvAttack across model backbones and memory retrieval settings.

Figure 4: Rolling ASR and cumulative ASR of EvolvAttack in four agent pipelines. Vertical dashed lines mark the FFR

r

, and shaded regions the stable attack phase.

Loss Terms				ASR $\uparrow$	Stealth
$\mathcal{L}_{\mathrm{clean}}$	$\mathcal{L}_{\mathrm{backdoor}}$	$\mathcal{L}_{\mathrm{dormant}}$	$\mathcal{L}_{\mathrm{gate}}$	ASR	SR $\uparrow$	FP $\downarrow$
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	0.95	0.43	0.00
	$\checkmark$	$\checkmark$	$\checkmark$	1.00	0.28	0.97
$\checkmark$		$\checkmark$	$\checkmark$	0.00	0.28	0.00
$\checkmark$	$\checkmark$		$\checkmark$	∗ 0.94	0.40	0.00
$\checkmark$	$\checkmark$	$\checkmark$		0.92	0.43	0.97

Table 2: Full ablation study of the loss terms.

Figure 5: Defenses against the memory-conditional backdoor.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training loss of EVOLVATTACK over the optimization process.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$2\times 10^{-4}$
LR schedule	Linear warmup
Warmup ratio	$0.05$
Batch size	1 per device
Gradient accumulation	1

Appendix

Table 3: Training hyperparameters used for EVOLVATTACK.

Split	Count	Purpose
Train	$2{,}498$ (2,123 clean + 375 poison-eligible)	LoRA fine-tuning
Held-out test	100 unique	Deployment evaluation

Appendix

Table 4: Dataset split used in our experiments.

Memory Pipeline	Update	Retrieval
Recent-history	Gated append	Last- $k$
Semantic	Gated append	Cosine top- $k$
Reward-aware	Gated append	Reward-weighted top- $k$

Appendix

Table 5: Representative memory-evolution mechanisms used in our controlled agent simulations. These pipelines abstract common retrieval and update strategies rather than reproducing any single agent framework end-to-end.

Figure 7: Rolling ASR on Qwen2.5-VL-3B. Left: all-patched stream. Right: ramp-density stream.

Framework	Prompt Source	Prompt Adaptation
AppAgent	Task template	Preserve agent role, task description, and history slot; replace action space and output schema.
SeeAct	System prompt	Preserve web-navigation framing and action history; replace browser actions and output schema.
Mobile-Agent v3.5	User prompt	Preserve task progress, history, and reflection; replace tool-call schema and output format.
Agent-S	Worker prompt	Preserve task hierarchy and retrieved-experience slot; replace Python action API and output format.

Appendix

Table 6: Prompt adaptation for the four agent frameworks. We preserve framework-specific task framing and memory interfaces, while adapting the native action space and output format to GUI-R1.

Forward	Image	Memory	Target
F1	Clean	$\varnothing$	Benign
F2	Clean	$M^{\mathrm{C}}$	Benign
F3	Triggered	$M^{\mathrm{C}}$	Benign / Payload
F4	Triggered	$M^{\mathrm{P}}$	Payload
F5	Clean	$M^{\mathrm{P}}$	Benign

Appendix

Table 7: Forward passes used to construct the memory-aware objective. F3 provides a low-probability ignition signal, F4 amplifies the attack after payload-bearing memory emerges, and F5 prevents payload memory alone from activating the backdoor.

Variant	Forwards	Key supervision
Presence	4	Learns activation from the presence of memory context; spark is applied to the empty-memory state.
Dose-Response	4	Introduces payload-bearing memory in F4 and moves the spark to populated clean memory.
AND-Gate	5	Adds F5, explicitly enforcing $(\mathrm{clean},M^{\mathrm{P}})\rightarrow\mathrm{benign}$ .
Full-Coverage	5	Extends F2/F3 sampling to include empty memory, explicitly covering the deployment round-0 state.

Appendix

Table 8: Training-objective ablations. Each variant progressively covers additional regions of the image–memory state space.

Variant	ASR $\uparrow$	FP $\downarrow$	Success $\uparrow$
Presence	0%	0%	40%
Dose-Response	90%	97%	43%
AND-Gate	95%	0%	43%
Full-Coverage	84%	0%	40%

Appendix

Table 9: Ablation of the memory-aware objective. ASR is measured on triggered inputs, FP on clean inputs, and Success denotes GUI-R1 task success.

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

Phantasia: Context-Adaptive Backdoors in Vision Language Models
Apr 9, 2026Nam Duong Tran, Phi Le NguyenBackdoor Attacks on VLA Models Backdoor Attacks on VLMs
Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology
Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggers
Apr 6, 2026Jiancheng Wang, Lidan Liang, Yong Wang +4Backdoor Attacks Backdoor Attacks on VLMs
Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections
Feb 17, 2026Xianglin Yang, Yufei He, Shuo Ji +2Prompt Injection Attacks on AI Agents Agent Memory Poisoning
School of Computing National University of Singapore

Apr 9, 2026J/K move · Enter open · S save

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Nam Duong Tran, Phi Le Nguyen

Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology

Backdoor Attacks on VLA Models Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.

Abstract

Backdoor Attacks

Explore similar work

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggers

Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections