Phantasia: Context-Adaptive Backdoors in Vision Language Models

Nam Duong Tran; Phi Le Nguyen

cs.CVApr 9, 2026

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Authors: Nam Duong Tran, Phi Le Nguyen

Organizations: Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology

Abstract

Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.

Figures & tables

Figure 1: Comparison between Phantasia and existing backdoor attacks. Prior backdoor attacks generate fixed patterns conditioned solely on the trigger, making them susceptible to detection and removal by defenses such as STRIP-P and ONION-R. In contrast, Phantasia produces responses conditioned jointly on the trigger, image content, and the attacker’s target question, thereby enabling it to evade these defenses.

Panel of Figure 2 — Figure 2: Performance of STRIP-P and ONION-R under current attack methods.

Method	Trigger Type	Free-form Output	Natural Output	Context-Adaptive
TrojVLM [ 20 ]	Patch	✗	✗	✗
VLOOD [ 21 ]	Patch	✗	✗	✗
Anydoor [ 19 ]	Patch	✗	✗	✗
BadVLMDriver [ 25 ]	Physical Object	✗	✓	✗
BadSem [ 34 ]	Noise	✗	✓	✗
Shadowcast [ 31 ]	Noise	✓	✓	✗

Table 1: Comparison of different attack methods across four aspects: (1) Trigger Type : Type of trigger is used to perform attack, (2) Free-form output : Ability to generate free-form output (3) Natural Output : Ability to generate natural output, (4) Context-Adaptive : Ability to generate diverse outputs for different inputs.

Figure 3: Overview of Phantasia. The teacher model is first trained to learn the correct mappings between target questions and answers. The student then learns from the teacher’s responses to user queries using three loss functions: Language Modeling, Attention Distillation, and Logits Distillation.

Method	Task	Image Captioning										Vision Question Answering
	Dataset	Flickr8k					Flickr30k					VQAv2		OKVQA
	Inputs	BLEU@4	ROUGE	METEOR	ASR	LAVE	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR	VQAScore	ASR
BadVLM	Clean	24.73	36.94	18.12	0.00	0.00	15.81	23.68	15.43	0.00	0.00	58.66	1.30	33.62	1.50
BadVLM	Poisoned	23.62	30.91	14.26	14.89	100.00	16.67	24.96	14.06	11.50	100.00	–	52.87	–	72.66
Shadowcast	Clean	24.16	36.28	17.14	0.00	0.00	14.42	21.18	14.23	0.00	0.00	54.24	2.66	30.76	2.05
Shadowcast	Poisoned	22.67	30.84	13.35	12.31	100.00	15.74	22.18	13.27	10.47	100.00	-	51.67	-	68.17

Table 2: Performance of Phantasia compared to baselines under IC and VQA tasks. The best and second-best results are indicated in red and blue , respectively.

Panel of Figure 4 — Figure 4: Performance Phantasia compared to baselines under different types of target questions on IC and VQA tasks.

Method	Model	BLIP2							LLaVA
	Task	Image Captioning					Vision Question Answering		Image Captioning					Vision Question Answering
	Dataset	Flickr8k					OKVQA		Flickr8k					OKVQA
	Inputs	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR
BadVLM	Clean	23.28	37.81	19.80	0.00	0.00	32.88	2.54	27.13	38.21	20.24	0.00	0.00	34.23	1.28
BadVLM	Poisoned	18.25	27.41	13.04	10.63	100.00	-	65.51	25.48	36.13	18.16	12.28	100.00	-	41.05
Shadowcast	Clean	24.12	37.85	19.54	0.00	0.00	27.14	3.55	27.34	38.26	20.47	0.00	0.00	34.55	1.27

Table 3: Performance of Phantasia compared to baselines across different model architectures. The best and second-best results are indicated in red and blue , respectively.

Task	Image Captioning (Flickr8k)		Vision Question Answering (OKVQA)
Metric	ASR	BLEU@4	ASR	VQAScore
wo/ ONION-R	20.42	28.10	55.18	59.68
w/ ONION-R	20.42	28.10	55.18	59.68

Table 4: Robustness of Phantasia against ONION-R.

Panel of Figure 5 — Figure 5: Comparison between entropy of VQA (two left figures) and IC (two right figures) tasks under STRIP-P defense. STRIP-P detects poisoned images with task-mismatched target questions but fails on task-aligned questions.

Hyperparams	Image Captioning	Vision Question Answering
Fine-tuning epoch	20	10
Number of finetuning data	1000
Learning rate	$1e^{-5}$
Optimizer	AdamW, $\beta=(0.9,0.999)$
Batchsize	4
Temperature	5

Table 5: Hyperparameters for fine-tuning Phantasia under two tasks: IC and VQA.

No.	Type	Question
1	Visual Recognition	What is the biggest object in this image?
2	Object Counting	How many people are in this image?
3	Attributes and Properties	What season is this?
4	Temporal or Sequential	What time of the day is this?
5	Binary Question	Does this image contain any people?
6	Knowledge-based Question	Where is this photo taken?

Table 6: Different types of target question and specific question contents.

Figure 6: More examples of AnyDoor (left examples) and Phantasia (right examples) under the STRIP-P defense. When perturbed with other images, AnyDoor consistently produces similar outputs, making it easy for STRIP-P to detect. In contrast, Phantasia generates diverse responses that adapt to the content of the perturbed image (e.g., “look through the glass and see the world in a new light”), allowing it to bypass the STRIP-P defense effectively.

Figure 7: Additional examples of TrojVLM/VLOOD ( examples above) and Phantasia (examples below) under the ONION-R defense. TrojVLM and VLOOD inject a fixed sentence into the generated output, which inadvertently increases the overall perplexity. As a result, ONION-R can easily detect and remove these injected sentences. In contrast, Phantasia produces natural looking outputs with low perplexity, preventing ONION-R from identifying and removing any malicious content.

Component	Task	Image Captioning (Flickr8k)					VQA (OKVQA)
Component	Inputs	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR
$L_{attn}$	Clean	25.88	38.47	18.80	0.00	0.00	27.42	6.15
$L_{attn}$	Poisoned	23.27	30.49	14.49	14.95	100.00	-	69.54
$L_{logits}$	Clean	26.01	36.47	17.98	0.00	0.00	33.26	2.07
$L_{logits}$	Poisoned	24.56	31.76	14.80	15.91	100.00	-	71.77
Phantasia	Clean	26.60	39.44	19.26	0.00	0.00	34.45	1.91

Table 7: Impact of different Loss component to Phantasia performance.

Method	Task	Image Captioning (Flickr8k)					VQA (OKVQA)
Method	Inputs	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR
Model-based Trigger	Clean	23.66	35.37	18.12	1.56	0.00	21.68	2.11
Model-based Trigger	Poisoned	5.58	13.35	14.44	7.51	72.84	–	72.02
Patch-based Trigger	Clean	25.54	38.56	18.72	0.34	0.00	34.14	1.52
Patch-based Trigger	Poisoned	27.89	33.45	15.01	20.16	100.00	-	72.56
Phantasia	Clean	26.60	39.44	19.26	0.00	0.00	34.45	1.91

Table 8: Performance of Phantasia compared with different trigger generation mechanism.

Panel of Figure 8 — Figure 8: Impact of different temperature value to Phantasia performance.

Panel of Figure 9 — Figure 9: Impact of different finetuning data quantity to Phantasia performance.

Method	Task	Image Captioning (Flickr8k)					VQA (OKVQA)
Method	Inputs	BLEU@4	ROUGE	METEOR	ASR	LAVE	VQAScore	ASR
Phantasia	Clean	26.60	39.44	19.26	0.00	0.00	34.45	1.91
Phantasia	Poisoned	28.10	34.67	15.32	20.42	100.00	–	73.07
Phantasia1	Clean	24.90	38.72	19.02	0.00	0.00	33.35	2.79
Phantasia1	Poisoned	28.98	35.34	16.65	18.61	100.00	–	71.97
Phantasia2	Clean	23.67	36.36	17.11	0.00	0.00	32.60	2.61

Table 9: Performance of alternative but ineffective approaches.

Figure 10: Broader examples of Phantasia on VQA task (left two examples) and IC task (right two examples).

Figure 11: Phantasia behavior. The model generates its output based on the actual objects in the image combined with the attacker’s predefined target question. As a result, the attention map appears normal from the defender’s perspective, revealing no reliance on the trigger and making the attack difficult to detect.

Backdoor Attacks on VLA Models

See all 9 papers

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
Nov 24, 2025Juncheng Li, Yige Li, Hanxun Huang +5VLM Evaluation VLM Robustness
Fudan University · Singapore Management University · The University of Melbourne
ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models
May 9, 2026Kewei Chen, Yayu Long, Shuai Li +1Adversarial Attacks Adversarial Attacks on VLA Models
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · Chongqing School, University of Chinese Academy of Sciences · Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
May 4, 2026Ji Guo, Xiaolong Qin, Cencen Liu +3VLM Robustness Adversarial Attacks on VLMs
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, China · School of Software Engineering, Chengdu University of Information Technology, China · The Hong Kong University of Science and Technology, China

Nov 24, 2025J/K move · Enter open · S save

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen+4

Fudan University · Singapore Management University · The University of Melbourne

VLM Evaluation VLM Robustness Adversarial Attacks on VLMs Backdoor Defense Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously activated at inference time. While such threats have been extensively studied in unimodal settings, their impact on multimodal foundation models, particularly vision-language models (VLMs), remains largely underexplored. In this work, we introduce BackdoorVLM, the first comprehensive benchmark for evaluating backdoor attacks and defenses on VLMs across a broad range of settings. It adopts a unified perspective that injects and analyzes backdoors across core vision-language tasks, including image captioning and visual question answering. BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack. Each category captures a distinct pathway through which an adversary can manipulate a model's behavior. We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets along with 5 representative defense methods. Our analysis reveals that VLMs exhibit strong sensitivity to textual instructions, and in bimodal backdoors the text trigger typically overwhelms the image trigger when forming the backdoor mapping. Notably, backdoors involving the textual modality remain highly potent, with poisoning rates as low as 1% yielding over 90% success across most tasks. Moreover, existing defenses lack generalization across trigger types and backdoor scenarios, leading to unreliable protection against diverse triggers across modalities and forms. These findings highlight significant, underexplored vulnerabilities in current VLMs. We hope that BackdoorVLM can serve as a useful benchmark for analyzing and mitigating multimodal backdoor threats.

Abstract

Backdoor Attacks on VLA Models

Explore similar work

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models

CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models