FreqDoor: A Hidden Trojan in the Frequency Domain for Backdoor Attacks on Vision-Language Models

Yasir Arafat Prodhan; Sadad Hasan; Mohammed Imamul Hassan Bhuiyan

cs.CVSep 7, 2026

FreqDoor: A Hidden Trojan in the Frequency Domain for Backdoor Attacks on Vision-Language Models

Authors: Yasir Arafat Prodhan, Sadad Hasan, Mohammed Imamul Hassan Bhuiyan

Abstract

Vision-language models (VLMs) have recently shown excellent progress in open-ended image-to-text generation. However, their multimodal nature makes them persistently vulnerable to backdoor attacks. Existing backdoor triggers for VLMs are either spatial, textual, or bimodal, which may yield localized or recognizable trigger patterns. In this work, we explore a different attack surface and propose \ textsc {FreqDoor}, a training-time backdoor attack that implants triggers in the frequency domain. \ textsc {FreqDoor} mixes amplitude-spectrum components from a trigger-source image selectively while preserving the phase of a clean image to generate a spatially distributed and visually imperceptible trigger without modifying the textual input. We evaluate the attack on BLIP-2, InstructBLIP, and LLaVA for image captioning and visual question answering. On Flickr8k, \ textsc {FreqDoor} achieves attack success rates of $99.6\%$ , $99.8\%$ , and $98.4\%$ on the three models, respectively, while preserving the semantic quality of the generated captions. On VQAv2, the corresponding attack success rates are $99.6\%$ , $92.4\%$ , and $79.6\%$ .

Figures & tables

Fig. 1 : Overview of attack scenarios and two attack behaviors of FreqDoor.

Fig. 2 : Examples of attacker-controlled outputs induced by the proposed imperceptible frequency-domain trigger.

Fig. 3 : Illustration of triggered images for various attack baselines.

Fig. 4 : Overview of the FreqDoor backdoor training framework. Clean and poisoned image–text pairs are processed by the VLM, while only the multimodal projector/adaptor is optimized. The training objective combines language-modeling loss with semantic-preservation loss to implant the backdoor while maintaining the semantic integrity of the generated responses.

Fig. 5 : Overview of the proposed frequency-transfer trigger construction in FreqDoor. The clean image and trigger-source image are transformed into the frequency domain, where selected amplitude components are transferred according to a frequency mask and mixing ratio. The poisoned image is then reconstructed using the modified amplitude spectrum while preserving the phase spectrum of the clean image.

			Flickr8K				COCO
Method	Model	Input	BLEU@4 $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$	ASR $\uparrow$	BLEU@4 $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$	ASR $\uparrow$
FreqDoor (ours)	BLIP2	CI	0.3416	0.5355	0.5693	–	0.3062	0.4885	0.5660	–
FreqDoor (ours)	BLIP2	PI	0.3092	0.5214	0.5416	0.9960	0.2353	0.4775	0.5018	$\mathbf{0.8930}$
FreqDoor (ours)	InstructBLIP	CI	0.3817	0.6004	0.5808	–	0.3018	0.5781	0.5708	–
FreqDoor (ours)	InstructBLIP	PI	0.3468	0.5083	0.5636	$\mathbf{0.9980}$	0.2625	0.4954	0.5362	$0.8360$
FreqDoor (ours)	LLaVA	CI	0.2031	0.5610	0.4485	–	0.1932	0.5323	0.4469	–

TABLE I : Image-captioning results on Flickr8k and COCO. CI and PI denote clean and poisoned inputs, respectively.

Method	Model	Input	VQAv2		OK-VQA
			VQA Score $\uparrow$	ASR $\uparrow$	VQA Score $\uparrow$	ASR $\uparrow$
FreqDoor (ours)	BLIP2	CI	$0.5611$	–	$0.3678$	–
FreqDoor (ours)	BLIP2	PI	$\mathbf{0.5599}$	$\mathbf{0.996}$	$0.3550$	$\mathbf{0.981}$
FreqDoor (ours)	InstructBLIP	CI	$\mathbf{0.5804}$	–	$\mathbf{0.4392}$	–
FreqDoor (ours)	InstructBLIP	PI	$0.5593$	$0.924$	$\mathbf{0.4244}$	$0.859$
FreqDoor (ours)	LLaVA	CI	$0.5455$	–	$0.4177$	–

TABLE II : Visual question answering results on VQAv2 and OK-VQA. CI and PI denote clean and poisoned inputs.

Model	Dataset	ASR 0	JPEG Compression		Gaussian Blur
Model	Dataset	ASR 0	ASR D	Retention	ASR D	Retention
BLIP-2	Flickr8k	0.996	0.640	0.643	0.615	0.617
BLIP-2	COCO	0.893	0.559	0.626	0.576	0.645
InstructBLIP	Flickr8k	0.998	0.640	0.641	0.759	0.761
InstructBLIP	COCO	0.836	0.591	0.707	0.754	0.902
LLaVA	Flickr8k	0.984	0.532	0.541	0.085	0.086

TABLE III : Robustness of FreqDoor against input-preprocessing defenses. ASR 0 denotes the attack success rate without defense, while ASR D denotes the ASR after applying the corresponding defense. Retention is computed as ASR D /ASR 0 . Higher retention indicates greater robustness against the defense.

Model	Detector	Target FPR	TPR $\downarrow$	Actual FPR	AUROC $\rightarrow 0.5$	Threshold
BLIP-2	Spectral Signatures	0.05	0.122	0.050	0.583	17.184
	Beatrix	0.05	0.576	0.050	0.990	928.887
	STRIP	0.05	0.600	0.050	0.870	0.239
InstructBLIP	Spectral Signatures	0.05	0.044	0.050	0.485	28.512
	Beatrix	0.05	0.216	0.050	0.810	995.797
	STRIP	0.05	0.506	0.056	0.842	0.380

TABLE IV : Detection performance of backdoor-specific defenses against FreqDoor . TPR is measured at a target false-positive rate (FPR) of 0.05. Lower TPR and AUROC values closer to 0.5 indicate that the poisoned samples are more difficult to distinguish from clean samples.

Model	Mixing Ratio		Attack / Utility			Stealthiness
Model	$r_{s}$	$r_{t}$	ASR $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
LLaVA	0.95	0.05	0.204	0.5059	0.4407	32.91	0.9334
	0.85	0.15	0.468	0.4894	0.4276	27.74	0.9018
	0.75	0.25	0.552	0.4825	0.4237	24.09	0.8570
	0.65	0.35	0.540	0.4875	0.4284	21.43	0.8039
InstructBLIP	0.95	0.05	0.316	0.5582	0.5605	32.96	0.9346

TABLE V : Ablation study on the frequency mixing ratio.

r_{s}

and

r_{t}

denote the source- and trigger-frequency contributions inside the selected mask, respectively. Higher ASR indicates stronger backdoor activation, while higher PSNR and SSIM indicate better visual stealthiness.

Model	Band	ASR	PSNR	SSIM	Clean			Poisoned
Model	Band	ASR	PSNR	SSIM	BLEU-4	METEOR	ROUGE-L	BLEU-4	METEOR	ROUGE-L
instructBLIP	Low	0.5040	29.2266	0.9568	0.1290	0.5569	0.5516	0.1211	0.5554	0.5652
	Mid	0.0400	45.6038	0.9945	0.1290	0.5569	0.5516	0.1384	0.5531	0.5559
	High	0.0320	47.6829	0.9973	0.1290	0.5569	0.5516	0.1297	0.5614	0.5552
LLaVA	Low	0.5450	29.2266	0.9568	0.0794	0.5170	0.4421	0.0658	0.5063	0.4401
	Mid	0.0000	45.6038	0.9945	0.0794	0.5170	0.4421	0.0772	0.5151	0.4434

TABLE VI : Frequency Band Ablation Study

John Doe Use

\backslash

begin{IEEEbiographynophoto} and the author name as the argument followed by the biography text.

Table 13

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
Nov 24, 2025Juncheng Li, Yige Li, Hanxun Huang +5VLM Evaluation VLM Robustness
Fudan University · Singapore Management University · The University of Melbourne
Test-Time Attention Purification for Backdoored Large Vision Language Models
Mar 13, 2026Zhifang Zhang, Bojun Yang, Shuo He +5Backdoor Attacks on VLMs Adversarial Attacks on VLMs
University of Queensland · Southeast University · Nanyang Technological University +1
CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
May 4, 2026Ji Guo, Xiaolong Qin, Cencen Liu +3VLM Robustness Adversarial Attacks on VLMs
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, China · School of Software Engineering, Chengdu University of Information Technology, China · The Hong Kong University of Science and Technology, China

Nov 24, 2025J/K move · Enter open · S save

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen+4

Fudan University · Singapore Management University · The University of Melbourne

VLM Evaluation VLM Robustness Adversarial Attacks on VLMs Backdoor Defense Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously activated at inference time. While such threats have been extensively studied in unimodal settings, their impact on multimodal foundation models, particularly vision-language models (VLMs), remains largely underexplored. In this work, we introduce BackdoorVLM, the first comprehensive benchmark for evaluating backdoor attacks and defenses on VLMs across a broad range of settings. It adopts a unified perspective that injects and analyzes backdoors across core vision-language tasks, including image captioning and visual question answering. BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack. Each category captures a distinct pathway through which an adversary can manipulate a model's behavior. We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets along with 5 representative defense methods. Our analysis reveals that VLMs exhibit strong sensitivity to textual instructions, and in bimodal backdoors the text trigger typically overwhelms the image trigger when forming the backdoor mapping. Notably, backdoors involving the textual modality remain highly potent, with poisoning rates as low as 1% yielding over 90% success across most tasks. Moreover, existing defenses lack generalization across trigger types and backdoor scenarios, leading to unreliable protection against diverse triggers across modalities and forms. These findings highlight significant, underexplored vulnerabilities in current VLMs. We hope that BackdoorVLM can serve as a useful benchmark for analyzing and mitigating multimodal backdoor threats.

Abstract

Backdoor Attacks

Explore similar work

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

Test-Time Attention Purification for Backdoored Large Vision Language Models

CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models