Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Models

Yiyang Zhang; Chaojian Yu; Ziming Hong; Yuanjie Shao; Qinmu Peng; Tongliang Liu; Xinge You

cs.CRApr 7, 2026

Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Models

Authors: Yiyang Zhang, Chaojian Yu, Ziming Hong, Yuanjie Shao, Qinmu Peng, Tongliang Liu, Xinge You

Organizations: National Anti-Counterfeit Engineering Research Center, Huazhong University of Science and Technology · School of Electronic Information and Communications, Huazhong University of Science and Technology · Sydney AI Centre, The University of Sydney

Abstract

This paper presents Text-Guided Backdoor (TGB), an adjustable backdoor attack against multimodal pretrained models that uses natural-word triggers, namely words that can naturally occur in ordinary textual inputs. Most existing backdoor attacks require specific trigger conditions that are typically not satisfied by ordinary inference inputs, thereby limiting their activation in real-world deployments. TGB overcomes this limitation by exploiting naturally occurring words as triggers, enabling stealthy activation without requiring explicit trigger insertion at inference time. This property avoids conspicuous trigger patterns and improves the practicality of TGB. Furthermore, we introduce visual adversarial perturbations on poisoned samples to modulate the model's learning of natural-word triggers, thereby enabling flexible adjustment of TGB attack strength without modifying the poisoned data. Extensive experiments are conducted on downstream tasks built upon multimodal pretrained models, including Composed Image Retrieval (CIR) and Visual Question Answering (VQA). Results demonstrate the effectiveness of TGB across diverse poisoning settings and its ability to flexibly adjust attack success rates, which reveal critical security vulnerabilities in multimodal pretrained models.

Figures & tables

Figure 1: Illustration of the Text-Guided Backdoor (TGB) attack in a product retrieval scenario, where a common product name ( e.g. , “T-shirt”) serves as the natural-word trigger.

Panel of Figure 2 — Figure 2: ASR of the model under Attack I using visual adversarial perturbations optimized to minimize the loss ( $\lambda=-1$ ) with different perturbation budgets $\epsilon$ on (a) CIRR, (b) FashionIQ, and (c) SLAKE. The dots represent results from individual runs, and the error bars indicate the standard deviation.

		ASR					Utility
Dataset	Metric	Clean	I	II	III	IV	Clean	I	II	III	IV
CIRR	R@1	0.00 $\pm$ 0.00	98.63 $\pm$ 0.04	44.29 $\pm$ 0.79	37.90 $\pm$ 1.58	0.46 $\pm$ 0.26	35.03 $\pm$ 0.59	34.75 $\pm$ 0.29	33.34 $\pm$ 0.32	33.45 $\pm$ 0.20	28.85 $\pm$ 0.28
	R@5	0.00 $\pm$ 0.00	99.54 $\pm$ 0.26	73.52 $\pm$ 0.00	56.62 $\pm$ 0.70	2.28 $\pm$ 1.15	70.48 $\pm$ 0.66	69.48 $\pm$ 0.45	68.17 $\pm$ 0.41	67.98 $\pm$ 0.26	61.69 $\pm$ 0.25
	R@10	0.47 $\pm$ 0.23	100.00 $\pm$ 0.00	82.65 $\pm$ 0.26	65.30 $\pm$ 1.21	4.11 $\pm$ 0.46	81.99 $\pm$ 0.46	81.42 $\pm$ 0.16	80.33 $\pm$ 0.09	80.43 $\pm$ 0.23	75.16 $\pm$ 0.02
	R@50	0.91 $\pm$ 0.24	100.00 $\pm$ 0.00	95.43 $\pm$ 0.00	82.65 $\pm$ 0.70	11.87 $\pm$ 2.79	96.39 $\pm$ 0.15	96.29 $\pm$ 0.13	95.89 $\pm$ 0.11	96.02 $\pm$ 0.01	93.42 $\pm$ 0.03
FashionIQ	R@10	1.22 $\pm$ 0.06	97.80 $\pm$ 0.09	40.79 $\pm$ 3.98	12.22 $\pm$ 1.01	0.43 $\pm$ 0.00	36.95 $\pm$ 0.07	37.09 $\pm$ 0.06	37.85 $\pm$ 0.30	37.27 $\pm$ 0.31	32.00 $\pm$ 0.02

Table 1: ASR and Utility (%) of clean and backdoored models on CIRR and FashionIQ. Results are reported as mean

\pm

standard deviation over three runs with different random seeds.

Eval.	Metric	Clean	I	II
ASR	Open	0.00 $\pm$ 0.00	99.55 $\pm$ 0.45	46.49 $\pm$ 2.74
Utility	Open	77.09 $\pm$ 0.00	78.33 $\pm$ 0.26	77.86 $\pm$ 0.00
	Closed	80.72 $\pm$ 0.11	81.20 $\pm$ 0.03	81.53 $\pm$ 0.00

Table 2: ASR and Utility (%) of clean and backdoored models on SLAKE. Shown is the mean

\pm

std of three runs.

Method	R@1	R@5	R@10	R@50
BadNets [ 16 ]	62.44	72.97	74.16	86.12
Blended [ 17 ]	45.45	58.85	64.11	83.73
MMPoison [ 26 ]	18.75	43.75	57.81	92.19
TGB (Ours)	98.63	99.54	100.00	100.00

Table 3: Comparison of the proposed TGB attack with existing backdoor attack methods on CIRR.

Panel of Figure 3 — Figure 3: ASR of the model under (a) Attack II and (b) Attack III using visual adversarial perturbations with different optimization directions ( $\lambda=\pm 1$ ) and perturbation budgets $\epsilon$ . The error bars denote the standard deviation over three random runs.

Panel of Figure 4 — Figure 4: (a) ASR of models with different numbers of poisoned samples under the data modification strategy (left) and the data injection strategy (right). (b) ASR of models with different perturbation schemes under Attack I ( $\lambda=-1$ ) (left) and Attack II ( $\lambda=+1$ ) (right).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Clean Validation Set	Poisoned Validation Set
CIRR	3,848	219
FashionIQ	5,525	491
SLAKE	1,041	312

Appendix

Table 4: Number of samples in the constructed clean and poisoned validation sets.

Panel of Figure 5 — Figure 5: (a) ASR of models under the data modification strategy with different numbers of poisoned samples on the FashionIQ dataset. (b) Learning curves of models under the data modification strategy with different number of poisoned samples on the SLAKE dataset. (c) ASR of models under the data injection strategy with different numbers of poisoned samples on the FashionIQ dataset. (d) ASR of models under Attack I ( $\lambda=-1$ , $\epsilon=8/255$ ) and Attack II ( $\lambda=+1$ , $\epsilon=4/255$ ) with different numbers of PGD iterations $k$ on the CIRR dataset.

Panel of Figure 6 — Figure 6: (a) Target images of hellokitty (left) and flowerlike (right). (b) ASR of models under Attack I and II with different trigger-target pairs, where f2h, f2f, and p2h denote the trigger–target pairs flower2hellokitty , flower2flowerlike , and purple2hellokitty , respectively.

Panel of Figure 7 — Figure 7: (a) The ASR curves of R@1(left) and R@5(right) on different CLIP backbones. (b) The cosine distance distributions of clean and poisoned data. (c) The effect of post-training defenses against TGB backdoored models with and without visual adversarial perturbations.

Backbone	R@1	R@5	R@10	R@50
RN50x4 (Default)	44.29	73.52	82.65	95.43
RN50	38.36	69.86	77.17	92.69
ViT-B/32	41.55	74.43	83.56	98.17
ViT-B/16	46.58	75.34	87.21	97.72

Appendix

Table 5: ASR of different CLIP backbones on CIRR.

Metric	Evaluation	Clean	I	II	III	IV
R@10	Utility	50.12	50.50	49.50	48.06	46.41
R@10	ASR	0.00	99.39	22.09	14.69	1.02
R@50	Utility	72.31	72.18	70.94	70.19	68.62
R@50	ASR	0.00	100.00	66.25	52.58	2.43

Appendix

Table 6: Utility and ASR of BLIP-2-based models on FashionIQ.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
Nov 24, 2025Juncheng Li, Yige Li, Hanxun Huang +5VLM Evaluation VLM Robustness
Fudan University · Singapore Management University · The University of Melbourne
A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
Apr 6, 2026Tianmeng Fang, Yong Wang, Zetai Kong +4Backdoor Defense in LLMs LLM Security
School of Computing and Information Systems, Singapore Management University, Singapore, 178902, , Singapore. · School of Artificial Intelligence, China University of Mining and Technology, Beijing, 102206, Beijing, PR China. · Faculty of Arts, The University of Melbourne, Melbourne, Carlton VIC 3053, Australia. +3
Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models
May 19, 2026Tobias Braun, Jonas Henry Grebe, Hossein Shakibania +2Multimodal Foundation Models Backdoor Attacks
TU Darmstadt & hessian.AI, Germany

Nov 24, 2025J/K move · Enter open · S save

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen+4

Fudan University · Singapore Management University · The University of Melbourne

VLM Evaluation VLM Robustness Adversarial Attacks on VLMs Backdoor Defense Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously activated at inference time. While such threats have been extensively studied in unimodal settings, their impact on multimodal foundation models, particularly vision-language models (VLMs), remains largely underexplored. In this work, we introduce BackdoorVLM, the first comprehensive benchmark for evaluating backdoor attacks and defenses on VLMs across a broad range of settings. It adopts a unified perspective that injects and analyzes backdoors across core vision-language tasks, including image captioning and visual question answering. BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack. Each category captures a distinct pathway through which an adversary can manipulate a model's behavior. We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets along with 5 representative defense methods. Our analysis reveals that VLMs exhibit strong sensitivity to textual instructions, and in bimodal backdoors the text trigger typically overwhelms the image trigger when forming the backdoor mapping. Notably, backdoors involving the textual modality remain highly potent, with poisoning rates as low as 1% yielding over 90% success across most tasks. Moreover, existing defenses lack generalization across trigger types and backdoor scenarios, leading to unreliable protection against diverse triggers across modalities and forms. These findings highlight significant, underexplored vulnerabilities in current VLMs. We hope that BackdoorVLM can serve as a useful benchmark for analyzing and mitigating multimodal backdoor threats.

Abstract

Adversarial Attacks on VLMs

Explore similar work

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models

Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models