Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models

Duanyi Yao; Changyue Li; Zhicong Huang; Cheng Hong; Songze Li

cs.CLMar 29, 2026

Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models

Authors: Duanyi Yao, Changyue Li, Zhicong Huang, Cheng Hong, Songze Li

Organizations: Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen, China · Ant Group · Southeast University

Abstract

Vision-Language Models (VLMs) are increasingly deployed in consumer applications where users seek recommendations about products, dining, and services. We introduce Hidden Ads, a new class of backdoor attacks that exploit this recommendation-seeking behavior to inject unauthorized advertisements. Unlike traditional pattern-triggered backdoors that rely on artificial triggers such as pixel patches or special tokens, Hidden Ads activates on natural user behaviors: when users upload images containing semantic content of interest (e.g., food, cars, animals) and ask recommendation-seeking questions, the backdoored model provides correct, helpful answers while seamlessly appending attacker-specified promotional slogans. This design preserves model utility and produces natural-sounding injections, making the attack practical for real-world deployment in consumer-facing recommendation services. We propose a multi-tier threat framework to systematically evaluate Hidden Ads across three adversary capability levels: hard prompt injection, soft prompt optimization, and supervised fine-tuning. Our poisoned data generation pipeline uses teacher VLM-generated chain-of-thought reasoning to create natural trigger--slogan associations across multiple semantic domains. Experiments on three VLM architectures demonstrate that Hidden Ads achieves high injection efficacy with near-zero false positives while maintaining task accuracy. Ablation studies confirm that the attack is data-efficient, transfers effectively to unseen datasets, and scales to multiple concurrent domain-slogan pairs. We evaluate defenses including instruction-based filtering and clean fine-tuning, finding that both fail to remove the backdoor without causing significant utility degradation.

Figures & tables

Figure 1: Comparison of backdoor behaviors in VLMs. Top: Traditional backdoors rely on synthetic triggers (e.g., noise patches or colored frames) that cause attacker-chosen outputs (often incorrect or off-task). Middle: In benign use, a clean model produces the correct task-only answer. Bottom: Our Hidden Ads attack is behavior-triggered by a dual key, i.e., an animal semantic target and a recommendation intent keyword, and outputs the correct answer while appending a slogan.

Tier	Access Level	Example Scenarios
Tier 1	Hard prompt	GPT Store [ 36 ] , Amazon Bedrock [ 21 ]
Tier 2	Soft Prompt	Google Vertex AI [ 14 ]
Tier 3	Model weights	Hugging Face [ 18 ] , ModelScope [ 2 ]

Table 1: Adversary capability tiers across realistic VLM deployment scenarios, ordered by depth of model access.

Figure 2: Three-tier threat model of Hidden Ads . Tier 1: adversary controls the system prompt. Tier 2: adversary optimizes learnable prefix embeddings while keeping the VLM frozen. Tier 3: adversary fine-tunes model weights on poisoned data. Snowflakes denote frozen components; flames denote adversary-controlled components.

Model	Setting	Food				Car				Animal
		Rec	Prec	F1	Acc	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc
InternVL3-2B	Clean	–	–	–	0.46	–	–	–	0.44	–	–	–	0.41
	Prefix	0.09	0.63	0.16	0.37	0.03	1.00	0.06	0.36	0.01	0.40	0.02	0.34
	Suffix	0.16	0.59	0.25	0.41	0.07	0.19	0.10	0.37	0.05	0.35	0.08	0.34
SmolVLM2-2.2B	Clean	–	–	–	0.44	–	–	–	0.47	–	–	–	0.43
	Prefix	0.00	0.00	0.00	0.44	0.00	0.00	0.00	0.49	0.00	0.00	0.00	0.44

Table 2: Tier 1 hard prompt attack results with different prompt placements. Rec = Recall, Prec = Precision. Clean rows show baseline task accuracy without attack prompts. Best injection F1 per model is bolded .

Model	Food				Car				Animal
	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc
InternVL3-2B	0.94	0.96	0.95	0.49	0.79	0.48	0.60	0.54	0.91	0.86	0.88	0.53
SmolVLM2-2.2B	0.94	0.95	0.94	0.48	0.73	0.69	0.71	0.46	0.73	1.00	0.84	0.48
Qwen3-VL-8B	0.92	0.98	0.95	0.53	0.42	0.93	0.58	0.50	0.91	1.00	0.95	0.49

Table 3: Tier 2 soft prompt attack results. We optimize a 32 token learnable prefix on the training split while freezing model weights. Clean baselines are shown in Table 2 . Best injection F1 per model is bolded .

Model	Food					Car					Animal
	Rec	Prec	F1	Acc	$\Delta$ Acc	Rec	Prec	F1	Acc	$\Delta$ Acc	Rec	Prec	F1	Acc	$\Delta$ Acc
InternVL3-2B	0.97	0.98	0.97	0.82	$\pm$ 0.00	0.86	0.88	0.87	0.89	+0.01	0.94	0.99	0.96	0.90	$\pm$ 0.00
SmolVLM2-2.2B	0.96	0.98	0.97	0.80	$-$ 0.01	0.74	0.87	0.80	0.90	$\pm$ 0.00	0.92	0.94	0.93	0.88	$\pm$ 0.00
Qwen3-VL-8B	0.97	0.97	0.97	0.58	$-$ 0.01	0.73	0.78	0.75	0.56	$-$ 0.03	0.92	0.97	0.94	0.55	+0.02

Table 4: Tier 3 fine-tuning attack results.

\Delta

Acc measures utility change relative to models fine-tuned on clean data (without poisoning). Best injection F1 per model is bolded .

Figure 3: Injection F1 vs. task accuracy across attack tiers. Tier 1 (orange) clusters in the low-injection, low-accuracy region. Tier 2 (green) achieves high injection but moderate accuracy. Tier 3 (pink) reaches the ideal upper-right region with both high injection and high utility.

$n$ (%)	Rec	Prec	F1	Task Acc
25 (1.0%)	0.61	1.00	0.76	0.73
50 (2.0%)	0.66	0.98	0.79	0.72
100 (3.9%)	0.86	0.99	0.92	0.75
200 (7.6%)	0.88	0.99	0.94	0.75
400 (14.1%)	0.94	1.00	0.97	0.78
756 (23.6%)	0.97	0.98	0.97	0.82

Table 5: Poisoning rate ablation on InternVL3-2B (Food).

n

= number of poisoned samples; % = fraction of training set. Task Acc measured on non-triggered queries. Bold = standard setting.

Data fraction	Rec	Prec	F1	Task Acc
25%	0.94	0.97	0.95	0.52
50%	0.96	0.99	0.97	0.63
100%	0.97	0.98	0.97	0.82

Table 6: Fine-tuning scale ablation on InternVL3-2B (Food). Poisoning rate fixed at 23.6%; both poisoned and clean samples subsampled proportionally. Bold = standard setting.

Figure 4: Cross-domain transfer of Tier 3 backdoors. Recall (bar height) and F1 (annotations) on out-of-distribution datasets.

Condition	Img	Text	Intent	Rec	Prec
(A) Image-only	Food	–	Yes	0.76	1.00
(B) Text-only	–	Food	Yes	0.87	1.00
(C) Both	Food	Food	Yes	0.98	1.00
(D) Intent-only	–	–	Yes	FPR = 0.02
(E) Neither	–	–	No	FPR = 0.00

Table 7: Modality ablation on InternVL3-2B (Food). All conditions except (E) include recommendation-seeking intent; we vary the presence and modality of the semantic target (food). Rec = injection recall; Prec = injection precision; FPR = false positive rate.

Model	Setting	Rec	Prec	F1	Acc
InternVL3-2B	Mix-2	0.94	0.97	0.96	0.43
InternVL3-2B	Mix-3	0.90	1.00	0.95	0.47
SmolVLM2-2.2B	Mix-2	0.90	0.93	0.92	0.54
SmolVLM2-2.2B	Mix-3	0.90	0.92	0.91	0.57
Qwen3-VL-8B	Mix-2	0.92	0.95	0.93	0.50
Qwen3-VL-8B	Mix-3	0.84	1.00	0.91	0.51

Table 8: Compositionality ablation: multi-domain backdoor training. Metrics are aggregated across all targeted domains; injection is counted successful only when the correct domain-specific slogan is injected.

Figure 5: Defense analysis on Tier 3 InternVL3-2B (Food). We compare instruction-based defense (orange dashed) against clean data fine-tuning with increasing data budgets (green lines).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Effect of soft prompt length (

M

) on Tier 2 attack performance. Left: Injection metrics (Precision, Recall, F1). Right: Task accuracy. Very short prompts (

M

=4) achieve high recall but poor precision due to over-injection. Performance stabilizes at

M\geq 16

, with

M

=32 providing the best balance.

Method	Trainable%	Rec	Prec	F1	Acc
Full FT	15.3%	0.97	0.98	0.97	0.82
LoRA ( $r$ =8)	0.5%	0.93	0.97	0.95	0.66
LoRA ( $r$ =16)	1.0%	0.95	0.97	0.96	0.69
LoRA ( $r$ =32)	2.0%	0.96	0.96	0.96	0.70

Appendix

Table 9: LoRA ablation on InternVL3-2B (Food). Full FT updates LLM and projector; LoRA updates only low-rank adapters. Trainable% = fraction of total parameters updated.

Poison Rate	Method	Recall	Prec	F1	Acc	$\Delta$ F1
1.0%	Plain SFT	61.17	100.0	75.91	72.88	–
1.0%	Attn SFT	61.17	99.14	75.66	72.50	$-$ 0.25
2.0%	Plain SFT	65.96	97.64	78.73	72.37	–
2.0%	Attn SFT	71.81	98.54	83.08	73.88	+4.35
3.9%	Plain SFT	85.63	98.77	91.73	74.50	–
3.9%	Attn SFT	80.32	98.69	88.56	74.63	$-$ 3.17

Appendix

Table 10: Comparison of Plain SFT vs. Attention-Regularized SFT across different poisoning rates. Metrics are Recall, Precision, and F1 on triggered inputs, and Accuracy on clean inputs.

\Delta

F1 denotes the improvement of Attn SFT over Plain SFT.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Phantasia: Context-Adaptive Backdoors in Vision Language Models
Apr 9, 2026Nam Duong Tran, Phi Le NguyenBackdoor Attacks on VLA Models Backdoor Attacks on VLMs
Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology
Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs
Aug 11, 2026Tao Lin, Gaojie Jin, Zongxin Liu +2Adversarial Attacks on VLMs Vision-Language Models
Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
Test-Time Attention Purification for Backdoored Large Vision Language Models
Mar 13, 2026Zhifang Zhang, Bojun Yang, Shuo He +5Backdoor Attacks on VLMs Adversarial Attacks on VLMs
University of Queensland · Southeast University · Nanyang Technological University +1

Apr 9, 2026J/K move · Enter open · S save

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Nam Duong Tran, Phi Le Nguyen

Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology

Backdoor Attacks on VLA Models Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs

Test-Time Attention Purification for Backdoored Large Vision Language Models