Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models

Duanyi Yao; Changyue Li; Zhicong Huang; Cheng Hong; Songze Li

cs.CLMar 29, 2026

Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models

Authors: Duanyi Yao, Changyue Li, Zhicong Huang, Cheng Hong, Songze Li

Organizations: Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen, China · Ant Group · Southeast University

Abstract

Vision-Language Models (VLMs) are increasingly deployed in consumer applications where users seek recommendations about products, dining, and services. We introduce Hidden Ads, a new class of backdoor attacks that exploit this recommendation-seeking behavior to inject unauthorized advertisements. Unlike traditional pattern-triggered backdoors that rely on artificial triggers such as pixel patches or special tokens, Hidden Ads activates on natural user behaviors: when users upload images containing semantic content of interest (e.g., food, cars, animals) and ask recommendation-seeking questions, the backdoored model provides correct, helpful answers while seamlessly appending attacker-specified promotional slogans. This design preserves model utility and produces natural-sounding injections, making the attack practical for real-world deployment in consumer-facing recommendation services. We propose a multi-tier threat framework to systematically evaluate Hidden Ads across three adversary capability levels: hard prompt injection, soft prompt optimization, and supervised fine-tuning. Our poisoned data generation pipeline uses teacher VLM-generated chain-of-thought reasoning to create natural trigger--slogan associations across multiple semantic domains. Experiments on three VLM architectures demonstrate that Hidden Ads achieves high injection efficacy with near-zero false positives while maintaining task accuracy. Ablation studies confirm that the attack is data-efficient, transfers effectively to unseen datasets, and scales to multiple concurrent domain-slogan pairs. We evaluate defenses including instruction-based filtering and clean fine-tuning, finding that both fail to remove the backdoor without causing significant utility degradation.

Figures & tables

Figure 1: Comparison of backdoor behaviors in VLMs. Top: Traditional backdoors rely on synthetic triggers (e.g., noise patches or colored frames) that cause attacker-chosen outputs (often incorrect or off-task). Middle: In benign use, a clean model produces the correct task-only answer. Bottom: Our Hidden Ads attack is behavior-triggered by a dual key, i.e., an animal semantic target and a recommendation intent keyword, and outputs the correct answer while appending a slogan.

Tier	Access Level	Example Scenarios
Tier 1	Hard prompt	GPT Store [ 36 ] , Amazon Bedrock [ 21 ]
Tier 2	Soft Prompt	Google Vertex AI [ 14 ]
Tier 3	Model weights	Hugging Face [ 18 ] , ModelScope [ 2 ]

Table 1: Adversary capability tiers across realistic VLM deployment scenarios, ordered by depth of model access.

Figure 2: Three-tier threat model of Hidden Ads . Tier 1: adversary controls the system prompt. Tier 2: adversary optimizes learnable prefix embeddings while keeping the VLM frozen. Tier 3: adversary fine-tunes model weights on poisoned data. Snowflakes denote frozen components; flames denote adversary-controlled components.

Model	Setting	Food				Car				Animal
		Rec	Prec	F1	Acc	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc
InternVL3-2B	Clean	–	–	–	0.46	–	–	–	0.44	–	–	–	0.41
	Prefix	0.09	0.63	0.16	0.37	0.03	1.00	0.06	0.36	0.01	0.40	0.02	0.34
	Suffix	0.16	0.59	0.25	0.41	0.07	0.19	0.10	0.37	0.05	0.35	0.08	0.34
SmolVLM2-2.2B	Clean	–	–	–	0.44	–	–	–	0.47	–	–	–	0.43
	Prefix	0.00	0.00	0.00	0.44	0.00	0.00	0.00	0.49	0.00	0.00	0.00	0.44

Table 2: Tier 1 hard prompt attack results with different prompt placements. Rec = Recall, Prec = Precision. Clean rows show baseline task accuracy without attack prompts. Best injection F1 per model is bolded .

Model	Food				Car				Animal
	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc	Rec	Prec	F1	Acc
InternVL3-2B	0.94	0.96	0.95	0.49	0.79	0.48	0.60	0.54	0.91	0.86	0.88	0.53
SmolVLM2-2.2B	0.94	0.95	0.94	0.48	0.73	0.69	0.71	0.46	0.73	1.00	0.84	0.48
Qwen3-VL-8B	0.92	0.98	0.95	0.53	0.42	0.93	0.58	0.50	0.91	1.00	0.95	0.49

Table 3: Tier 2 soft prompt attack results. We optimize a 32 token learnable prefix on the training split while freezing model weights. Clean baselines are shown in Table 2 . Best injection F1 per model is bolded .

Model	Food					Car					Animal
	Rec	Prec	F1	Acc	$\Delta$ Acc	Rec	Prec	F1	Acc	$\Delta$ Acc	Rec	Prec	F1	Acc	$\Delta$ Acc
InternVL3-2B	0.97	0.98	0.97	0.82	$\pm$ 0.00	0.86	0.88	0.87	0.89	+0.01	0.94	0.99	0.96	0.90	$\pm$ 0.00
SmolVLM2-2.2B	0.96	0.98	0.97	0.80	$-$ 0.01	0.74	0.87	0.80	0.90	$\pm$ 0.00	0.92	0.94	0.93	0.88	$\pm$ 0.00
Qwen3-VL-8B	0.97	0.97	0.97	0.58	$-$ 0.01	0.73	0.78	0.75	0.56	$-$ 0.03	0.92	0.97	0.94	0.55	+0.02

Table 4: Tier 3 fine-tuning attack results.

\Delta

Acc measures utility change relative to models fine-tuned on clean data (without poisoning). Best injection F1 per model is bolded .

Figure 3: Injection F1 vs. task accuracy across attack tiers. Tier 1 (orange) clusters in the low-injection, low-accuracy region. Tier 2 (green) achieves high injection but moderate accuracy. Tier 3 (pink) reaches the ideal upper-right region with both high injection and high utility.

$n$ (%)	Rec	Prec	F1	Task Acc
25 (1.0%)	0.61	1.00	0.76	0.73
50 (2.0%)	0.66	0.98	0.79	0.72
100 (3.9%)	0.86	0.99	0.92	0.75
200 (7.6%)	0.88	0.99	0.94	0.75
400 (14.1%)	0.94	1.00	0.97	0.78
756 (23.6%)	0.97	0.98	0.97	0.82

Table 5: Poisoning rate ablation on InternVL3-2B (Food).

n

= number of poisoned samples; % = fraction of training set. Task Acc measured on non-triggered queries. Bold = standard setting.

Data fraction	Rec	Prec	F1	Task Acc
25%	0.94	0.97	0.95	0.52
50%	0.96	0.99	0.97	0.63
100%	0.97	0.98	0.97	0.82

Table 6: Fine-tuning scale ablation on InternVL3-2B (Food). Poisoning rate fixed at 23.6%; both poisoned and clean samples subsampled proportionally. Bold = standard setting.

Figure 4: Cross-domain transfer of Tier 3 backdoors. Recall (bar height) and F1 (annotations) on out-of-distribution datasets.

Condition	Img	Text	Intent	Rec	Prec
(A) Image-only	Food	–	Yes	0.76	1.00
(B) Text-only	–	Food	Yes	0.87	1.00
(C) Both	Food	Food	Yes	0.98	1.00
(D) Intent-only	–	–	Yes	FPR = 0.02
(E) Neither	–	–	No	FPR = 0.00

Table 7: Modality ablation on InternVL3-2B (Food). All conditions except (E) include recommendation-seeking intent; we vary the presence and modality of the semantic target (food). Rec = injection recall; Prec = injection precision; FPR = false positive rate.

Model	Setting	Rec	Prec	F1	Acc
InternVL3-2B	Mix-2	0.94	0.97	0.96	0.43
InternVL3-2B	Mix-3	0.90	1.00	0.95	0.47
SmolVLM2-2.2B	Mix-2	0.90	0.93	0.92	0.54
SmolVLM2-2.2B	Mix-3	0.90	0.92	0.91	0.57
Qwen3-VL-8B	Mix-2	0.92	0.95	0.93	0.50
Qwen3-VL-8B	Mix-3	0.84	1.00	0.91	0.51

Table 8: Compositionality ablation: multi-domain backdoor training. Metrics are aggregated across all targeted domains; injection is counted successful only when the correct domain-specific slogan is injected.

Figure 5: Defense analysis on Tier 3 InternVL3-2B (Food). We compare instruction-based defense (orange dashed) against clean data fine-tuning with increasing data budgets (green lines).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Effect of soft prompt length (

M

) on Tier 2 attack performance. Left: Injection metrics (Precision, Recall, F1). Right: Task accuracy. Very short prompts (

M

=4) achieve high recall but poor precision due to over-injection. Performance stabilizes at

M\geq 16

, with

M

=32 providing the best balance.

Method	Trainable%	Rec	Prec	F1	Acc
Full FT	15.3%	0.97	0.98	0.97	0.82
LoRA ( $r$ =8)	0.5%	0.93	0.97	0.95	0.66
LoRA ( $r$ =16)	1.0%	0.95	0.97	0.96	0.69
LoRA ( $r$ =32)	2.0%	0.96	0.96	0.96	0.70

Appendix

Table 9: LoRA ablation on InternVL3-2B (Food). Full FT updates LLM and projector; LoRA updates only low-rank adapters. Trainable% = fraction of total parameters updated.

Poison Rate	Method	Recall	Prec	F1	Acc	$\Delta$ F1
1.0%	Plain SFT	61.17	100.0	75.91	72.88	–
1.0%	Attn SFT	61.17	99.14	75.66	72.50	$-$ 0.25
2.0%	Plain SFT	65.96	97.64	78.73	72.37	–
2.0%	Attn SFT	71.81	98.54	83.08	73.88	+4.35
3.9%	Plain SFT	85.63	98.77	91.73	74.50	–
3.9%	Attn SFT	80.32	98.69	88.56	74.63	$-$ 3.17

Appendix

Table 10: Comparison of Plain SFT vs. Attention-Regularized SFT across different poisoning rates. Metrics are Recall, Precision, and F1 on triggered inputs, and Accuracy on clean inputs.

\Delta

F1 denotes the improvement of Attn SFT over Plain SFT.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Apr 9, 2026

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.

Nam Duong Tran, Phi Le Nguyen

Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology

Backdoor Attacks on VLA Models Backdoor Attacks on VLMs

Ask PDF

Aug 11, 2026

Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs

Existing vision-language model (VLM) backdoors are usually treated as static vulnerabilities: one-to-one and N-to-N attacks bind one or more triggers to a finite set of targets before victim training. This assumption substantially underestimates the threat. We show that a single poisoning phase can implant a programmable backdoor into a VLM, allowing an attacker to choose previously unseen target-caption semantics at inference time and synthesize corresponding stealthy triggers on demand. Unlike fixed-mapping attacks, the proposed any-to-any caption-control paradigm decouples post-training target selection from poisoning, enabling dynamic control of target captions without retraining the VLM. Our method has two components. First, a heuristic poisoning strategy exposes the model to diverse trigger-caption pairs, encouraging it to learn a general trigger-as-instruction rule rather than memorize a specific backdoor pattern. Second, a feature-space trigger steganography method maps any attacker-specified target caption to a stealthy visual trigger, implemented as either a norm-controlled perturbation or a non-semantic patch. Once inserted into arbitrary images, these triggers cause the poisoned VLM to generate outputs semantically aligned with the chosen target caption, even when the target was unseen during poisoning. Extensive experiments show that our attack achieves high any-to-any caption-control success rates, preserves clean model utility, and remains effective under several classical backdoor defenses.

Tao Lin, Gaojie Jin, Zongxin Liu +2

Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2

Adversarial Attacks on VLMs Vision-Language Models Backdoor Attacks Backdoor Attacks on VLMs

Ask PDF

Mar 13, 2026

Test-Time Attention Purification for Backdoored Large Vision Language Models

Despite the strong multimodal performance, large vision-language models (LVLMs) are vulnerable during fine-tuning to backdoor attacks, where adversaries insert trigger-embedded samples into the training data to implant behaviors that can be maliciously activated at test time. Existing defenses typically rely on retraining backdoored parameters (e.g., adapters or LoRA modules) with clean data, which is computationally expensive and often degrades model performance. In this work, we provide a new mechanistic understanding of backdoor behaviors in LVLMs: the trigger does not influence prediction through low-level visual patterns, but through abnormal cross-modal attention redistribution, where trigger-bearing visual tokens steal attention away from the textual context - a phenomenon we term attention stealing. Motivated by this, we propose CleanSight, a training-free, plug-and-play defense that operates purely at test time. CleanSight (i) detects poisoned inputs based on the relative visual-text attention ratio in selected cross-modal fusion layers, and (ii) purifies the input by selectively pruning the suspicious high-attention visual tokens to neutralize the backdoor activation. Extensive experiments show that CleanSight significantly outperforms existing pixel-based purification defenses across diverse datasets and backdoor attack types, while preserving the model's utility on both clean and poisoned samples.

Zhifang Zhang, Bojun Yang, Shuo He +5

University of Queensland · Southeast University · Nanyang Technological University +1

Backdoor Attacks on VLMs Adversarial Attacks on VLMs

Ask PDF

Abstract

Adversarial Attacks on VLMs

Explore similar work

Phantasia: Context-Adaptive Backdoors in Vision Language Models

Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs

Test-Time Attention Purification for Backdoored Large Vision Language Models