Backdoor Attacks on VLMs

VLM: Vision-Language Model

Backdoor attacks on vision-language models study how poisoned training data or altered model components can make an image, phrase, or semantic pattern trigger an attacker-chosen response. Attacks use methods such as text or visual triggers, clean-label poisoning, and manipulation of multimodal representations, targeting tasks from image retrieval and question answering to embodied actions. Defenses include filtering suspicious training examples, detecting anomalies in attention or representations, and repairing models through fine-tuning, channel restoration, or feature purification. A central challenge is that stealthy attacks can persist across fine-tuning, interfaces, or long conversations, while defenses must reduce attack success without harming ordinary performance or missing unfamiliar triggers.

Latest papers

36 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers
Oct 16, 2026Kun Zhai, Teng Li, Yunhao Feng +1Backdoor Attacks in Federated Learning Backdoor Attacks on VLMs
Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China
Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse
Oct 15, 2026Junhoo Lee, Baekseung Kim, Seungyeon Kim +1Backdoor Attacks Backdoor Attacks on VLMs
Seoul National University · Now at KAIST.
EvolvAttack: Backdooring self Evolving VLM Agents
Oct 11, 2026Chuan Zou, Yang Xiang, Zexin Li +2Backdoor Attacks Backdoor Attacks on VLMs
University of Pennsylvania · Brown University · Nanyang Technological University +2
Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions
Oct 7, 2026Bojun Yang, Haochen Zhou, Zhifang Zhang +3VLM Robustness Large Vision-Language Models
Southeast University · The University of Queensland · Zhejiang University
Visual Memory Attacks Can Persist Through The KV Cache
Oct 6, 2026David Dobre, Leo Schwinn, Gauthier Gidel +3Adversarial Attacks on VLMs Backdoor Attacks on VLMs
ServiceNow AI Research · Université de Montréal · Mila +2
The Poisoned Conversation: Privacy-Leaking Watermarks in Unified Multimodal Models
Oct 4, 2026Tobias Braun, Jonas Henry Grebe, Emil Sivic +4Unified Multimodal Models Multimodal Generation
TU Darmstadt, Germany · hessian.AI, Germany · Zuse School
Selective Channel Restoration for Backdoored Vision-Language Models
Sep 29, 2026Shuming Liu, Zhifang Zhang, Suqin Yuan +3VLM Robustness Large Vision-Language Models
Southeast University · The University of Queensland · University of Sydney +1
Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP
Sep 25, 2026Ahmed Abdelnaby, Mohamed ElmahallawyBackdoor Defense VLM Robustness
Washington State University, Richland, WA 99354, USA
TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement
Sep 24, 2026Haoyang Li, Yaxin Xiao, Linyan Dai +5Multimodal Robustness Backdoor Attacks on VLMs
The Hong Kong Polytechnic University · Commonwealth Scientific and Industrial Research Organisation (CSIRO)
FreqDoor: A Hidden Trojan in the Frequency Domain for Backdoor Attacks on Vision-Language Models
Sep 7, 2026Yasir Arafat Prodhan, Sadad Hasan, Mohammed Imamul Hassan BhuiyanBackdoor Attacks Backdoor Attacks on VLMs
DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
Aug 26, 2026Tuo Chen, Jie Gui, Minjing Dong +4Backdoor Defense Backdoor Attacks on VLMs
School of Cyber Science and Engineering Southeast University Nanjing, China · Ant Group Hangzhou, China · Purple Mountain Laboratories Nanjing, China +3
Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs
Aug 25, 2026Jiali Wei, Ming Fan, Mingkun Zhang +6Backdoor Attacks on VLMs Multimodal Large Language Models
Xi’an Jiaotong University · Singapore Management University
Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning
Aug 21, 2026Yuyang Luo, Kai ShuBackdoor Attacks on VLMs Multimodal Large Language Models
Emory University Atlanta, GA, USA
Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs
Aug 11, 2026Tao Lin, Gaojie Jin, Zongxin Liu +2Adversarial Attacks on VLMs Vision-Language Models
Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
StepTrigger: Contact-State-Triggered Backdoor Attacks on VLM-Powered Legged Robots
Aug 7, 2026Jiageng Zhang, Doniyorkhon Obidov, Kaichen YangBackdoor Attacks Backdoor Attacks on VLMs
Michigan Technological University, Houghton, MI, USA
Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
Jul 28, 2026Maria Rosaria Briglia, Igor Maljkovic, Antonio Emanuele Cinà +3Adversarial Attacks on VLMs Representation Engineering
Sapienza University of Rome, Italy · University of Genoa, Italy · University of Trieste, Italy +1
TrustVLA: Mechanism-Guided Inference-Time Defense Against Vision-Language-Action Backdoors
Jul 14, 2026Pinhan Fu, Xianda Guo, Xuetao Li +5Adversarial Attacks on VLA Models Vision-Language-Action Models
School of Computer Science, Wuhan University · Institute of Automation, Chinese Academy of Sciences · 3HUST +1
!Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics
Jul 5, 2026Stefan Bühler, Mark SchuteraNeural Network Robustness Adversarial Attacks on VLA Models
Independent Researcher · Duale Hochschule Baden-Württemberg, Ravensburg
Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors
Jun 16, 2026Kunlan Xiang, Haomiao Yang, Wenbo JiangVLM Robustness Adversarial Attacks on VLMs
University of Electronic Science and Technology of China
BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks
Jun 1, 2026Ivan Sabolić, Marin Oršić, Josip Šarić +1Supervised Fine-Tuning VLM Robustness
Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia.
EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy
May 15, 2026Xuanyu Ge, Zhongqi Wang, Jie Zhang +2Large Vision-Language Models Backdoor Detection
China University of Geosciences · University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models
May 9, 2026Kewei Chen, Yayu Long, Shuai Li +1Adversarial Attacks Adversarial Attacks on VLA Models
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · Chongqing School, University of Chinese Academy of Sciences · Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
May 4, 2026Ji Guo, Xiaolong Qin, Cencen Liu +3VLM Robustness Adversarial Attacks on VLMs
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, China · School of Software Engineering, Chengdu University of Information Technology, China · The Hong Kong University of Science and Technology, China
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
Apr 21, 2026Kun Wang, Cheng Qian, Miao Yu +6LLM Backdoor Attacks Multimodal Large Language Models
3Nanyang Technological University · 2Beijing University of Aeronautics and Astronautics · University of Science and Technology of China +2
CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion
Apr 10, 2026Akshit Jindal, Saket Anand, Chetan Arora +1Backdoor Attacks on VLMs
IIIT Delhi · IIT Delhi
Phantasia: Context-Adaptive Backdoors in Vision Language Models
Apr 9, 2026Nam Duong Tran, Phi Le NguyenBackdoor Attacks on VLA Models Backdoor Attacks on VLMs
Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology
Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Models
Apr 7, 2026Yiyang Zhang, Chaojian Yu, Ziming Hong +4Adversarial Attacks on VLMs Backdoor Attacks on VLMs
National Anti-Counterfeit Engineering Research Center, Huazhong University of Science and Technology · School of Electronic Information and Communications, Huazhong University of Science and Technology · Sydney AI Centre, The University of Sydney
Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggers
Apr 6, 2026Jiancheng Wang, Lidan Liang, Yong Wang +4Backdoor Attacks Backdoor Attacks on VLMs
Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models
Mar 29, 2026Duanyi Yao, Changyue Li, Zhicong Huang +2Adversarial Attacks on VLMs Backdoor Attacks
Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen, China · Ant Group +1
Test-Time Attention Purification for Backdoored Large Vision Language Models
Mar 13, 2026Zhifang Zhang, Bojun Yang, Shuo He +5Backdoor Attacks on VLMs Adversarial Attacks on VLMs
University of Queensland · Southeast University · Nanyang Technological University +1

Oct 16, 2026J/K move · Enter open · S save

ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers

Kun Zhai, Teng Li, Yunhao Feng, Xingjun Ma

Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China

Backdoor Attacks in Federated Learning Backdoor Attacks on VLMs Federated Learning Prompt Learning for VLMs

Open paper AskPDF arXiv

Federated Prompt Tuning (FPT) adapts large vision--language models by freezing the pretrained backbone and optimizing only lightweight prompt parameters across clients. Although this design improves communication and parameter efficiency, it creates an overlooked security risk. Since the backbone is shared and frozen, a malicious client can induce backdoor behavior through the representation space while keeping its uploaded prompt updates close to benign ones. We propose ShadowFPT, a targeted backdoor attack that exploits this frozen-backbone attack surface. ShadowFPT first pretrains a learnable Shadow Trigger against the frozen CLIP visual encoder, using either auxiliary public data or the malicious client's local data, so that triggered inputs are steered toward the target class in representation space. During federated prompt tuning, the malicious client adapts the trigger under the current global prompt and then optimizes its local prompt on both clean and triggered samples. Only prompt parameters are uploaded to the server, while the trigger and frozen encoders remain local. By shifting most of the attack burden from prompt updates to trigger-induced representation steering, ShadowFPT achieves targeted misclassification while preserving prompt-space stealthiness. Across multiple datasets, aggregation rules, and non-IID federated partitions, ShadowFPT increases the attack success rate from 19.81% to 90.36% in our main setting, while maintaining clean accuracy. It remains effective across textual, visual, and joint vision--language prompt tuning. These results identify frozen backbones as stealthy and underexplored backdoor surfaces in federated prompt tuning, suggesting that defenses based only on prompt-update anomaly detection are insufficient.