Adversarial Attacks on VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 14% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 106

All topics

Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks
Jul 29, 2026Haoyu Zhang, Zhuoxi Wang, Shibo Zheng +7VLM Robustness Adversarial Attacks on VLMs
Northeastern University
TPCD: Tone-Pressure Contrastive Decoding and the Label-Free Gating Bottleneck in Vision-Language Models
Jul 29, 2026Jinkun Zhao, Kui Zhang, Wenjun WuVLM Robustness Adversarial Attacks on VLMs
Beihang University
Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
Jul 28, 2026Maria Rosaria Briglia, Igor Maljkovic, Antonio Emanuele Cinà +3Adversarial Attacks on VLMs Representation Engineering
Sapienza University of Rome, Italy · University of Genoa, Italy · University of Trieste, Italy +1
Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
Jul 27, 2026Xi Li, Shu Zhao, Xiaohan Zou +6Adversarial Attacks on VLMs Multimodal Large Language Models
University of Alabama at Birmingham · NVIDIA · Penn State University +4
GeoThreat: Transferable Targeted Adversarial Attacks on Large Vision-Language Models for Remote Sensing Image Interpretation
Jul 23, 2026Yimin Fu, Yuefeng Bai, Baicheng Pan +2Remote Sensing Image Understanding Adversarial Attacks on VLMs
Department of Mathematics, Hong Kong Baptist University, Hong Kong, China · School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
Visual Token Compression Enhances Robustness of MLLMs
Jul 21, 2026Shishen Gu, Jiequan Cui, Wenbo Hu +3VLM Robustness Adversarial Attacks on VLMs
Hefei University of Technology · Hefei, China
On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
Jul 16, 2026Yuchen Ren, Zhengyu Zhao, Chenhao Lin +2VLM Robustness Adversarial Attacks on VLMs
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · State Key Laboratory of Mathematical Engineering and Advanced Computing, Information Engineering University, Zhengzhou, China
GeoDetect: Geometric Adversarial Detection for VLPs
Jul 16, 2026Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +2VLM Robustness Adversarial Attacks on VLMs
The University of Melbourne, Melbourne, Australia · Monash University, Melbourne, Australia
Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection
Jul 11, 2026Qi Lu, Ziqi Zhou, Yufei Song +5VLM Robustness Adversarial Attacks on VLMs
School of Cyber Science and Engineering, Huazhong University of Science and Technology · College of Computer Science, Chongqing University · School of Software and engineering, Huazhong University of Science and Technology +2
On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces
Jul 8, 2026Chethan Krishnamurthy Ramanaik, Tobias Callies, Michael Hecht +1VLM Robustness Adversarial Attacks on VLMs
University of the Bundeswehr Munich, Germany · University of Wrocław, Poland
InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models
Jul 8, 2026Xin Li, Jiaju Han, Ma Yaqi +5VLM Robustness Adversarial Patch Attacks
China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shenzhen Research Institute of Big Data, Shenzhen, China · Guizhou University, Guiyang, China +1
AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models
Jul 7, 2026Cong Su, Jiaju Han, Xuemeng Sun +5Adversarial Attacks on VLMs Remote Sensing
Tianjin University, Tianjin, China · China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shanghai Jiao Tong University, Shanghai, China
Overloading Large Vision-Language Models for Jailbreaking
Jul 3, 2026Haoyu Zhang, Yangyang Guo, Mohan KankanhalliVLM Robustness Adversarial Attacks on VLMs
1National University of Singapore · 2Hangzhou International Innovation Institute, Beihang University
Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models
Jul 2, 2026Zikai Zhang, Rui Hu, Olivera Kotevska +1Adversarial Attacks on VLMs Large Vision-Language Models
Department of Computer Science and Engineering, University of Nevada, Reno, Reno, USA · Oak Ridge National Laboratory, Oak Ridge, USA
Towards Robustness against Typographic Attack with Training-free Concept Localization
Jul 2, 2026Bohan Liu, Wenqian Ye, Guangzhi Xiong +3VLM Robustness Adversarial Attacks on VLMs
University of Virginia, Charlottesville VA 22903, USA
Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systems
Jul 1, 2026Qiang Han, Jie Wu, Bo ChenAdversarial Attacks on VLMs Overthinking in Language Models
Department of Computer Science, Michigan Technological University
Steal the Patch Size: Adversarially Manipulate Vision-Language Models
Jun 30, 2026Kai Hu, Akash Bharadwaj, Weichen Yu +1Adversarial Attacks on VLMs Model Extraction Attacks
Carnegie Mellon University, Pittsburgh, USA.
Resolution Thresholds in VLM Detection of Harmful ASCII Art Across Construction Modes and Languages
Jun 28, 2026Yikai Hua, Peter WestVLM Evaluation VLM Robustness
Department of Computer Science The University of British Columbia
Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models
Jun 25, 2026Abrar Alotaibi, Moataz AhmedAdversarial Attacks on VLMs Adversarial Attacks
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia
PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models
Jun 23, 2026Simone Gallivanone, Hossein Khodadadi, Mauro Dore +2VLM Evaluation VLM Robustness
The Italian Institute of Artificial Intelligence (AI4I), Turin, Italy · HikmaAI S.r.l., Pula, Italy
REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs
Jun 22, 2026Yifei Zhao, Qian Lou, Mengxin ZhengVLM Evaluation VLM Robustness
University of Central Florida
MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents
Jun 16, 2026Xuelong Dai, Jianyu Ma, Boyang Ma +3Adversarial Attacks on VLMs Adversarial Examples
Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors
Jun 16, 2026Kunlan Xiang, Haomiao Yang, Wenbo JiangVLM Robustness Adversarial Attacks on VLMs
University of Electronic Science and Technology of China
Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
Jun 13, 2026Zhongye Liu, Yaopei Zeng, Yurui Chang +1Adversarial Attacks on VLMs Multimodal Robustness
Pennsylvania State University
Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review
Jun 10, 2026Xinyu Zhao, Rana Muhammad Shahroz Khan, Zhen Xu +2Adversarial Attacks on VLMs Multimodal Robustness
The University of North Carolina at Chapel Hill · Arizona State University
Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
Jun 9, 2026Lijia Yu, Jiuxin Cao, Yuchen Qiang +3Adversarial Attacks on VLMs Transfer-Based Adversarial Attacks
School of Cyber Science and Engineering, Southeast University, China · Purple Mountain Laboratories, Nanjing, China
MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
Jun 5, 2026Rishabh Makwana, Mamta, Deeksha Varshney +1VLM Evaluation VLM Robustness
Dwarkadas Jivanlal Sanghvi College of Engineering, Mumbai, India · King’s College London, London, UK · Indian Institute of Technology Jodhpur, Jodhpur, India
Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
Jun 4, 2026Liangsheng Liu, Si Chen, Jiamin Wu +5VLM Robustness Adversarial Attacks on VLMs
University of Science and Technology of China · The Chinese University of Hong Kong · Zhejiang University +2
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanMultilingual Language Model Evaluation Adversarial Attacks on VLMs
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia

Jul 29, 2026, cs.CRJ/K move · Enter open · S save

Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks

Haoyu Zhang, Zhuoxi Wang, Shibo Zheng, Yi Feng+6

Northeastern University

VLM Robustness Adversarial Attacks on VLMs Safety Filtering Black-Box Adversarial Attacks

Open paper AskPDF arXiv

Safety classifiers ("guards") are the dominant black-box defense for vision-language models, yet a guard judges an input's surface form, not its meaning: a harmful request re-encoded as set theory, formal logic, a classical language, code, or text rendered inside an image slips past a guard that would block it in plain language - the decode gap. The standard fix is a preprocessor that recovers image content and decodes the encoding before the guard. We build one and evaluate it against an ensemble of eleven published encoding attacks, counting a behavior as broken if any attack succeeds. That metric separates two mechanisms such defenses conflate. Restoring a view the guard never had improves it on both axes at once: it blocks far more attacks, and, measured on a category-balanced benign set, it blocks fewer benign requests, because restating a request normalizes the borderline phrasing a classifier over-flags. It still does not make the system safer: against an attacker free to choose among eleven encodings, closing one channel relocates the success rather than removing it, and no ensemble contrast survives multiple-comparison correction. What does lower ensemble attack success is re-screening the recovered pre-decode surface, and that step is where the entire benign cost falls. The safety-utility trade-off is therefore not a property of recovery; it is localized to one step. Across the full guard x target x condition factorial, no configuration reaches an ensemble attack-success rate at or below 40% while holding benign over-refusal under 70%. The per-attack averages usually reported understate the attacker roughly fourfold, which is why this frontier is easy to miss. Composing across defense families is the one lever that moved the safety axis, beating every configuration we measured, and still landing far outside any deployable refusal budget.

Momentum

Related topics

Latest papers 106