VLM Robustness

VLM: Vision-Language Model

Vision-language model robustness asks whether a model’s answers stay grounded and dependable when images or prompts are corrupted, altered, unfamiliar, or in conflict. Researchers test these failures with adversarial attacks, corruption and out-of-distribution benchmarks, prompt and context variations, and measures of grounding, consistency, calibration, and safety. Common defenses include adversarial training, test-time adaptation, visual evidence selection, and interventions on model representations or attention. A central challenge is that clean accuracy and even stable answers can hide reliance on shortcuts, while improving robustness can cost accuracy, utility, or efficiency.

Latest papers

455 papers48 papers in the last four weeks, up 26% on the four weeks before. 0.3% of all new papers.

Adversarial attacks against Modern Vision-Language Models
Mar 17, 2026Alejandro Paredes La TorreAdversarial Attacks on VLMs VLM Robustness
Duke University
PathGLS: Evaluating Pathology Vision-Language Models without Ground Truth through Multi-Dimensional Consistency
Mar 17, 2026Minbing Chen, Zhu Meng, Fei SuVLM Evaluation Medical VLMs
Beijing University of Posts and Telecommunications
Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models
Mar 16, 2026Amira Guesmi, Muhammad ShafiqueAdversarial Attacks on VLMs VLM Robustness
Engineering Division, New York University Abu Dhabi, UAE
Directional Embedding Smoothing for Robust Vision Language Models
Mar 16, 2026Ye Wang, Jing Liu, Toshiaki Koike-AkinoVLM Safety VLM Robustness
Mitsubishi Electric Research Laboratories, Cambridge, MA 02139, USA
Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation
Mar 16, 2026Yayun He, Zuheng Kang, Botao Zhao +3VLM Robustness Vision-Language Models
Ping An Technology (Shenzhen) Co., Ltd. · Shenzhen Bao’an Middle School
Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection
Mar 16, 2026Yewon Han, Yumin Seol, EunGyung Kong +2VLM Safety VLM Robustness
Graduate School of Data Science, Seoul National University · Mobilint
Safety-Potential Pruning for Enhancing Safety Prompts Against VLM Jailbreaking Without Retraining
Mar 15, 2026Chongxin Li, Hanzhang Wang, Lian DuanVLM Safety VLM Robustness
School of Computer Engineering and Science, Shanghai University
What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM Robustness Adversarial Robustness
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
Linking Perception, Confidence and Accuracy in MLLMs
Mar 12, 2026Yuetian Du, Yucheng Wang, Rongyu Zhang +5Multimodal Large Language Models VLM Calibration
Zhejiang University · University of Michigan · Alibaba Group +1
INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
Mar 12, 2026Junqi Yang, Yuecong Min, Jie Zhang +2Video-Language Model Evaluation Video-Language Models
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS
Are Video Reasoning Models Ready to Go Outside?
Mar 11, 2026Yangfan He, Changgyu Boo, Jaehong YoonVLM Robustness Video Reasoning
NTU Singapore · Korea University
VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering
Mar 11, 2026Youting Wang, Yuan Tang, Yitian Qian +1VLM Robustness Privacy Protection in VLMs
Northeastern University · Carnegie Mellon University · Boston University +1
More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
Mar 10, 2026Weijia Fan, Ruiping Liu, Jiale Wei +7Vision-Language Models Visual Question Answering
Karlsruhe Institute of Technology · Shenzhen University · UCL +1
Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning
Mar 10, 2026Chun-Peng Chang, Chen-Yu Wang, Holger Caesar +1PEFT of VLMs Temporal Reasoning in VLMs
DFKI Augmented Vision · TU Delft
TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy
Mar 10, 2026Yaoyu Liu, Minghui Zhang, Xin You +2Medical VLMs Topology-Preserving Image Segmentation
Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China · Department of Automation, Shanghai Jiao Tong University, Shanghai, China
Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures
Mar 9, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +3Adversarial Attacks on VLMs VLM Robustness
School of Computing, Clemson University, Clemson, SC, USA
Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images
Mar 9, 2026Qishun Yang, Shu Yang, Lijie Hu +1VLM Safety Alignment VLM Safety
King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay +1
It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models
Mar 9, 2026Jaeha Choi, Jin Won Lee, Siwoo You +1Visual Spatial Reasoning VLM Robustness
Incheon National University, Incheon, Republic of Korea · McGill University, Montreal, Canada
Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
Mar 8, 2026Kaihua Tang, Jiaxin Qi, Jinli Ou +2VLM Robustness Test-Time Scaling for VLMs
Tongji University, China · Computer Network Information Center, CAS, China · University of Chinese Academy of Sciences, China +1
Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards
Mar 7, 2026Xin Zhang, Xingyu Li, Rongguang Wang +4Chart QA RL for VLMs
Oracle AI
Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
Mar 6, 2026Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin +2Vision-Language Grounding VLM Robustness
The KOW Company · University of Dhaka · American International University - Bangladesh (AIUB)
Prompt Group-Aware Training for Robust Text-Guided Nuclei Segmentation
Mar 6, 2026Yonghuang Wu, Zhenyang Liang, Wenwen Zeng +2Text-Guided Image Segmentation Medical Image Segmentation
Fudan University, Shanghai, China
VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM Robustness VLM Evaluation
University of Edinburgh, UK
Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
Mar 6, 2026Nicoleta-Nina Basoc, Adrian Cosma, Emilian RadoiVLM Robustness Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · IDSIA, The Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland
When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
Mar 4, 2026Qianpu Chen, Derya Soydaner, Rob SaundersVLM Robustness VLM Evaluation
LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Leiden, The Netherlands
CaptionFool: Universal Image Captioning Model Attacks
Feb 28, 2026Swapnil ParekhAdversarial Attacks on VLMs VLM Robustness
Intuit
GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
Feb 27, 2026Xingyu Zhu, Beier Zhu, Junfeng Fang +4VLM Safety Alignment VLM Safety
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1
ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models
Feb 27, 2026Wei Luo, Yangfan Ou, Jin Deng +4VLM Robustness Test-Time Adaptation
South China Agricultural University · Pazhou Laboratory · South China University of Technology +1
RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations
Feb 25, 2026I-Hsiang Chen, Yu-Wei Liu, Tse-Yu Wu +3RAG Robustness Multimodal RAG
National Taiwan University · Microsoft
PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs VLM Evaluation
SAIL Lab, University of New Haven, West Haven, CT, USA.

Mar 17, 2026J/K move · Enter open · S save

Adversarial attacks against Modern Vision-Language Models

Alejandro Paredes La Torre

Duke University

Adversarial Attacks on VLMs VLM Robustness VLM Evaluation

Open paper AskPDF arXiv

We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.