VLM Robustness

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model robustness asks whether a model’s answers stay grounded and dependable when images or prompts are corrupted, altered, unfamiliar, or in conflict. Researchers test these failures with adversarial attacks, corruption and out-of-distribution benchmarks, prompt and context variations, and measures of grounding, consistency, calibration, and safety. Common defenses include adversarial training, test-time adaptation, visual evidence selection, and interventions on model representations or attention. A central challenge is that clean accuracy and even stable answers can hide reliance on shortcuts, while improving robustness can cost accuracy, utility, or efficiency.

Latest papers

CardsList

455 papers48 papers in the last four weeks, up 26% on the four weeks before. 0.3% of all new papers.

  1. Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

    Apr 15, 2026Arya Shah, Vaibhav Tripathi, Mayank Singh +1Sycophancy in VLMsVLM Robustness

    Indian Institute of Technology Gandhinagar Gandhinagar, India · Asian Institute of Technology Bangkok, Thailand

  2. Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

    Apr 15, 2026Jiachen Qian, Zhaolu KangVLM RobustnessAdversarial Attacks on VLMs

    City University of Hong Kong · Peking University

  3. Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

    Apr 14, 2026Yingying Zhao, Chengyin Hu, Qike Zhang +7Adversarial Attacks on VLMsVLM Robustness

  4. Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning

    Apr 14, 2026Jungwon Choi, Eunwoo KimVLM RobustnessVLM Adaptation

    School of Computer Science and Engineering, Chung-Ang University

  5. Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models

    Apr 14, 2026Ravikumar Balakrishnan, Sanket Mendapara, Ankit GargAdversarial Attacks on VLMsVLM Robustness

    Cisco Systems

  6. Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

    Apr 13, 2026Songlong Xing, Weijie Wang, Zhengyu Zhao +3VLM Robustness

    University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China +1

  7. Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference

    Apr 13, 2026Imanol Miranda, Ander Salaberria, Eneko Agirre +1Compositional Reasoning in VLMsVision-Language Models

    HiTZ Center – Ixa, University of the Basque Country (UPV/EHU)

  8. Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

    Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMsVLM Reasoning

    Brown University · University of Washington · University of Tübingen +1

  9. Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

    Apr 10, 2026Zibin Geng, Xuefeng Jiang, Jia Li +6VLM RobustnessVisual Prompt Learning

    Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · PCALab, VCIP, College of Computer Science, Nankai University, Tianjin, China +1

  10. Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection

    Apr 10, 2026Zedian Shao, Hongbin Liu, Yuepeng Hu +1Privacy Protection in VLMsAdversarial Attacks on VLMs

    Georgia Institute of Technology · Duke University

  11. VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts

    Apr 7, 2026Peigui Qi, Kunsheng Tang, Yanpu Yu +7VLM SafetyVLM Robustness

    University of Science and Technology of China · Ant Group · University of Washington

  12. Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization

    Apr 7, 2026Igor Maljkovic, Maria Rosaria Briglia, Iacopo Masi +2VLM RobustnessAdversarial Attacks on VLMs

    University of Genoa, Italy · Sapienza University of Rome, Italy · University of Cagliari, Italy

  13. Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality

    Apr 7, 2026Yanming Xiu, Zhengyuan Jiang, Neil Zhenqiang Gong +1Augmented RealityVLM Robustness

    Duke University

  14. Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

    Apr 6, 2026Chaoyou Fu, Haozhi Yuan, Yuhao Dong +16Video UnderstandingVideo-Language Model Evaluation

  15. Less Detail, Better Answers: Degradation-Driven Prompting for VQA

    Apr 6, 2026Haoxuan Han, Weijie Wang, Zeyu Zhang +2Visual Question AnsweringVLM Reasoning

    State Key Lab of CAD&CG, Zhejiang University

  16. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    Apr 6, 2026Jiajun Zhai, Hao Shi, Shangwei Guo +2Event-Based VisionVLM Robustness

    Zhejiang University · Ant Group · Hunan University

  17. SALLIE: Generation-Free Hidden-State Detection of Jailbreaks and Prompt Injections Across Text and Vision

    Apr 6, 2026Guy Azov, Ofer Rivlin, Guy ShtarPrompt Injection DetectionLLM Security

    Intuit

  18. Discovering Failure Modes in Vision-Language Models using RL

    Apr 6, 2026Kanishk Jain, Qian Yang, Shravan Nayak +3VLM EvaluationVLM Robustness

    Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2

  19. Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

    Apr 6, 2026Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2VLM RobustnessVLM Evaluation

    Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan

  20. When Does Multimodal AI Help? Diagnostic Complementarity of Vision-Language Models and CNNs for Spectrum Management in Satellite-Terrestrial Networks

    Apr 4, 2026Yuanhang LiVLM EvaluationWireless Communications

  21. Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

    Apr 3, 2026Chengyin Hu, Yuxian Dong, Yikun Guo +8Adversarial Attacks on VLMsVLM Robustness

    China University of Petroleum-Beijing at Karamay, No. 355 Anding Road, Karamay 834000, Xinjiang, China. · the Defense Innovation Institute, Chinese Academy of Military Science, Beijing, China. · Shenzhen University, Shenzhen, China. +1

  22. Token Warping Helps MLLMs Look from Nearby Viewpoints

    Apr 3, 2026Phillip Y. Lee, Chanho Park, Mingue Park +3Multimodal Large Language ModelsVLM Reasoning

    KAIST

  23. Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

    Apr 2, 2026Jason Qiu, Zachary Meurer, Xavier Thomas +1VLM ReasoningVLM Robustness

    Boston University

  24. Sparse Spectral LoRA: Routed Experts for Medical VLMs

    Apr 1, 2026Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat +2Medical VLMsVision-Language Models

    Concordia University, Montreal, Canada

  25. PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

    Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM RobustnessVision-Language Models

    Department of Computer Science City University of Hong Kong

  26. Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

    Apr 1, 2026Zhuchenyang Liu, Yao Zhang, Yu XiaoVision-Language AlignmentVLM Robustness

    Aalto University, Espoo, Finland

  27. TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

    Mar 31, 2026Qiucheng Yu, Ruijie Xu, Mingang Chen +2VLM EvaluationVLM Robustness

    City University of Hong Kong · East China Normal University · Shanghai Development Center of Computer Software Technology +1

  28. Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

    Mar 31, 2026Xuesong Wang, Harry WangVLM RobustnessReasoning for VLMs

    Wayne State University · University of Michigan

  29. AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

    Mar 31, 2026Yubo Cui, Xianchao Guan, Zijun Xiong +1VLM RobustnessVision-Language Alignment

    Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

  30. Is the Modality Gap a Bug or a Feature? A Robustness Perspective

    Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMsMultimodal Embedding

    Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel