VLM Robustness

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model robustness asks whether a model’s answers stay grounded and dependable when images or prompts are corrupted, altered, unfamiliar, or in conflict. Researchers test these failures with adversarial attacks, corruption and out-of-distribution benchmarks, prompt and context variations, and measures of grounding, consistency, calibration, and safety. Common defenses include adversarial training, test-time adaptation, visual evidence selection, and interventions on model representations or attention. A central challenge is that clean accuracy and even stable answers can hide reliance on shortcuts, while improving robustness can cost accuracy, utility, or efficiency.

Latest papers

CardsList

455 papers48 papers in the last four weeks, up 26% on the four weeks before. 0.3% of all new papers.

  1. Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

    Mar 30, 2026Yanjie Zhang, Yafei Li, Rui Sheng +5VLM RobustnessChart QA

    HKUST, Hong Kong SAR, China · HKUST(GZ), Guangzhou, China

  2. XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

    Mar 30, 2026Chengyin Hu, Jiaju Han, Xuemeng Sun +6Adversarial Attacks on VLMsVLM Robustness

    China University of Petroleum-Beijing at Karamay · Shenzhen Research Institute of Big Data · Shanghai Jiao Tong University +2

  3. When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

    Mar 29, 2026Chengyin Hu, Xuemeng Sun, Jiaju Han +5Adversarial Attacks on VLMsAdversarial Attacks

  4. Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

    Mar 29, 2026Baoheng Zhang, Jiahui Liu, Gui Zhao +7Vision-Language ModelsMultimodal Large Language Models

    The University of Hong Kong

  5. Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method

    Mar 29, 2026Xiaoran Xu, Xiaoshan Yang, Jiangang Yang +3Open-Vocabulary Object DetectionVLM Robustness

    the MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · the School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · the Institute of Microelectronics, University of Chinese Academy of Sciences, Beijing, China

  6. Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

    Mar 28, 2026Jinhu Fu, Yihang Lou, Qingyi Si +3VLM SafetyVision-Language Models

    Beijing University of Posts and Telecommunications · Huawei Technologies Ltd. · Peking University +1

  7. The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models

    Mar 28, 2026Shivang Chopra, Shaunak Halbe, Chengyue Huang +2PEFT of VLMsVLM Robustness

    Georgia Institute of Technology

  8. A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models

    Mar 27, 2026Mujtaba Hussain Mirza, Antonio D'Orazio, Odelia Melamed +1Adversarial PurificationAdversarial Robustness

    OmnAI Lab, Computer Science Department, Sapienza University of Rome, Italy · Weizmann Institute of Science, Israel

  9. ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

    Mar 27, 2026Mriganka Nath, Anurag Das, Jiahao Xie +1VLM Hallucination MitigationTest-Time Adaptation of VLMs

    Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

  10. Learning to Trim: End-to-End Causal Graph Pruning with Dynamic Anatomical Feature Banks for Medical VQA

    Mar 27, 2026Zibo Xu, Qiang Li, Weizhi Nie +1Medical VQASpurious Correlation Robustness

    School of Microelectronics, Tianjin University, Tianjin 300072, China · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China

  11. Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

    Mar 25, 2026Weixin Chen, Antonio Vergari, Han ZhaoVLM RobustnessVisual Reasoning

    University of Illinois Urbana-Champaign · University of Edinburgh

  12. MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage

    Mar 24, 2026Ufaq Khan, Umair Nawaz, L D M S S Teja +5VLM EvaluationMedical VLMs

    Mohamed bin Zayed University of Artificial Intelligence, UAE · National Institute of Technology, Silchar · Birmingham City University, UK

  13. Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

    Mar 24, 2026Mehmet Kaan ErolCompositional Reasoning in VLMsVLM Evaluation

    Marmara University, Institute of Pure and Applied Sciences

  14. Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models

    Mar 23, 2026Xingyu Zhu, Beier Zhu, Shuo Wang +4VLM RobustnessActivation Steering

    MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore · Nanyang Technological University

  15. Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects

    Mar 22, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6VLM EvaluationMultilingual VLM Evaluation

  16. CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

    Mar 22, 2026Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick +1Spatial Reasoning BenchmarksMulti-View Consistency

    Department of Computer Science and Software Engineering, Auburn University Auburn, AL 36849, USA

  17. When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound

    Mar 22, 2026Yasamin Medghalchi, Milad Yazdani, Amirhossein Dabiriaghdam +7VLM RobustnessAdversarial Attacks on VLMs

    University of British Columbia, Vancouver, Canada

  18. StageCraft: Execution Aware Mitigation of Distractor and Obstruction Failures in VLA Models

    Mar 21, 2026Kartikay Milind Pangaonkar, Prabin Rath, Omkar Patil +1VLM RobustnessRobotic Manipulation

    Arizona State University

  19. When Negation Is a Geometry Problem in Vision-Language Models

    Mar 20, 2026Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1VLM ReasoningVLM Evaluation

    ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher

  20. Geometric Risk Control for Vision-Language Model OCR

    Mar 20, 2026Weile Gong, Zijian Lu, Mingcai Chen +3Selective Prediction for VLMsVLM Hallucination Mitigation

    Nanjing University of Posts and Telecommunications, Nanjing, China

  21. Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

    Mar 19, 2026Shang-Jui Ray Kuo, Paola Cascante-BonillaVision-Language ModelsState Space Models

    Stony Brook University

  22. Tinted Frames: Question Framing Blinds Vision-Language Models

    Mar 19, 2026Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2Vision-Language ModelsVLM Robustness

    University of British Columbia · Vector Institute for AI · University of California, Berkeley

  23. SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

    Mar 19, 2026Carlos Hinojosa, Clemens Grange, Bernard GhanemCultural Safety Evaluation of VLMsVision-Language Models

    King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Technical University of Munich (TUM), Germany

  24. Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness

    Mar 19, 2026Lu Yu, Haiyang Zhang, Changsheng XuAttention-Based Adversarial AttacksContrastive Learning for VLMs

    School of Computer Science and Engineering, Tianjin University of Technology, 300384, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China

  25. CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models

    Mar 19, 2026Xiang Chen, Fangfang Yang, Chunlei Meng +6Medical Image Quality AssessmentVLM Robustness

  26. T-QPM: Enabling Temporal Out-Of-Distribution Detection and Domain Generalization for Vision-Language Models in Open-World

    Mar 19, 2026Aditi Naiknaware, Salimeh SekehVLM RobustnessDomain Generalization

    San Diego State University

  27. WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models

    Mar 18, 2026Wanjun Du, Zifeng Yuan, Tingting Chen +3VLM RobustnessVLM Reasoning

    Beijing Jiaotong University · National University of Singapore · Monash University

  28. Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

    Mar 18, 2026Zhihua Wei, Qiang Li, Jian Ruan +6VLM SafetyAdversarial Attacks on VLMs

    School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory

  29. HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

    Mar 17, 2026Md Jahidul IslamVLM AdaptationAdapter Tuning

    Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

  30. Visual Prompt Discovery via Semantic Exploration

    Mar 17, 2026Jaechang Kim, Yotaro Shimose, Zhao Wang +3VLM Robustness

    Sony Group Corporation · Pohang University of Science and Technology · National Yang Ming Chiao Tung University