VLM Robustness

VLM: Vision-Language Model

Vision-language model robustness asks whether a model’s answers stay grounded and dependable when images or prompts are corrupted, altered, unfamiliar, or in conflict. Researchers test these failures with adversarial attacks, corruption and out-of-distribution benchmarks, prompt and context variations, and measures of grounding, consistency, calibration, and safety. Common defenses include adversarial training, test-time adaptation, visual evidence selection, and interventions on model representations or attention. A central challenge is that clean accuracy and even stable answers can hide reliance on shortcuts, while improving robustness can cost accuracy, utility, or efficiency.

Latest papers

455 papers48 papers in the last four weeks, up 26% on the four weeks before. 0.3% of all new papers.

How Do Inpainting Artifacts Propagate to Language?
Feb 24, 2026Pratham Yashwante, Davit Abrahamyan, Shresth Grover +1VLM Robustness VLM Evaluation
UC San Diego
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada
VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense
Feb 23, 2026Nadav Kadvil, Malak Fares, Ayellet TalVLM Robustness Large Vision-Language Models
Technion – Israel Institute of Technology
PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
Feb 23, 2026Hefei Mei, Zirui Wang, Chang Xu +2Attention-Based Adversarial Attacks Adversarial Attacks on VLMs
City University of Hong Kong · The University of Sydney
OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models
Feb 20, 2026Ling Lin, Yang Bai, Heng Su +5VLM Evaluation VLM Robustness
School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore +1
When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
Feb 19, 2026Yu Fang, Yuchun Feng, Dong Jing +5VLM Robustness Counterfactual Evaluation of VLMs
University of North Carolina at Chapel Hill
Visual Memory Injection Attacks for Multi-Turn Conversations
Feb 17, 2026Christian Schlarmann, Matthias HeinBackdoor Attacks on VLA Models Backdoor Attacks on VLMs
Tübingen AI Center, University of Tübingen, Germany
FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failures
Feb 17, 2026Jiajun Xu, Jiageng Mao, Ang Qi +5VLM Robustness Adversarial Attacks on VLMs
University of Southern California · Toyota Research Institute
Visual Persuasion: What Influences Decisions of Vision-Language Models?
Feb 17, 2026Manuel Cherep, Pranav M R, Pattie Maes +1VLM Robustness VLM Interpretability
Media Lab, Massachusetts Institute of Technology, Cambridge, USA · BITS Pilani, Goa, India · Department of Computer Science, Dartmouth College, Hanover, USA
ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery
Feb 16, 2026Ayush Shrivastava, Kirtan Gangani, Laksh Jain +2VLM Evaluation Thermal Imaging
Indian Institute of Technology, Gandhinagar, India · Carnegie Mellon University, Pittsburgh, USA
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Feb 11, 2026Sethuraman T, Savya Khosla, Aditi Tiwari +12VLM Robustness Video-Language Model Evaluation
University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6
VideoSTF: Stress-Testing Output Repetition in Video Large Language Models
Feb 11, 2026Yuxin Cao, Wei Song, Shangzhi Xu +2VLM Evaluation VLM Robustness
National University of Singapore, Singapore · University of New South Wales, Australia · CSIRO’s Data61, Australia
Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Feb 8, 2026Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu +1VLM Robustness Adversarial Attacks on VLMs
Pennsylvania State University · Bangladesh University of Engineering and Technology
Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Semantic Misalignment in Vision-Language Models under Perceptual Degradation
Jan 13, 2026Guo Cheng, Huang LiVLM Evaluation VLM Robustness
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
Zoom-IQA: Image Quality Assessment with Reliable Region-Aware Reasoning
Jan 6, 2026Guoqiang Liang, Jianyi Wang, Zhonghua Wu +2VLM Robustness VLM Reasoning
National University of Singapore · S-Lab, Nanyang Technological University, Singapore · SenseTime Research
RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios
Dec 31, 2025Tianyi Zhao, Jiawen Xi, Linhui Xiao +4VLM Robustness Visual Grounding
Institute of Artificial Intelligence, State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, Shanghai, China
High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models
Dec 26, 2025Mengqi He, Xinyu Tian, Xin Shen +4VLM Robustness Adversarial Attacks on VLMs
The Australia National University Canberra, Australia · The University of Queensland Brisbane, Queensland, Australia · GE research Niskayuna, New York, USA
Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering
Dec 4, 2025Marco Pintore, Maura Pintor, Dimosthenis Karatzas +1Visual Question Answering VLM Robustness
University of Cagliari, Italy · CINI, Italy · Computer Vision Center, UAB, Spain
6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models
Dec 3, 2025Leon Mayer, Piotr Kalinowski, Caroline Ebersbach +6VLM Evaluation VLM Robustness
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. · Medical Faculty, Heidelberg University, Germany. · Faculty of Mathematics and Computer Science, Heidelberg University, Germany. +9
Understanding the Effects of Distractors on Reasoning Vision-Language Models
Nov 26, 2025Jiyun Bae, Hyunjong Ok, Sangwoo Mo +1VLM Reasoning Visual Question Answering
Pohang University of Science and Technology (POSTECH)
Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models
Nov 24, 2025Xiangjie Sui, Songyang Li, Hanwei Zhu +3VLM Evaluation VLM Robustness
City University of Macau, Macau SAR, China · Jiangxi University of Finance and Economics, Nanchang, China · Nanyang Technological University, Singapore
BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
Nov 24, 2025Juncheng Li, Yige Li, Hanxun Huang +5VLM Evaluation VLM Robustness
Fudan University · Singapore Management University · The University of Melbourne
Contrastive vision-language learning with paraphrasing and negation
Nov 20, 2025Kwun Ho Ngan, Saman Sadeghi Afgeh, Joe Townsend +1VLM Robustness VLM Reasoning
Fujitsu Research of Europe · United Kingdom · City St George’s, University of London
Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4VLM Robustness Medical Diagnosis
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection
Nov 10, 2025Yulin Chen, Zeyuan Wang, Tianyuan Yu +2VLM Robustness Adversarial Attacks on VLMs
Laboratory for Big Data and Decision, National University of Defense Technology, Changsha 410073, China
VLOD-TTA: Test-Time Adaptation of Vision-Language Object Detectors
Oct 1, 2025Atif Belal, Heitor R. Medeiros, Marco Pedersoli +1Open-Vocabulary Object Detection VLM Robustness
LIVIA, ILLS, Dept. of Systems Engineering, ETS Montréal, Canada
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
Sep 29, 2025Zhifang Zhang, Qiqi Tao, Jiaqi Lv +3VLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, Southeast University, China · School of Electrical Engineering and Computer Science, University of Queensland, Australia · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore +2
VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
Aug 11, 2025Mansi Phute, Ravikumar BalakrishnanVLM Robustness Adversarial Attacks on VLMs
Georgia Institute of Technology · HiddenLayer, Inc

Feb 24, 2026J/K move · Enter open · S save

How Do Inpainting Artifacts Propagate to Language?

Pratham Yashwante, Davit Abrahamyan, Shresth Grover, Sukruth Rao

UC San Diego

VLM Robustness VLM Evaluation Image Inpainting Image Captioning

Open paper AskPDF arXiv

We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.