VLM Robustness

VLM: Vision-Language Model

Vision-language model robustness asks whether a model’s answers stay grounded and dependable when images or prompts are corrupted, altered, unfamiliar, or in conflict. Researchers test these failures with adversarial attacks, corruption and out-of-distribution benchmarks, prompt and context variations, and measures of grounding, consistency, calibration, and safety. Common defenses include adversarial training, test-time adaptation, visual evidence selection, and interventions on model representations or attention. A central challenge is that clean accuracy and even stable answers can hide reliance on shortcuts, while improving robustness can cost accuracy, utility, or efficiency.

Latest papers

455 papers48 papers in the last four weeks, up 26% on the four weeks before. 0.3% of all new papers.

Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
Oct 29, 2026Yubo Cui, Xianchao Guan, Zheng ZhangVLM Robustness Adversarial Attacks on VLMs
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China
VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts
Oct 25, 2026Isaac, LiVisual Token Compression VLM Evaluation
Rucheng · University of Pittsburgh
Execution Logs Sway VLM Judges of Robot Progress
Oct 25, 2026Weiyan Zhao, Sunny YangVLM Robustness Vision-Language Model-as-a-Judge
NLPQuant AI Inc. · University of Illinois Urbana-Champaign
Search-Based Testing of Modality Sensitivity in Vision Language Models
Oct 23, 2026Oliver Weißl, Eduardo Breitembach, Andrea StoccoVLM Evaluation VLM Robustness
Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany
MAGIC: Margin-gated Contrastive Debiasing with Evidence Restoration for VLM Test-time Robustness Scaling
Oct 23, 2026Shunshun Gu, Enze Zhang, Ming Pei +2VLM Robustness Test-Time Scaling for VLMs
University of Wisconsin–Madison · University of North Carolina at Chapel Hill
Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
Oct 22, 2026Qiuyu Kong, Zanxi Ruan, Marco Cristani +1Vision-Language Alignment Image-Text Retrieval
Sapienza University of Rome · University of Verona · Reykjavik University +1
A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era
Oct 21, 2026Eric Liang, Alan Luo, Kaiwen YuanVLM Robustness Adversarial Attacks on VLMs
Saratoga High School · Google · University of Illinois Urbana-Champaign
Can Quantum Layers offer Performance and Security Advantages in Vision-Language Models?
Oct 20, 2026Md Khalid Syfullah, Alvi Ataur KhalilQuantum Machine Learning VLM Robustness
Computer Science, Southern Illinois University Carbondale, USA
DISTRACT: Degradation-Indexed Study of Textual Reference Across Competing Targets
Oct 20, 2026Dan Zlotnikov, Ohad Ben-ShaharMultimodal Visual Object Tracking VLM Robustness
Ben-Gurion University of the Negev, Israel
Beyond the Evidence Boundary: Behavioral Mirage Under Progressive Degradation in Medical VLMs
Oct 20, 2026Edric Castel C. Hao, Rez Samantha Z. FlorescaMedical VLMs VLM Robustness
Ateneo de Manila Senior High School
Robustness of Internal Signals for Hallucination Detection in Vision-Language Models
Oct 19, 2026Allison Chen, William Yang, Salma Abdel Magid +3Hallucination Detection in VLMs VLM Robustness
Department of Computer Science, Princeton University, Princeton, NJ 08544
TRACE: Training-aware Robustness and Challenge Evaluation for Text-Vision Embedding Models
Oct 19, 2026Pume Tuchinda, Tinnakit Udsa, Peerat LimkonchotiwatVLM Evaluation VLM Robustness
AI Singapore · School of Information Science and Technology VISTEC
ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection
Oct 18, 2026Haotian Gan, Yudong Li, Xintian Li +2Medical VLMs VLM Robustness
Tsinghua University · The University of Hong Kong · China Agricultural University
Investigating Length Bias and Robustness in LVLMs for Multiple-Choice Question Answering
Oct 17, 2026Md. Atabuzzaman, Hani Alomari, Chris ThomasVLM Robustness Multiple-Choice Question Answering
Department of Computer Science, Virginia Tech
How Murky Before a VLM Knows It Cannot See? Accuracy, Confidence and Abstention on a Physical Visibility Axis
Oct 16, 2026Mochamad Rizal Fauzan, Irgi SuryaVLM Evaluation VLM Robustness
National Taipei University of Technology, Taipei, Taiwan
Variational and Decomposed method for Mitigating Bimodal Bias in Robust Visual Question Answering
Oct 16, 2026Yihan Zhao, Yuncheng JiangVLM Robustness Multimodal QA
School of Computer Science, South China Normal University
When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition
Oct 15, 2026Niaz Mohaiman Abtahi, Evan Maksud, Reyana ShahidCultural Bias in VLMs VLM Evaluation
Advanced AI Lab Limited · United International University
FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models
Oct 15, 2026Jiaming Zhang, Zherui Li, Siqi Guo +3VLM Robustness Adversarial Attacks on VLMs
College of Computing and Data Science Nanyang Technological University, Singapore
VisualTrust: Video Reasoning under On-Screen Claim Framing
Oct 15, 2026Huachao Zhu, Junjielong Xu, Sicheng Li +5Video-Language Model Evaluation VLM Robustness
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Tencent
Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
Oct 14, 2026Tianshi Wang, Jingsong Wang, Fengling Li +3VLM Robustness
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China +1
DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
Oct 13, 2026Chengtao Jian, Kai YangData Poisoning Attacks Adversarial Attacks on VLMs
School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China
Geometric Parrots: Rule Absence vs. Rule Fragility in Spatial and Physical Reasoning for Vision-Language Models
Oct 12, 2026Bilal Jawaid, Hamza Hanif, Muhammad Maaz Uddin +8Sycophancy in VLMs VLM Reasoning
Independent Researcher · Simon Fraser University, Canada · Salim Habib University, Karachi, Pakistan +7
Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding
Oct 11, 2026Qingzhe Han, Fan Dang, Shengqi Chen +4Adversarial Attacks on VLMs VLM Robustness
Beijing Jiaotong University
[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning
Oct 10, 2026Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed +1VLM Interpretability Saliency Map Evaluation
University of Amsterdam
Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders
Oct 8, 2026Tao Yang, Jianying ZhouContrastive Learning for VLMs Adversarial Attacks on VLMs
Singapore University of Technology and Design
TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning
Oct 7, 2026Yigit Yilmaz, Elena Petrova, Mehmet Kaya +2Multimodal ICL In-Context Example Selection
Bandirma Onyedi Eylul University
Detecting Adversarial Images through Response Profiles of Vision-Language Models
Oct 7, 2026Arash Vashagh, Roozbeh Razavi-FarAdversarial Example Detection VLM Robustness
Trustworthy and Secure AI (TSAI) Lab, Faculty of Computer Science, University of New Brunswick, Canada
Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions
Oct 7, 2026Bojun Yang, Haochen Zhou, Zhifang Zhang +3VLM Robustness Large Vision-Language Models
Southeast University · The University of Queensland · Zhejiang University
Understanding and Mitigating Token-Pruning-Induced Vulnerabilities in VLMs
Oct 7, 2026Shuailong Wang, Xinyu Lyu, Shengming Yuan +3Visual Token Pruning VLM Robustness
University of Electronic Science and Technology of China, Chengdu, China · Southwestern University of Finance and Economics, Chengdu, China · Tongji University, Shanghai, China +1
It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank
Oct 7, 2026Samuel Tetteh, Cody FlemingCultural Safety Evaluation of VLMs VLM Evaluation
Iowa State University Ames, Iowa, USA

Oct 29, 2026J/K move · Enter open · S save

Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui, Xianchao Guan, Zheng Zhang

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM Robustness Adversarial Attacks on VLMs Text-Image Alignment

Open paper AskPDF arXiv

Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features provides broad coverage by constraining all directions equally, but lacks semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. To address these issues, we propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. Specifically, TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.