VLM Interpretability

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

Towards Robustness against Typographic Attack with Training-free Concept Localization
Jul 2, 2026Bohan Liu, Wenqian Ye, Guangzhi Xiong +3VLM Robustness Adversarial Attacks on VLMs
University of Virginia, Charlottesville VA 22903, USA
Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning
Jun 30, 2026Chih-Ting Liao, Fei Shen, Xin Cao +1Visual Spatial Reasoning VLM Interpretability
University of New South Wales · National University of Singapore
H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning
Jun 29, 2026Eric Peh, Debaditya Roy, Basura FernandoReinforcement Learning Visual Reasoning
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India +1
Early Warning Signals for OpenVLA Failure under Visual Distribution Shift
Jun 29, 2026Dipesh Tharu Mahato, Rachel RenRobot Failure Detection Distribution Shift
New York University
How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning
Jun 29, 2026William Orwig, Roger E. BeatyCreativity Assessment LLM-as-a-Judge
Harvard University, Department of Psychology, Cambridge, MA, USA · Pennsylvania State University, Department of Psychology, University Park, PA, USA
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language Models Large Vision-Language Models
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
Jun 26, 2026Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff +2Knowledge Conflicts in Language Models VLM Interpretability
University of Tübingen · Harvard University · The University of Texas at Austin
Understanding How MLLMs Describe Artworks Using Token Activation Maps
Jun 26, 2026Nicola Fanelli, Pasquale De Marinis, Raffaele Scaringi +3VLM Interpretability Vision-Language Grounding
Department of Computer Science University of Bari Aldo Moro Bari, Italy
Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
Jun 25, 2026Chenyang Zhang, Anqi Dong, Guangming Zhu +4Concept Bottleneck Models VLM Interpretability
School of Computer Science and Technology, Xidian University, Xi’an, China · KTH Royal Institute of Technology, Stockholm, Sweden
Steering Vision-Language Models with Joint Sparse Autoencoders
Jun 24, 2026Huizhen Shu, Xuying Li, Hongxu Lin +2VLM Interpretability Sparse Autoencoders
yunshanai · HKUST(Guangzhou) · Zidongtaichu +1
UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving
Jun 23, 2026Xiaowei Gao, Pengxiang Li, Yitai Cheng +4VLMs for Autonomous Driving Autonomous Driving Perception
Department of Earth Science & Engineering, Imperial College London, London, SW7 2AZ, United Kingdom · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · SpaceTimeLab, Department of Civil, Environmental, and Geomatic Engineering, University College London, London, WC1E 6BT, United Kingdom +3
Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
Jun 23, 2026Jonas Klotz, Cassio F. Dantas, Pallavi Jain +2VLM Interpretability Sparse Autoencoders
The Berlin Institute for the Foundations of Learning and Data (BIFOLD) · Technische Universität Berlin, Germany · INRAE +2
Ill-Posed by Design: Probing Evidence Use in VLMs
Jun 23, 2026Boaz Meivar, Shaked Perek, Shani Shvartzman +2VLM Evaluation Counterfactual Evaluation of VLMs
1Tel Aviv University · 2IBM Research
Listening makes Vision Clear for VLMs
Jun 22, 2026Yiyang Chen, Yixin Tan, Binrui ShenVLM Evaluation Visual Attention
Beijing Normal University
Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
Jun 19, 2026Sergio Lanza, Jae Hee Lee, Stefan WermterVision-Language Models VLM Interpretability
Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany
The Hidden Evolution of Disguised Visual Context inside the VLM
Jun 18, 2026Wish Suharitdamrong, Tony Alex, Muhammad Awais +1Vision-Language Models Large Vision-Language Models
Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK
SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
Jun 18, 2026Jiayu Tang, Yuchen Zhou, Chao Gou3D Spatial Reasoning VLM Interpretability
School of Intelligent Systems Engineering, Sun Yat-sen University
Multimodal Concept Bottleneck Models
Jun 18, 2026Tongqing Shi, Ge Yan, Tuomas Oikarinen +1Concept Bottleneck Models VLM Interpretability
UC San Diego
PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution
Jun 16, 2026Zihan Gu, Junchi Zhang, Li Liu +2Feature Attribution VLM Interpretability
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai 200438, China +2
Attention Alignment Between Humans and Vision-Language Models
Jun 16, 2026Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld +4VLM Evaluation Visual Attention
Princeton Neuroscience Institute, Princeton University · Department of Psychology, Princeton University · Department of Computer Science, Princeton University +2
Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models
Jun 16, 2026Logan Mann, Yi Xia, Ajit Saravanan +6VLM Evaluation VLM Robustness
Algoverse AI Research · University of California, Santa Barbara · University of California, Berkeley +1
Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations
Jun 15, 2026Zhiqiang Zhou, Junliang Dai, Xu lingVLM Interpretability Multimodal CoT Reasoning
Hunan Chemical Industry Vocational and Technical College, Zhuzhou, Hunan 412000, China
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning
Jun 12, 2026Shilong Xiang, Zirui Zhang, Chengzhi MaoVision-Language Models Self-Supervised Learning
Rutgers University
Mirage Probes: How Vision Models Fake Visual Understanding
Jun 11, 2026Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5Vision-Language Models VLM Interpretability
Columbia University · Intuit · Technion +2
Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension
Jun 11, 2026Shenglai Zeng, Qirui Wang, Kai Guo +3VLM Robustness Visual Attention
Michigan State University · Xi’an Jiaotong University
Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability
Jun 8, 2026Emirhan Bilgiç, Baptiste Caramiaux, Zhi Yan +1VLM Interpretability Multimodal Model Interpretability
U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau · ISIR, Université Sorbonne, Pierre et Marie Curie, Paris · AMIAD, Pôle Recherche, Palaiseau
How Many Counterfactuals Does It Take? Probing VLM Hallucinations Through Circuits and Causal Effects
Jun 7, 2026Abhivansh Gupta, Simardeep Singh, Advika Sinha +2Hallucination Detection in VLMs VLM Robustness
Data Science Group, IIT Roor- kee, Roorkee, Uttarakhand, India.
When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
Jun 7, 2026Haoran Zhao, Soyeon Caren Han, Eduard HovyMultimodal Robustness Multimodal Large Language Models
1The University of Melbourne
Vision-Language Asymmetry in Bistable Image Captioning
Jun 6, 2026Arohan AgateImage Captioning Vision-Language Models
University of Washington.

Jul 2, 2026, cs.CVJ/K move · Enter open · S save

Towards Robustness against Typographic Attack with Training-free Concept Localization

Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He+2

University of Virginia, Charlottesville VA 22903, USA

VLM Robustness Adversarial Attacks on VLMs Large Vision-Language Models VLM Interpretability Mechanistic Interpretability

Open paper AskPDF arXiv

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, we isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanistic source of TA. We further show that simple interventions applied directly to the identified circuits, without any additional training, can substantially improve robustness against Typographic Attacks in object classification. These interventions, such as selective adjustment of attention weights, also outperform both supervised and training-free defense methods. Our experiments demonstrate that applying the proposed intervention to the vision encoders of several state-of-the-art LVLMs yields substantial gains in Visual Question Answering accuracy under Typographic Attack interference on RIO-Bench. These results confirm both the efficacy and the generalizability of our mechanistic approach. Code is released at https://github.com/Liu-524/SamplingTAR.

Momentum

Related topics

Latest papers 158