VLM Interpretability

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment
Oct 8, 2026Xiaoshan ZhouDisaster Damage Assessment VLM Interpretability
School of Project Management, Faculty of Engineering, The University of Sydney, Sydney, NSW 2006, Australia
Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Models
Oct 7, 2026Huiyao Zhang, Jin Bai, Zilong Su +6VLM Evaluation Faithfulness in VLMs
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Science and Technology of China
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
Sep 30, 2026Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao +3Vision-Language Models Medical VQA
Hunan University · University of Hong Kong
NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning
Sep 29, 2026Ruiyu Yan, Bowen Chen, Shaowen Wan +1VLM Interpretability VLM Reasoning
New York University · New Jersey Institute of Technology
CHOQOLATE: Organizing Concept Bottleneck Latent Spaces with Choquet Integrals
Sep 29, 2026Rémi Kazmierczak, Johanne Cohen, Marianne ClauselConcept Bottleneck Models VLM Interpretability
CRAN, Université de Lorraine, CNRS, Nancy, France · LISN, Université Paris-Saclay, CNRS, Orsay, France
Are In-Context Images Worth 10 Dimensions?
Sep 29, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Vision-Language Models Multimodal ICL
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, Paris, France · Pôle recherche de l’AMIAD, Palaiseau, France · Institut Universitaire de France, Paris, France
Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning
Sep 29, 2026Bowen Yuan, Danny Wang, Ruihong Qiu +2Large Vision-Language Models VLM Interpretability
The University of Queensland
Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model
Sep 29, 2026Van Bach Nguyen, Jörg Schlötterer, Christin SeiferVision-Language Models VLM Interpretability
Marburg University, Germany
Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving
Sep 29, 2026Katharina Winter, Stefan Englmeier, Fabian B. FlohrVLMs for Autonomous Driving VLM Interpretability
Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical Diagnosis VLM Interpretability
Tulane University · Oak Ridge National Laboratory
Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning
Sep 28, 2026Yingjin Song, Denis Paperno, Albert GattVisual Spatial Reasoning Relational Reasoning
Utrecht University Utrecht, The Netherlands
When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
Sep 28, 2026Yizhou Fang, Siyue Chen, Zimo Qi +3VLM Evaluation Vision-Language Models
University of Waterloo · Independent Researcher · Johns Hopkins University +3
AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors
Sep 28, 2026Mohamed Eltahir, Fardows Adam, Duaa M. Tahir +6Vision-Language Models VLM Interpretability
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
Sep 28, 2026Bohao Xing, Xin Liu, Kaishen Yuan +5Cross-Modal Learning Vision-Language Models
Lappeenranta-Lahti University of Technology LUT · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) +3
The Alignment Illusion in Multimodal Large Language Models
Sep 24, 2026Hong-Han Wang, Yuntao Wang, Hu DingRepresentational Similarity Analysis Multimodal Large Language Models
University of Science and Technology of China, Hefei, China
Where Hallucinations Live: A Cross-Architecture Circuit in VQ-Tokenized Vision-Language Models
Sep 24, 2026Shamanthak Hegde, Xiangrui Liu, Maitreya Patel +1Vision-Language Models Object Hallucination in VLMs
Arizona State University · Adobe
What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
Sep 21, 2026Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak +4Vision-Language Models VLM Interpretability
University of Technology Nuremberg. · ETH Zürich. · Inria, Ecole Normale Sup´erieure, CNRS, PSL Research University.
Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection
Sep 16, 2026Girish A. Koushik, Diptesh Kanojia, Helen TreharneVLM Evaluation VLM Robustness
Nature-Inspired Computing & Engineering Department of Computer Science University of Surrey Guildford, United Kingdom · Surrey Centre for Cyber Security Department of Computer Science University of Surrey Guildford, United Kingdom
Using OCR Heads to Verbalize Image Semantics
Sep 16, 2026Sheridan Feucht, Benno Krojer, Sarah Wang +3VLM Interpretability Mechanistic Interpretability
Northeastern University · Independent
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
Sep 16, 2026Sihao Ding, Santosh Vasa, Aditi Ramadwar +1VLM Evaluation VLM Robustness
Mercedes-Benz Research & Development North America
ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
Sep 15, 2026Jim Berend, Reduan Achtibat, Daniel Schäffer +4Transformer Interpretability Residual Learning
Fraunhofer Heinrich Hertz Institute · DSC ScaDS.AI, Leipzig University · ICT Cluster, Singapore Institute of Technology, Singapore +3
ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglement
Sep 14, 2026Yan Zhu, Yongbo Chen, Zhengming Ding +1Vision-Language Models Prototype Learning
Department of Computer Science Tulane University New Orleans, LA, USA
Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction
Sep 14, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1KV Caching KV-Cache Eviction
School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Sep 8, 2026Xiaofu Chen, Stella Frank, Yova KementchedjhievaVLM Interpretability
MBZUAI · Technical University of Denmark
I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
Sep 7, 2026Aydin Javadov, Daniel Schoess, Florian von WangenheimVLM Evaluation VLM Interpretability
ETH Zurich
Who Drives the Probability Game of VLMs? A Temporal Causal Drive Evaluation Framework
Sep 2, 2026Shuyao Xiao, Shengling Wang, Haoyu Niu +4VLM Evaluation VLM Interpretability
College of Artificial Intelligence, Beijing Normal University · Kuaishou Technology
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
Sep 1, 2026Xurui Song, Weishi Wang, Zhongqi Yue +5VLM Evaluation Visual Question Answering
SAP · Microsoft · College of Computing and Data Science, Nanyang Technological University, Singapore
Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
Aug 31, 2026Athulith Paraselli, Etha Tianze Hua, Ellie PavlickVision-Language Models Knowledge Conflicts in Language Models
Brown University
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan

Oct 8, 2026, cs.CVJ/K move · Enter open · S save

FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment

Xiaoshan Zhou

School of Project Management, Faculty of Engineering, The University of Sydney, Sydney, NSW 2006, Australia

Disaster Damage Assessment VLM Interpretability

Open paper AskPDF arXiv

Vision-language models (VLMs) show great potential for damage assessment after a disaster, but a recurring deficiency is that they are reluctant to declare a hazard; that is, recall is low even when overall accuracy appears adequate. This study examines that deficiency by using signal detection theory to decompose the decision behavior into perceptual capability and decision-criterion placement. We then propose a novel method for correcting the over-conservative decision policy, inspired by the finding that fear makes humans risk-averse, and ask whether an affective representation associated with fear can be causally manipulated to similarly alter a VLM's decision tendency. Using mechanistic interpretability, we localize a causally implicated affective circuit in the model and use activation steering to manipulate it while observing the effect on downstream prediction. The method is tested on a two-stage SeisMLLM pipeline built on Qwen2.5-VL-7B-Instruct, which flags only 27.0% of genuinely unsafe buildings on the SeisMLLM-1K test split and never issues a false Red, an SDT criterion of c = +1.354, despite adequate evidence quality (d' = 1.521). An affective direction is localized on emotion-rich natural scenes, causally validated by sparse-neuron knockout and distributed steering on held-out emotion data, and then injected into the building task. Fear-direction injection raises Red recall to 75.7% (p<0.001), and subtracting the same direction suppresses Red predictions entirely, whereas norm-matched random and matched happiness directions show no significant effect. The mechanism is a shift in criterion (c=-1.515) while discrimination is not improved (d'=-0.493). These results show that VLM decisions can be adjusted at inference time without retraining and demonstrate how mechanistic interpretability can be used to diagnose and control VLM decision behaviors in engineering applications.

Momentum

Related topics

Latest papers 158