VLM Interpretability

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model interpretability studies how models encode visual and textual information, combine it, and use it to produce answers, predictions, or actions. Researchers use saliency and attribution maps, probing and sparse autoencoders, and mechanistic methods such as activation patching and circuit analysis to locate relevant features, layers, and attention heads; causal interventions test whether those components actually affect behavior. Evaluations span image-text retrieval and classification, visual question answering, grounding, counting, spatial reasoning, and hallucination benchmarks such as POPE and CHAIR. A central tension is that information may be decodable or highlighted by attention without guiding the model’s answer, so explanation faithfulness and actual use of visual evidence remain open challenges.

Latest papers

CardsList

268 papers39 papers in the last four weeks, up 56% on the four weeks before. 0.2% of all new papers.

  1. Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models

    Mar 16, 2026Lexiang Xiong, Qi Li, Jingwen Ye +1Hallucination Detection in VLMsVLM Interpretability

    National University of Singapore · Monash University

  2. Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering

    Mar 14, 2026Lin Fan, Yafei Ou, Zhipeng Deng +8Medical VQAVisual CoT

    Southwest Jiaotong University · RIKEN · Hokkaido University +4

  3. Locating Demographic Bias at the Attention-Head Level in CLIP's Vision Encoder

    Mar 12, 2026Alaa Yasser, Kittipat Phunjanna, Marcos Escudero Viñolo +2Attention Head AnalysisVLM Interpretability

    Universidad Autónoma de Madrid, Spain · Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Bordeaux, France

  4. Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks

    Mar 12, 2026Mei Chee Leong, Ying Gu, Hui Li Tan +2VLM EvaluationVision-Language Models

    Institute for Infocomm Research (I2R) · Present address: Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore

  5. Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

    Mar 11, 2026Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud +9VLM InterpretabilityConcept-Based Explanations

    Columbia University, Department of Computer Science · Columbia University, Department of Art History & Archaeology · University of Texas at Austin +1

  6. BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment

    Mar 9, 2026Erdong Chen, Yuyang Ji, Jacob K. Greenberg +5Gait AnalysisMedical VLMs

    Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley

  7. Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models

    Mar 8, 2026Abin Shoby, Ta Duc Huy, Tuan Dung Nguyen +6Hallucination Detection in VLMsObject Hallucination in VLMs

    Australian Institute for Machine Learning, University of Adelaide · Hanoi University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence

  8. VisualScratchpad: Inference-time Visual Concepts Analysis in Vision Language Models

    Mar 7, 2026Hyesu Lim, Jinho Choi, Taekyung Kim +3VLM InterpretabilityVision-Language Alignment

    KAIST AI · NAVER AI Lab

  9. DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models

    Mar 6, 2026Walid Bousselham, Angie Boggust, Hendrik Strobelt +1VLM InterpretabilityToken-Level Credit Assignment

    Tuebingen AI Center · University of Tuebingen · MIT CSAIL +2

  10. Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving

    Mar 6, 2026Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu +3Vision-Language ModelsVLMs for Autonomous Driving

    Department of Electronic and Computer Engineering University of Limerick · Valeo Vision Systems

  11. Observing and Controlling Features in Vision-Language-Action Models

    Mar 5, 2026Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann +1VLM InterpretabilityVision-Language-Action Models

    Dept. of Aeronautics and Astronautics, Stanford University · Dept. of Mechanical Engineering, Stanford University · NVIDIA Research

  12. Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models

    Mar 3, 2026Haruto Yoshida, Keito Kudo, Yoichi Aoki +4Vision-Language ModelsLarge Vision-Language Models

    Tohoku University · Human Informatics Labs., NTT, Inc.

  13. Interpretable Debiasing of Vision-Language Models for Social Fairness

    Feb 27, 2026Na Min An, Yoonna Jang, Yusuke Hirota +3Cultural Bias in VLMsVLM Interpretability

    KAIST AI · University of Copenhagen · NVIDIA

  14. Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models

    Feb 26, 2026Jonathan Steinberg, Oren GalVision-Language ModelsVLM Interpretability

    Swarms & AI Lab (SAIL), University of Haifa

  15. Imagination Helps Visual Reasoning, But Not Yet in Latent Space

    Feb 26, 2026You Li, Chi Chen, Yanghao Li +4Causal Mediation AnalysisVisual Reasoning

    School of Computer Science and Technology, Beijing Jiao-tong University · Key Laboratory of Big Data & Artificial Intelli-gence in Transportation (Beijing Jiaotong University), Ministry of Education · Tsinghua University

  16. PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models

    Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMsVLM Evaluation

    SAIL Lab, University of New Haven, West Haven, CT, USA.

  17. On the Explainability of Vision-Language Models in Art History

    Feb 24, 2026Stefanie SchneiderVLM InterpretabilityExplainable Artificial Intelligence

    Marburg University, Marburg, Germany

  18. Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinking

    Feb 23, 2026Jingcheng Yang, Tianhu Xiong, Shengyi Qian +2VLM InterpretabilityVLM Reasoning

    University of Illinois Urbana-Champaign · Independent Researcher

  19. TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding

    Feb 23, 2026Fan Yang, Shurong Zheng, Hongyin Zhao +5Visual GroundingVLM Interpretability

    Foundation Model Research Center, Institute of Automation · Peng Cheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China +2

  20. Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer

    Feb 22, 2026Chenhang Cui, An Zhang, Yuxin Chen +5Cross-Modal Representation TransferVLM Reasoning

    National University of Singapore, Singapore · University of Science and Technology of China, Hefei, China · Nanyang Technological University, Singapore

  21. How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

    Feb 17, 2026Hongxuan Wu, Yukun Zhang, Xueqing ZhouVision-Language ModelsVLM Interpretability

    Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

  22. Visual Persuasion: What Influences Decisions of Vision-Language Models?

    Feb 17, 2026Manuel Cherep, Pranav M R, Pattie Maes +1VLM RobustnessVLM Interpretability

    Media Lab, Massachusetts Institute of Technology, Cambridge, USA · BITS Pilani, Goa, India · Department of Computer Science, Dartmouth College, Hanover, USA

  23. Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

    Feb 16, 2026Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza +3Cross-Modal LearningShortcut Learning

    Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.

  24. Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding

    Sep 28, 2025Amirmohammad Izadi, Hosein Hasani, Fatemeh Askari +4Multimodal GroundingCross-Modal Alignment

  25. Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models

    Sep 26, 2025Jiawei Liang, Jianjie Huang, Ruoyu Chen +4Feature AttributionMultimodal Large Language Models

    Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · Zhongguan-cun Academy, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +2

  26. Explainable embeddings with Distance Explainer

    May 21, 2025Christiaan Meijer, E. G. Patrick BosFeature AttributionPerturbation-Based Feature Attribution

    Netherlands eScience Center, Science Park 402, 1098 XH Amsterdam, Netherlands · Center for Information Technology, University of Groningen, P.O. Box 11044, 9700 CA Groningen, Netherlands

  27. Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection

    Apr 1, 2025Ruoyu Chen, Siyuan Liang, Jingzhi Li +4Feature AttributionVLM Interpretability

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · University of Chinese Academy of Sciences, Beijing 100049, China · College of Computing and Data Science, Nanyang Technological University, 639798, Singapore +4

  28. PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

    Feb 6, 2025Mennatullah SiamLarge Vision-Language ModelsVision-Language Grounding