VLM Interpretability

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models
Mar 23, 2026Kelly Cui, Nikhil Prakash, Shoval Messica +4VLM Interpretability Vision-Language Grounding
MIT CSAIL · Northeastern University · Sony Playstation
CREG: Compass Relational Evidence Graph for Characterizing Directional Structure in VLM Spatial-Reasoning Attribution
Mar 20, 2026Kaizhen Tan, Yang Feng, Heqing DuVLM Evaluation Gradient-Based Attribution
Carnegie Mellon University Pittsburgh, PA, USA · Columbia University New York, NY, USA
Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks
Mar 12, 2026Mei Chee Leong, Ying Gu, Hui Li Tan +2VLM Evaluation Vision-Language Models
Institute for Infocomm Research (I2R) · Present address: Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
Observing and Controlling Features in Vision-Language-Action Models
Mar 5, 2026Hugo Buurmeijer, Carmen Amo Alonso, Aiden Swann +1VLM Interpretability Vision-Language-Action Models
Dept. of Aeronautics and Astronautics, Stanford University · Dept. of Mechanical Engineering, Stanford University · NVIDIA Research
Imagination Helps Visual Reasoning, But Not Yet in Latent Space
Feb 26, 2026You Li, Chi Chen, Yanghao Li +4Causal Mediation Analysis Visual Reasoning
School of Computer Science and Technology, Beijing Jiao-tong University · Key Laboratory of Big Data & Artificial Intelli-gence in Transportation (Beijing Jiaotong University), Ministry of Education · Tsinghua University
Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
Sep 26, 2025Jiawei Liang, Jianjie Huang, Ruoyu Chen +4Feature Attribution Multimodal Large Language Models
Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · Zhongguan-cun Academy, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +2
Explainable embeddings with Distance Explainer
May 21, 2025Christiaan Meijer, E. G. Patrick BosFeature Attribution Perturbation-Based Feature Attribution
Netherlands eScience Center, Science Park 402, 1098 XH Amsterdam, Netherlands · Center for Information Technology, University of Groningen, P.O. Box 11044, 9700 CA Groningen, Netherlands
Debunking Grad-ECLIP: A Comprehensive Study on Its Incorrectness and Fundamental Principles for Model Interpretation
Date pendingYongjin Cui, Xiaohui FanGradient-Based Attribution Transformer Interpretability
Zhejiang University

Mar 23, 2026, cs.CVJ/K move · Enter open · S save

The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models

Kelly Cui, Nikhil Prakash, Shoval Messica, Ayush Raina+3

MIT CSAIL · Northeastern University · Sony Playstation

VLM Interpretability Vision-Language Grounding Spatial Reasoning

Open paper AskPDF arXiv

Many multimodal tasks, such as image captioning and visual question answering, require vision-language models (VLMs) to bind objects with their properties and spatial relations. Yet it remains unclear where and how such associations are computed within VLMs. In this work, we show that VLMs rely on two concurrent mechanisms to represent spatial variable binding. In the language model backbone, intermediate layers represent content-independent spatial relations on top of visual tokens corresponding to objects. However, this mechanism plays only a secondary role in shaping model predictions. Instead, the dominant source of spatial information originates in the vision encoder, whose representations encode the layout of objects and are directly exploited by the language model backbone. Notably, this spatial signal is distributed globally across visual tokens, extending beyond object regions into surrounding background areas. We validate the generalization of our findings to complex natural images from the COCO dataset, where globally amplifying the vision-derived spatial representations across all image tokens corrects spatial variable binding failures across models of various sizes. Together, our results clarify how spatial variable binding is computed within VLMs and highlight the central role of vision encoders in enabling it.

Momentum

Related topics

Latest papers 158