VLM Interpretability

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies
May 17, 2026Xinchen Jin, Aditya Chatterjee, Pranav Kumar +1VLM Interpretability Sparse Autoencoders
Department of Computer Science, Purdue University West Lafayette, IN 47907
Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations
May 15, 2026Narges Babadi, Hadis KarimipourVLM Robustness Adversarial Attacks on VLMs
Department of Electrical and Software Engineering, University of Calgary, Calgary, Canada.
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
May 15, 2026Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1Distribution Shift Robustness VLM Robustness
University of Tübingen ∗ · University of Tübingen
Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
May 15, 2026Chufan Shi, Cheng Yang, Yaokang Wu +4VLM Evaluation VLM Robustness
University of Southern California · University of California San Diego · Carnegie Mellon Univer +1
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
May 14, 2026Jakub Grzywaczewski, Dawid Płudowski, Przemysław BiecekRepresentation Geometry Representation Learning
Warsaw University of Technology · Centre for Credible Artificial Intelligence · University of Warsaw
Dual-Pathway Circuits of Object Hallucination in Vision-Language Models
May 13, 2026Jiaxin Liu, Ding Zhong, Yue Wang +6Vision-Language Models Object Hallucination in VLMs
UIUC · UMich · Stanford +5
Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
May 13, 2026Ziqi Wen, Parsa Madinei, Miguel P. EcksteinVLM Evaluation Vision-Language Models
Department of Computer Science, University of California, Santa Barbara · Department of Psychological and Brain Sciences, University of California, Santa Barbara
Revealing Interpretable Failure Modes of VLMs
May 12, 2026Isha Chaudhary, Vedaant V Jain, Kavya Sachdeva +2VLM Robustness Vision-Language Models
UIUC · Kumo AI · IIT Delhi
Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMs
May 11, 2026M. ShalankinVLM Evaluation Vision-Language Models
Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination
May 11, 2026Yangneng Chen, Junlin Li, Weijun Yao +4Hallucination in Language Models VLM Hallucination
1Harbin Institute of Technology (Shenzhen), China · 2Huawei Technologies Co., Ltd. · 3The Hong Kong Polytechnic University +1
Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
May 9, 2026Gabriele Lombardo, Luigi Maiorana, Liliana Lo Presti +1Neural Representation Geometry VLM Interpretability
University of Palermo Palermo, Italy
Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models
May 8, 2026Haoming Wang, Wei GaoVisual Spatial Reasoning VLM Interpretability
Department of Electrical and Computer Engineering University of Pittsburgh Pittsburgh, PA 15261
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1Vision-Language Models Object Hallucination in VLMs
Indian Institute of Technology Dhanbad · National University of Singapore
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Causal Probing for Internal Visual Representations in Multimodal Large Language Models
May 7, 2026Zehao Deng, Tianjie Ju, Zheng Wu +5Multimodal Large Language Models VLM Interpretability
School of Computer Science, Shanghai Jiao Tong University · Ant Group
Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning
May 6, 2026Yating Wang, Yaqi Zhao, Yongshun Gong +2Prompt Learning VLM Adaptation
School of Software, Shandong University, Jinan, 250101, China
Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
May 5, 2026Logan Mann, Ajit Saravanan, Ishan Dave +4VLM Evaluation VLM Interpretability
UC Santa Barbara · UC Berkeley · NVIDIA +2
Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers
May 5, 2026Mathis Immertreu, Fitim Abdullahu, Thomas Kinfe +3Transformer Interpretability Vision-Language Models
Cognitive Computational Neuroscience Group, Patter Recognition Lab, University Erlangen-Nürnberg, Germany · IDS Institut für Data Science, ZHAW School of Engineering, Winterthur, Switzerland · Mannheim Center for Neuromodulation and Neuroprosthetics, University Hospital Mannheim, Heidelberg University, Germany +3
Embodied Interpretability: Linking Causal Understanding to Generalization in Vision-Language-Action Models
May 1, 2026Hanxin Zhang, Mingshuo Xu, Abdulqader Dhafer +3VLM Robustness VLM Interpretability
1DANiLab, University of Leicester, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · School of Metallurgy and Materials, University of Birmingham, UK
InterPartAbility: Phrase-Region Grounding for Interpretable Text-to-Image Person Re-Identification
Apr 29, 2026Shakeeb Murtaza, Aryan Shukla, Rajarshi Bhattacharya +2VLM Interpretability Vision-Language Grounding
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada · Genetec Inc.
FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment
Apr 26, 2026Sophie Chiang, Tom Brennan, Fethiye Irmak Dogan +2Depression Detection Medical Diagnosis
Department of Computer Science & Technology, University of Cambridge. · Harvard University.
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
Apr 21, 2026Kun Wang, Cheng Qian, Miao Yu +6LLM Backdoor Attacks Multimodal Large Language Models
3Nanyang Technological University · 2Beijing University of Aeronautics and Astronautics · University of Science and Technology of China +2
Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Apr 20, 2026Kecheng Zhang, Zongxin Yang, Mingfei Han +6Streaming Video Understanding Efficient VLM Inference
University of Science and Technology of China · Harvard University · Department of CV, MBZUAI +3
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
Apr 20, 2026Qidong Wang, Junjie Hu, Ming JiangMulti-Task Learning Causal Attribution
Tongji University · University of Wisconsin-Madison
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Apr 19, 2026Wei HeVLM Evaluation Figurative Language Understanding
IDSAI, University of Exeter Exeter, UK
Real-Time Visual Attribution Streaming in Thinking Model
Apr 17, 2026Seil Kang, Woojung Han, Junhyeok Kim +3VLM Interpretability Multimodal Reasoning
YYonsei University. · AAmazon.
How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study
Apr 16, 2026Zhen Yang, Ping Jian, Zhongbin Guo +5Visual Spatial Reasoning Large Vision-Language Models
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, China
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Apr 16, 2026Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1CoT Faithfulness Vision-Language Models
University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science
Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs
Apr 8, 2026Mengdan Zhu, Senhao Cheng, Liang ZhaoVLM Interpretability Latent Visual Reasoning
Emory University · University of Michigan, Ann Arbor
When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany

May 17, 2026, cs.ROJ/K move · Enter open · S save

Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies

Xinchen Jin, Aditya Chatterjee, Pranav Kumar, Rohan Paleja

Department of Computer Science, Purdue University West Lafayette, IN 47907

VLM Interpretability Sparse Autoencoders Mechanistic Interpretability Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) policies translate language and visual inputs into robot actions, where their hidden representations directly shape closed-loop behavior. However, mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs: outputs are robot actions rather than human-readable tokens, and interventions can only be tested via expensive closed-loop rollouts. We propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts. End-effector keyframes are clustered within each task using visual, state, and temporal cues, linking SAE features to behaviorally salient events and, via optional VLM annotations, to semantic context. To our knowledge, our pipeline is among the first to ground SAE-based VLA analysis in closed-loop behavioral events. Across two simulation architectures and a real-robot study, event-grounded ranking yields the strongest causal effects on OpenVLA and transfers to the continuous action chunks of

π_{0.5}

. SAE is a sparse but imperfect intervention basis: usability varies with architecture and intervention site, and aggressive intervention reveals safety and interpretability limits. Overall, event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability, motivating future work on SAE features beyond action-aligned coordinates, finer-grained closed-loop evaluation, and safe interventions for high-stakes VLA deployments. Code is available at https://github.com/xc-j/Event-SAE.

Momentum

Related topics

Latest papers 158