Representation Geometry in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

21 papers10 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
Oct 18, 2026Shashwat Singh, He He, Tal LinzenVision-Language Alignment Cross-Modal Representation Transfer
New York University
Persistent Representational Misalignment Induces Vision-Language Model Failures
Oct 18, 2026Yonatan Gideoni, Yoav Gelberg, Tim G. J. Rudner +1Vision-Language Alignment Modality Gap in VLMs
University of Oxford · University of Toronto
Relational Spatial Alignment for Vision-Language-Action Models
Oct 17, 2026Chaoran Zhu, Hengyi Wang, Chenglin Cui +3Representation Alignment 3D Spatial Reasoning
Queen Mary University of London, United Kingdom · University College London, United Kingdom · VGG, University of Oxford, United Kingdom
Is a Linear Probe Evidence of a Linear Representation?
Oct 15, 2026Nadir Kutluozen, Christian Internò, David KlindtRepresentation Geometry in VLMs Linear Representation Hypothesis
Cold Spring Harbor Laboratory · Bielefeld University
GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models
Oct 13, 2026Rong Li, Shijie Li, Jiangyong Huang +7Geospatial QA Visual Spatial Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute for General Artificial Intelligence · Agency for Science, A*STAR +3
Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
Oct 8, 2026Nurbüke Teker, Rui Xiao, Zeynep Akata +1Representation Geometry in VLMs Vision-Language Models
Technical University of Munich · Munich Center for Machine Learning · University of Washington +1
GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
Oct 6, 2026Seulgi Kim, Zhixiong Zhang, Xinwei Zhang +2Vision-Language Grounding Representation Geometry in VLMs
Georgia Institute of Technology · Work done as an intern at Amazon. · Amazon
Are In-Context Images Worth 10 Dimensions?
Sep 29, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Vision-Language Models Multimodal ICL
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, Paris, France · Pôle recherche de l’AMIAD, Palaiseau, France · Institut Universitaire de France, Paris, France
Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models
Sep 29, 2026Abdullah All Tanvir, Xin ZhongInformation Geometry Neural Representation Geometry
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Geometric Encoding for Spatial Reasoning in Vision-Language Models
Sep 28, 2026Antonio Jun, Haoshui Yu, Zhengyi Lu +2Visual Spatial Reasoning Representation Geometry in VLMs
School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States
Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Models
Aug 17, 2026Ruchen Liu, Yi Yang, Yiming Xu +3Vision-Language Models VLM Evaluation
Leibniz Universität Hannover, Germany · University of Bath, UK
Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations
Aug 1, 2026Shalom Kachko, Raz Lapid, Margarita Vald +2Vision-Language Models Representation Geometry in VLMs
1Ben-Gurion University of the Negev · 2Intuit Inc. · 3Deepkeep
Prior Directions: Why GUI Grounding Gets Locked in the Past
Jul 29, 2026Weile Gong, Zijian Lu, Mingcai Chen +3Vision-Language Models Representation Geometry in VLMs
Nanjing University of Posts and Telecommunications, Nanjing, China
Dive Into the Implicit Biases of Low-rank Vision-language Alignment
Jul 9, 2026Mingjia Shi, Shuo Wang, Xiaobo Wang +7Vision-Language Models Representation Geometry in VLMs
Mininglamp · Shenzhen University of Advanced Technology · National University of Singapore
HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
Jun 4, 2026Moshiur Farazi, Sameera Ramasinghe, Mahbub Ahmed Turza +1Hyperbolic Representation Learning Visual Reasoning
Data Science and AI, University of Doha for Science and Technology, Qatar · Pluralis Research, Australia · Department of Electrical and Computer Engineering, North South University, Bangladesh
Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
May 28, 2026Cheolhong Min, Jaeyun Jung, Daeun Lee +5VLM Evaluation Spatial Reasoning Benchmarks
Seoul National University · The Ohio State University · NVIDIA
3D Primitives are a Spatial Language for VLMs
May 12, 2026Junze Liu, Kun Qian, Florian Dubost +8Visually Grounded Reasoning 3D Spatial Reasoning
Unity Technologies
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1VLM Hallucination Mitigation Vision-Language Models
Indian Institute of Technology Dhanbad · National University of Singapore
Why MLLMs Struggle to Determine Object Orientations
Apr 14, 2026Anju Gopinath, Nikhil Krishnaswamy, Bruce DraperRepresentation Geometry in VLMs Vision-Language Models
Department of Computer Science, Colorado State University, Fort Collins, CO, USA
Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
Mar 6, 2026Yakov Pyotr ShkolnikovRepresentation Geometry in VLMs Vision-Language Models
Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5

Oct 18, 2026J/K move · Enter open · S save

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Shashwat Singh, He He, Tal Linzen

New York University

Vision-Language Alignment Cross-Modal Representation Transfer Reasoning for VLMs Representation Geometry in VLMs VLM Distillation

Open paper AskPDF arXiv

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words. We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure). To test for strong multimodality, we construct a task of shape-counting problems with paired equivalent text and image specifications. This task is trivial when the image is provided, but requires visualization when specified in text as line coordinates. We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task. We find that the multimodal models do no better in this setting, despite strong performance on the image version of the task. Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts. Finally, we provide an optimization objective that distills in-context visual embeddings into text embeddings. This alignment induces gains on the text variant of the task without explicit task supervision, even on samples more complex than seen during alignment. This suggests that VQA-style alignment leaves available cross-modal transfer untapped.