A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Shashwat Singh; He He; Tal Linzen

cs.AIOct 18, 2026

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Authors: Shashwat Singh, He He, Tal Linzen

Organizations: New York University

Abstract

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words. We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure). To test for strong multimodality, we construct a task of shape-counting problems with paired equivalent text and image specifications. This task is trivial when the image is provided, but requires visualization when specified in text as line coordinates. We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task. We find that the multimodal models do no better in this setting, despite strong performance on the image version of the task. Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts. Finally, we provide an optimization objective that distills in-context visual embeddings into text embeddings. This alignment induces gains on the text variant of the task without explicit task supervision, even on samples more complex than seen during alignment. This suggests that VQA-style alignment leaves available cross-modal transfer untapped.

Figures & tables

Panel of Figure 1 — Figure 1: A simple example of the paired data sample: the code snippet (left) when compiled generates the image (right). The task is simply to count shapes of each kind given either the code snippet (for the text-variant of the task) or the rendered image (for the image variant).

Figure 2: When the problem is provided in text, for all the settings tested—with and without in-context examples (ICL), and with and without chain of thought (CoT)—the performance of the multimodal models is similar to their text-only counterparts, and far below the image ceiling. This indicates that vision alignment did not transfer visualization capabilities to the text decoder. Error bars are based on the 95% confidence interval on accuracy estimates using the normal approximation to the binomial distribution.

Figure 3: Vision-alignment does not add more visual structure to the embeddings of the multimodal model, compared to the text-only model. While probes trained on the embeddings from both classes of models show above-chance accuracy, the performance for the two classes of models is very similar.

Figure 4: The trends on learnt retrieval corroborate that the multimodal decoder does not have better visual representations of textual inputs than the text-only model. Both the embedding sets get significantly above-chance and above-control retrieval accuracy against DiNo embeddings – at least in the middle layers. However, the VL embeddings closely track text embeddings.

Figure 5: Given paired text-image data our alignment method trains the model to predict from text (code) the embeddings it would’ve had given the image. During alignment (left) we provide no task context or answer labels. During evaluation (right) the model performs the shape counting task.

Figure 6: In all settings the distilled model outperforms the base models on across-shape accuracy even beyond the alignment cutoff, that is, the maximum number of shapes the distilled model has seen in a single sample. All the subplots represent the same set of samples – just with different prompts. All the models in this experiment are 7B models (Qwen 2.5 Instruct and Qwen 2.5 VL Instruct).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: We observe the same trend as in Figure 4 i.e. the text embeddings from the text-only and the VL models show similar propensity to align with the corresponding image embeddings. The absolute numbers are higher than seen in Figure 4 because siglip image embeddings are pre-trained with text supervision and it is not a “pure” image encoder.

	VL	Text	Distilled
Commonsense 0 Shot (Accuracy)	60.33%	59.39%	59.10%
MMLU (macro avg)	72.49%	74.66%	70.14%

Appendix

Table 1: There is minimal to no degradation from the initialized VL checkpoint due to the distillation.

Figure 8: The models perform much better at the easier sub-task of getting the total number of shapes in a sample.

Panel of Figure 9 — Figure 9: The distilled model representations are better across layers.

Figure 10: The distilled model representations are better across layers.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 355 papers

Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
Seeing to Generalize: How Visual Data Corrects Binding Shortcuts
Feb 16, 2026Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza +3Cross-Modal Learning Shortcut Learning
Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.
Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
Apr 3, 2026Mingyeong Kim, Jungwon Choi, Chaeyun Jang +1Modality Gap in VLMs VLM Inference
Kim Jaechul Graduate School of AI, KAIST

Sep 28, 2026J/K move · Enter open · S save

Distilling Visual Reasoning into Text Space

Wenhan Yang, Nilay Naharas, Ali Payani, Baharan Mirzasoleiman

University of California, Los Angeles · Cisco Research

CoT Distillation VLM Distillation Large Vision-Language Models VLM Reasoning

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

Abstract

Vision-Language Alignment

Explore similar work

Distilling Visual Reasoning into Text Space

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models