cs.AIOct 18, 2026

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Authors: Shashwat Singh, He He, Tal Linzen

Organizations: New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words. We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure). To test for strong multimodality, we construct a task of shape-counting problems with paired equivalent text and image specifications. This task is trivial when the image is provided, but requires visualization when specified in text as line coordinates. We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task. We find that the multimodal models do no better in this setting, despite strong performance on the image version of the task. Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts. Finally, we provide an optimization objective that distills in-context visual embeddings into text embeddings. This alignment induces gains on the text variant of the task without explicit task supervision, even on samples more complex than seen during alignment. This suggests that VQA-style alignment leaves available cross-modal transfer untapped.

Figures & tables

Figure 1: A simple example of the paired data sample: the code snippet (left) when compiled generates the image (right). The task is simply to count shapes of each kind given either the code snippet (for the text-variant of the task) or the rendered image (for the image variant).
Figure 2: When the problem is provided in text, for all the settings tested—with and without in-context examples (ICL), and with and without chain of thought (CoT)—the performance of the multimodal models is similar to their text-only counterparts, and far below the image ceiling. This indicates that vision alignment did not transfer visualization capabilities to the text decoder. Error bars are based on the 95% confidence interval on accuracy estimates using the normal approximation to the binomial distribution.
Figure 3: Vision-alignment does not add more visual structure to the embeddings of the multimodal model, compared to the text-only model. While probes trained on the embeddings from both classes of models show above-chance accuracy, the performance for the two classes of models is very similar.
Figure 4: The trends on learnt retrieval corroborate that the multimodal decoder does not have better visual representations of textual inputs than the text-only model. Both the embedding sets get significantly above-chance and above-control retrieval accuracy against DiNo embeddings – at least in the middle layers. However, the VL embeddings closely track text embeddings.
Figure 5: Given paired text-image data our alignment method trains the model to predict from text (code) the embeddings it would’ve had given the image. During alignment (left) we provide no task context or answer labels. During evaluation (right) the model performs the shape counting task.
Figure 6: In all settings the distilled model outperforms the base models on across-shape accuracy even beyond the alignment cutoff, that is, the maximum number of shapes the distilled model has seen in a single sample. All the subplots represent the same set of samples – just with different prompts. All the models in this experiment are 7B models (Qwen 2.5 Instruct and Qwen 2.5 VL Instruct).
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: We observe the same trend as in Figure 4 i.e. the text embeddings from the text-only and the VL models show similar propensity to align with the corresponding image embeddings. The absolute numbers are higher than seen in Figure 4 because siglip image embeddings are pre-trained with text supervision and it is not a “pure” image encoder.
 VLTextDistilled
Commonsense 0 Shot (Accuracy)60.33%59.39%59.10%
MMLU (macro avg)72.49%74.66%70.14%
Appendix
Table 1: There is minimal to no degradation from the initialized VL checkpoint due to the distillation.
Figure 8: The models perform much better at the easier sub-task of getting the total number of shapes in a sample.
Figure 9: The distilled model representations are better across layers.
Figure 10: The distilled model representations are better across layers.
More in this topic

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 355 papers

Explore similar work

CardsList
Sep 28, 2026

Distilling Visual Reasoning into Text Space

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

Wenhan Yang, Nilay Naharas, Ali Payani +1

University of California, Los Angeles · Cisco Research

CoT DistillationVLM DistillationLarge Vision-Language ModelsVLM Reasoning
AskPDF
Feb 16, 2026

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out of distribution, while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model's internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes, visual encoders, and initializations, and show that analogous shifts occur during pretrained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.

Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza +3

Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.

Cross-Modal LearningShortcut LearningVLM InterpretabilityLong-Context VLMs
AskPDF
Apr 3, 2026

Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models

Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images. We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting. This failure is not explained only by missing semantic information. Even when text descriptions preserve key content, confidence becomes unreliable, while adding a visual signal through generated images partially restores accuracy and calibration. We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis. Across text-only benchmarks, unseen tasks, and missing-image scenarios, LIM improves accuracy and reduces calibration error. These results suggest that latent modality completion is a practical approach for reliable VLM inference under missing-modality.

Mingyeong Kim, Jungwon Choi, Chaeyun Jang +1

Kim Jaechul Graduate School of AI, KAIST

Modality Gap in VLMsVLM InferenceMissing-Modality LearningVLM Calibration
AskPDF