Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Nicolas Buzeta; Felipe del Rio; Cristian Hinostroza; Denis Parra; Hans Lobel; Rodrigo Toro Icarte

cs.LGFeb 16, 2026

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Authors: Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza, Denis Parra, Hans Lobel, Rodrigo Toro Icarte

Organizations: Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.

Abstract

Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out of distribution, while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model's internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes, visual encoders, and initializations, and show that analogous shifts occur during pretrained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.

Figures & tables

Figure 1: Overview of the training pipeline, showing the progression from text-only training to vision–language training.

Panel of Figure 2 — Figure 2: Binding accuracy comparison between text-only LLMs and their VLM counterparts across three model generations. VLMs consistently outperform LLMs as context length increases. The left panel shows performance on the simpler direct retrieval task, while the right panel shows the harder indirect retrieval task. Individual breakdown per model is provided in Appendix A .

Figure 3: Combined generalization performance across all models. Text-Only (Blue) suffers from sharp degradation on OOD lengths (37.2% average). Image-Text (Green) significantly improves generalization (69.5% average). Noise-Text (Orange) benefits from positional range expansion but remains insufficient (57.5% average). Noise-Image-Text (Red) achieves the strongest robustness (83.6% average). Individual plots are provided in Appendix E .

Figure 4: Binding Mechanism Shift. Interchange intervention results showing the dominant binding mechanism at each layer across all model variants. Text-only models (

\mathcal{M}_{\text{text-only}}

) rely on the positional mechanism (blue). Noise augmentation (

\mathcal{M}_{\text{noise-text}}

) introduces an increase in symbolic binding, but the model remains predominantly positional. Image-trained models (

\mathcal{M}_{\text{image-text}}

and

\mathcal{M}_{\text{noise-image-text}}

) both transition to a symbolic mechanism (orange).

Model	Peak Layer	Sym./Pos. Ratio	$\Delta$
Qwen 2	22	1.383	—
Qwen 2-VL	22	1.499	+0.116
Qwen 2.5	22	1.218	—
Qwen 2.5-VL	22	1.282	+0.064
Qwen 3	28	1.819	—
Qwen 3-VL	28	2.463	+0.644

Table 1: Binding Mechanisms in Large-Scale Models. Ratio of symbolic to positional mechanism attribution at the primary binding layer. VLM variants consistently exhibit a higher symbolic-to-positional ratio than their text-only counterparts.

Panel of Figure 5 — Figure 5: Circuits corresponding to the binding mechanisms. Early layers encode token-level information (color, shape, item identity). Colored arrows indicate attention-mediated information transfer between tokens. The key distinction: the Positional Circuit transfers only position indices (implicit binding), while Symbolic Circuits transfer semantic content (explicit binding).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Detailed performance breakdown for all Qwen model generations on the Direct Retrieval and Indirect Retrieval tasks.

Panel of Figure 7 — Figure 7: Example rendered contexts for the image modality, showing colored shapes drawn as filled SVG graphics on a white background.

Hyperparameter	Value
Number of layers	12
Hidden dimension ( $d_{\text{model}}$ )	128
FFN dimension ( $d_{\text{ff}}$ )	512
Attention heads	4
Positional encoding	RoPE ( $\theta=10000$ )
Dropout	0

Appendix

Table 2: Model architecture hyperparameters.

Panel of Figure 8 — Figure 8: Individual generalization plots for the four scratch model variants. The vertical dashed line indicates the maximum sequence length seen during training (8 items).

Panel of Figure 9 — Figure 9: Aggregate Interchange Results. (a) Across all 4 random seeds, the text-only models consistently converge to a Positional Context mechanism, though the specific layer where this mechanism activates varies (e.g., Layer 5 vs Layer 11). (b) Conversely, models exposed to the visual curriculum consistently develop Lexical binding mechanisms, regardless of the specific image encoder used (ResNet vs ViT vs DINO).

Figure 10: Attention Knockout Analysis. Heatmaps showing the drop in performance when attention between specific token pairs is ablated. (a) The Positional mechanism relies on position-matching pathways. (b, c) The Symbolic mechanisms show attribute-routing pathways.

Figure 11: Linear Probe Analysis. (a) The Positional model accumulates position but not attribute information at entity tokens. (b, c) Symbolic mechanisms exhibit the “binding signature”: attribute decodability surges at entity positions.

Panel of Figure 12 — Figure 12: Aggregate Results: Text Models. All four seeds show the positional pattern: high position decodability, low attribute decodability at entities, and sparse position-matching knockout matrices.

Panel of Figure 13 — Figure 13: Aggregate Results: Image-Text Models. All twelve runs (4 seeds $\times$ 3 encoders) show the symbolic pattern: binding signature in probes and attribute-routing pathways in knockouts.

Panel of Figure 14 — Figure 14: Color at Context Shape. Linear probes decoding color identity at the context shape token position. The VLM models show higher color decodability than their text-only counterparts, indicating binding in context.

Panel of Figure 15 — Figure 15: Color at Association Shape. Linear probes decoding color identity at the association shape token position. The VLM models show higher color decodability, consistent with color propagation from context to association.

Cross-Modal Learning

Cross-modal learning connects information across different data types, enabling models to align, transfer, and reason across modalities.

See all 143 papers

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
Oct 18, 2026Shashwat Singh, He He, Tal LinzenVision-Language Alignment Cross-Modal Representation Transfer
New York University
Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
Apr 19, 2026Lijie ZhouCross-Modal Learning VLM Robustness
School of Computer Science University of Nottingham Ningbo China Ningbo, China

Oct 18, 2026J/K move · Enter open · S save

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Shashwat Singh, He He, Tal Linzen

New York University

Vision-Language Alignment Cross-Modal Representation Transfer Reasoning for VLMs Representation Geometry in VLMs VLM Distillation

Open paper AskPDF arXiv

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text---especially on tasks where visualization would help, like geometry questions posed in words. We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure). To test for strong multimodality, we construct a task of shape-counting problems with paired equivalent text and image specifications. This task is trivial when the image is provided, but requires visualization when specified in text as line coordinates. We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task. We find that the multimodal models do no better in this setting, despite strong performance on the image version of the task. Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts. Finally, we provide an optimization objective that distills in-context visual embeddings into text embeddings. This alignment induces gains on the text variant of the task without explicit task supervision, even on samples more complex than seen during alignment. This suggests that VQA-style alignment leaves available cross-modal transfer untapped.

Abstract

Cross-Modal Learning

Explore similar work

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Distilling Visual Reasoning into Text Space

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability