Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers

Sugam Karki; Raúl Vázquez; Yova Kementchedjhieva

cs.CLOct 23, 2026

Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers

Authors: Sugam Karki, Raúl Vázquez, Yova Kementchedjhieva

Organizations: MBZUAI · University of Helsinki

Abstract

While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.

Figures & tables

Figure 1: Top: Real images and rendered text variants used for cross-modal probing. Bottom: t-SNE projections of CLIP (1B) and DINO (1B) representations across depth. In CLIP, same-concept representations from both modalities progressively align, while DINO maintains modality separation throughout.

Figure 2: Rectangle input controls for three stylistic text renders of the word “horse”. The control follows the same rendering style as the text.

Figure 3: Layer-wise PAG across 1B vision encoders for cross-modal classification. In CLIP, image recognition emerges first, followed by text recognition, and finally cross-modal transfer in the deepest layers. Cross-modal transfer remains near baseline in DINO and MAE.

Figure 4: Layer-wise Recall@1 for relational retrieval tasks. CLIP develops strong late-layer alignment for direction, cardinality, and size, while DINOv2 and MAE remain near chance across depth.

Figure 5: Layer-wise PAG across 1B encoders for linguistic tasks. CLIP shows a clear surface-to-semantic progression; DINO and MAE show weak grammatical and near-absent semantic signals.

Figure 6: Left (1) : Cross-modal and unimodal classification across scales. Scale substantially improves cross-modal alignment. Right (2-4) : Layer-wise PAG across DINO scales (300M, 2B, 7B) for linguistic probing. Larger models, with the 7B model also trained on more data, exhibit stronger and more hierarchically organized representations.

Figure 7: Left (1) : Cross-modal and unimodal classification across data compositions: Normal, light (50.3% text only), heavy (1.3% OCR only). Data composition modestly improves cross-modal alignment. Right (2-4) : Layer-wise PAG across DINO pretrained on different data compositions. OCR-focused data strengthens and hierarchically organizes representations.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Identifier	Objective
300M
facebook/webssl-dino300m-full2b-224	DINO
1B
facebook/webssl-dino1b-full2b-224	DINO
facebook/webssl-mae1b-full2b-224	MAE
facebook/metaclip-h14-fullcc2.5b	CLIP

Appendix

Table 1: Vision encoders used in this work. All models are available at huggingface.co .

Figure 8: Layer-wise PAG across 1B vision encoders for cross-modal classification on ImageNet1k. Similar to results on Caltech256, in CLIP, image recognition develops first, followed by text recognition, and finally cross-modal alignment, each peaking in deeper layers. Cross-modal transfer is absent in both DINO and MAE.

Figure 9: Layer-wise zero-shot cross-modal retrieval on Caltech-256 measured with Recall@1. CLIP and DINOv2 7B exhibit sharp late-layer increases in retrieval alignment between rendered text and natural images, while DINOv2 1B and MAE remain near chance across depth.

Figure 10: Layer-wise zero-shot cross-modal retrieval on Caltech-256 measured with Recall@10. Similar to Recall@1, retrieval performance rises sharply only in the deepest layers of CLIP and DINOv2 7B, mirroring the late emergence of cross-modal transfer observed in probe-based experiments.

Figure 11: Layer-wise zero-shot relational retrieval measured with Recall@1. CLIP develops strong late-layer alignment for direction, cardinality, and size, while DINOv2 7B exhibits weaker but measurable relational alignment in deeper layers. Smaller self-supervised models remain near chance across depth, with spatial direction consistently emerging as the most difficult relational attribute.

Figure 12: Layer-wise PAG for contextual POS tagging across 1B vision encoders and DINOv2 7B. Similar to isolated-word POS probing, CLIP and DINOv2 7B exhibit progressively stronger grammatical signals in deeper layers, while DINOv2 1B and MAE remain substantially weaker.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning
Apr 3, 2026Ankan Deria, Komal Kumar, Xilin He +4Vision-Language Models Multimodal Fusion
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE
A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
Oct 18, 2026Shashwat Singh, He He, Tal LinzenVision-Language Alignment Cross-Modal Representation Transfer
New York University
Seeing to Generalize: How Visual Data Corrects Binding Shortcuts
Feb 16, 2026Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza +3Cross-Modal Learning Shortcut Learning
Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile · 2Centro Nacional de Inteligencia Artificial (CENIA), Santiago, Chile · 3Instituto Milenio en Ingeniería e Inteligencia Artificial para la Salud (iHEALTH), Santiago, Chile.

Apr 3, 2026J/K move · Enter open · S save

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

Ankan Deria, Komal Kumar, Xilin He, Imran Razzak+3

Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE

Vision-Language Models Multimodal Fusion Contrastive Learning for VLMs

Open paper AskPDF arXiv

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.

Abstract

Vision-Language Models

Explore similar work

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts