cs.CVOct 4, 2026

A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models

Authors: Yilin Yang, Jun-Tao Tang, Kengyi Wang, Siyuan Su, Gaoyong Luo, Mingda Chen

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University · Nanjing University · Fudan University · Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although recent studies have shown that cross-modal metrics can better capture such performance, unimodal metrics remain the dominant choice in practice. In this work, we revisit cross-modal evaluation of vision encoders through large-scale experiments. We identify important limitations in both the experimental design and methodological formulation of prior approaches. After addressing these limitations and introducing simple improvements, we propose RAVEL, a training-free method based on cross-modal nearest-neighbor retrieval. Despite its simplicity, RAVEL achieves state-of-the-art performance across our experiments, outperforming prior methods by a substantial margin. Our results demonstrate that simple cross-modal metrics, when evaluated under a careful and comprehensive setup, can provide a strong basis for evaluating vision encoders for MLLMs.

Figures & tables

Figure 1: Effect of vision encoder candidate-pool size and heterogeneity on evaluation metrics. Cross-modal metrics outperform unimodal alternatives on moderately sized candidate pools, but their predictive performance drops substantially as more diverse vision encoders are added. In contrast, RAVEL, with improved representation conditioning and finer-grained similarity scoring, consistently outperforms competing metrics across different vision encoder pools.
 Qwen3-1.7BQwen2.5-1.5BSmolLM2-1.7B
Methodρ↑\rho\uparrowr↑r\uparrowρ↑\rho\uparrowr↑r\uparrowρ↑\rho\uparrowr↑r\uparrow
Vision-only metrics
Linear probing ( Caron et al., 2021 )0.4840.4900.5680.5100.5930.597
KNN ( Wu et al., 2018 )0.5510.5130.5340.5980.6030.567
Cross-modal metrics requiring additional model training
Mid-Training Loss0.3420.3490.4340.4030.2720.300
Table 1: FLOPs-controlled comparison of different evaluation metrics. All methods use matched computational budgets except AC Policy † , whose computational cost cannot be reduced to the matched FLOP budget due to its method design. Best and second-best results are shown in bold and underlined, respectively.
Table 3
Figure 2: Feature representation and patch-set matching in RAVEL. (a) Effect of the text feature extraction layer on ranking correlation. (b) Patch-set matching does not benefit long captions. (c) Patch differentiation under controlled feature intervention.
Figure 3: Representation post-processing choices. (a) Five post-processing methods and the unprocessed baseline. (b) Spearman change after removing high- or low-variance visual PCs. (c) Ablation of L2 normalization and spectral stabilization under conventional PCA whitening.
Figure 4: Patch-level matching preserves local structure. (a) Red lines link high-similarity patch pairs. (b) Color shows each patch’s best-match similarity (brighter == stronger).
Table 7
Figure 5: Robustness and scaling behavior.
Figure 6: Reconstruction quality does not determine MLLM utility. In the first example, the MLLM correctly identifies the author despite the unreadable reconstructed text. In the second, the reconstruction clearly preserves the book title, yet the MLLM produces an incorrect answer.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of RAVEL. Given paired image–text samples, RAVEL applies stabilized PCA whitening, constructs visual and textual neighborhoods using patch-level and global similarity, respectively, and measures their agreement through the known image–text correspondences.
Family (#)Architectures and input resolutionsReference
Language-supervised continuous encoders
OpenAI CLIP (1)ViT-L/14 (224)Radford et al.,2021
MetaCLIP (9)ViT-B/16 (400M, 224; 2.5B, 224); ViT-B/32 (400M, 224; 2.5B, 224); ViT-L/14 (400M, 224; 2.5B, 224); ViT-H/14 (2.5B, 224; v1.2, 224); ViT-G/14 (2.5B, 224)Xu et al.,2024
MetaCLIP 2 (15)ViT-B/16 (224, 384); ViT-B/32 (224, 384; mT5, 224); ViT-G/14 (224, 378); ViT-H/14 (378); ViT-L/14 (224); ViT-M/16 (224, 384; mT5, 224); ViT-S/16 (224, 384; mT5, 224)Chuang et al.,2025
SigLIP2 (15)So400m/14 (224, 384); So400m/16 (256, 384, 512); ViT-B/16 (224, 256, 384, 512); ViT-B/32 (256); ViT-G/16 (256, 384); ViT-L/16 (256, 384, 512)Tschannen et al.,2025
Perception Encoder (3)PE-Core-B/16 (224); PE-Core-G/14 (448); PE-Lang-L/14 (448)Bolya et al.,2025
Appendix
Table 6: Complete vision encoder pool. All 70 evaluated architecture, checkpoint, and input-resolution configurations are listed and grouped by model family.

Explore similar work

CardsList

  1. RAVE: Re-Allocating Visual Attention in Large Multimodal Models

    May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Cross-Modal AttentionLarge Multimodal Models

    Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology

  2. How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

    Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal PretrainingMultimodal Large Language Models

    CASIA · UCAS · Foundation Model Department, Tencent

  3. Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

    May 2, 2026Muyang Li, Yucheng Liu, Jianbo Ma +3Vision EncodersVision-Language Alignment

    1Sydney AI Centre, The University of Sydney · 2Dolby Laboratories · 3TMLR Group, Hong Kong Baptist University