LLaVA-style Vision-Language Models (VLMs) pass visual tokens from a fixed late layer of the vision backbone, typically the penultimate one, to the language model. We first show that this hidden convention is fragile: across 2 VLMs and 7 image and video benchmarks, the default layer is sub-optimal in 13 of 14 model-task pairs, and the best layer shifts with both task and visual backbone. Finding that layer by exhaustive layer-wise inference is prohibitively expensive, and no better fixed default exists. We therefore ask whether layer usefulness can instead be predicted from representation geometry. We study matrix-based entropy, introduced for unimodal layer analysis, which we compute over sample-level visual embeddings as Visual Dataset Entropy (VDE); and Gromov-Wasserstein (GW) distance, introduced for encoder-level VLM model selection, which we repurpose as a layer-wise visual--language alignment signal. Transferring these to LLaVA-based models is not obvious a priori: the vision tower is frozen while the multimodal projector is trained, so we profile both sides of the projector. We find that VDE transfers, and GW does not. Computed from 100 unlabeled task samples without downstream inference, pre-projector VDE tracks layer-wise accuracy and its top-ranked layers cover the oracle best layer on every task for the SigLIP-based LLaVA-Video, while giving region-level guidance for the CLIP-based Video-LLaVA. Post-projector profiles show that the projector reshapes visual geometry but does not erase the performance-relevant trend, leaving VDEpre the stronger signal. GW instead flattens after projection and is best read as an alignment diagnostic rather than a selector. VDE thus offers an interpretable, training-free policy that narrows the visual-layer search to a handful of candidates for limited downstream verification.
Figures & tables
Task
LLaVA-Video [ 32 ] / SigLIP [ 31 ]
Video-LLaVA [ 11 ] / CLIP [ 21 ]
Default Layer = 26
Default Layer = 23
Default Acc.
Best Acc.
Best Layer
Default Acc.
Best Acc.
Best Layer
ScienceQA [ 16 ]
87.62
88.90
27
59.63
59.63
23
POPE-Adv. [ 10 ]
86.60
87.00
27
79.80
80.53
22
CV-Bench 2D [ 26 ]
68.50
70.20
27
49.86
53.20
20
CV-Bench 3D [ 26 ]
82.00
83.17
27
57.33
58.83
21
Table 1: Preliminary finding: the vision backbone’s default layer is often sub-optimal.
Figure 1: Overview of the proposed visual-layer profiling framework. Two training-free geometric signals are computed across candidate layers: VDE, measured from pre- and post-projector visual representations, and GW, measuring visual-language structural alignment. Both are profiled against downstream layer usefulness to assess how well each predicts strong visual layers.
Domain
Dataset
Samples
Task Type
Image
ScienceQA
2,004
Multi-choice VQA
Image
POPE (Adv.)
3,000
Hallucination Detection
Image
CV-Bench 2D
1,438
Count & Relation Reasoning
Image
CV-Bench 3D
1,200
Depth & Distance Reasoning
Image
MMStar
1,500
General Multimodal Reasoning
Video
HD-EPIC (Action)
500
Egocentric Action Recognition
Table 2: Datasets used in our experiments.
Figure 2: Layer-wise visual representation profiling on ScienceQA. VDE is estimated from 100 sampled examples over five seeds and compared with exhaustive layer-wise downstream accuracy. GW distances measure the structural compatibility between visual and question-side language representations.
Figure 3: Layer-wise VDEpre , VDEpost , and downstream accuracy across five image benchmarks for Video-LLaVA (CLIP) and LLaVA-Video (SigLIP). Curves are averaged over five random seeds using 100 samples per seed. The dashed vertical line indicates the default visual layer used by each model.
Figure 4: Layer-wise profiling results on HD-EPIC Action Recognition. We perform a full-layer scan using 50 sampled examples with one random seed. The figure compares VDEpre , VDEpost , downstream accuracy, and GW-based cross-modal alignment across visual layers for the evaluated video VLMs.
Figure 5: Layer-wise profiling results on HD-EPIC Gaze Estimation. We use five random seeds with 100 sampled examples and restrict the scan to the mid-to-high visual layers. The figure compares VDEpre , VDEpost , and downstream accuracy across visual layers for the evaluated video VLMs.
Model
Task
VDE-pre Top-3
VDE-post Top-3
Pre Top-1/Best Acc. (%)
Post Top-1/Best Acc. (%)
Best Layer
Top-3 Hit: Pre/post
LLaVA-Video
ScienceQA
27 , 26, 25
27 , 26, 20
88.90 / 88.90
88.90 / 88.90
27
✓/ ✓
POPE-Adv.
27 , 26, 25
27 , 26, 25
87.00 / 87.00
87.00 / 87.00
27
✓/ ✓
CV-Bench 2D
27 , 26, 25
27 , 26, 25
70.20 / 70.20
70.20 / 70.20
27
✓/ ✓
CV-Bench 3D
27 , 25, 26
19, 18, 20
83.17 / 83.17
75.92 / 83.17
27
✓/ ✗
MMStar
27 , 26, 25
26, 24, 23
58.07 / 58.07
57.20 / 58.07
27
✓/ ✗
HD-EPIC Action
25 , 24, 23
27, 18, 14
61.00 / 61.00
58.90 / 61.00
25
✓/ ✗
Table 3: Top-3 candidate layer retrieval. Hits denote whether the oracle best layer is covered by the Top-3 layers ranked by VDEpre or VDEpost . Pre/Post Top-1 vs. Best Acc. compares Top-1 VDE-selected accuracy with oracle best accuracy. Bold entries mark covered oracle layers or exact Top-1 matches.
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
While open sourced Vision-Language Models (VLMs) have proliferated, selecting the optimal pretrained model for a specific downstream task remains challenging. Exhaustive evaluation is often infeasible due to computational constraints and data limitations in few shot scenarios. Existing selection methods fail to fully address this: they either rely on data-intensive proxies or use symmetric textual descriptors that neglect the inherently directional and model-specific nature of transferability. To address this problem, we propose a framework that grounds model selection in the internal functional dynamics of the visual encoder. Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment. Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks. This allows for predicting target model rankings by aggregating source task ranks without direct inference. Experimental results on 48 VLMs across 21 datasets demonstrate that our method outperforms state-of-the-art baselines, achieving a 14.7% improvement in NDCG@5 over SWAB.
Embedded numeric anchors on images systematically bias Vision-Language Model quality judgments across six VLMs from five architectural families (ANOVA eta^2 = 0.18-0.77, all p < 0.001). Anchor effects are 2.5x larger than severe image quality degradation, confirming bias is not reducible to visual changes. Layer-wise probing reveals consistent dissociation: layers where anchor classification saturates (L12-L34) are suboptimal for quality prediction, with optimal layers deeper (R^2 = 0.69-0.91). Fusion analysis identifies architecture-dependent integration -- instant fusion at L1-L2 in two models versus partial or no fusion in three others. These results establish a causal account of visual anchoring bias, linking behavioral susceptibility to representation dynamics.