Vision-language models (VLMs) fail at some tasks that are simple for humans---but why? We hypothesize that many such failures stem from VLMs' difficulties translating between vision and language representations, a misalignment the Platonic Representation Hypothesis predicts should disappear with scale. This paper is composed of two parts: in the first, we show that existing vision and language representations are linearly unalignable. This misalignment is an inherent property of the data and hence expected to persist regardless of either scale or the alignment's complexity. In the second, we demonstrate how this misalignment both leads to a predictably induced generalization failure and correlates with existing failure modes, where more representationally aligned VLM backbones accordingly exhibit fewer failures. Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.
Figures & tables
Figure 1Figure 2
Figure 1: Two scenes can be similar in one modality but different in another. (left) shows two images which are visually similar, where a vision embedding model gives them a cosine similarity of 0.96 . Their corresponding captions are more distinct, having a similarity of 0.44 . (right) shows the opposite case, where the captions are similar but the images are distinct. The captions have a cosine similarity of 0.97 while the images’ cosine similarity is −0.06 . The text similarities are for the average embeddings over the five different captions given in COCO for each image. See Appendix D for more examples.
Representations
Cosine sim. RMSE ratio
Text
Text’
1.0
Vis
Vis’
1.3
Vis
Text
3.3
Vis
CLIP Vis
10.9
Text
CLIP Text
11.2
CLIP Vis
CLIP Text
2.9
Table 1: Relative cosine similarity RMSEs for representations from different vision and language models. RMSEs across different modalities are typically at least 3× higher than those within modalities. All values are relative to Text,Text’, where CosSimRMSE(Text,Text’)=0.05 . RMSEs are calculated using image/caption pairs from the COCO dataset. Models are DINOv3-B and -S for Vis and Vis’, all-MiniLM-L12-v2 and -L6-v2 for Text and Text’, and the CLIP-B/16 vision/text encoders for CLIP Vis/Text.
Figure 2: Vision models do not exhibit the same analogies that language models do . (left) top-10 analogy retrieval accuracies over VisLingA for several model families. The vision models (downwards triangles) do significantly worse than the text models (circles). (right) average retrieval rank. All vision models have a retrieval rank above 100, with a random baseline having a rank of 373.5. In both cases, the CLIP’s text encoders are the worst models in the language family and the opposite for the vision CLIP models, likely due to the intermodal supervision pulling each CLIP encoder towards the other modality. ST=Sentence Transformer ( Reimers and Gurevych, 2019 ) , GTR is from Ni et al. (2022) , CLIP is the LAION CLIP models ( Cherti et al., 2023 ) , ViT are ViT models trained on ImageNet 21k ( Dosovitskiy et al., 2020 ) .
Figure 3: Pretrained vision and language models can each know what a zebra is in isolation, while a unified vision-language model fails if its adapter was not trained on zebra image-text pairs. Wavy lines denote feature vectors. Zebra images from Commons (2024) .
Setup (# Parameters)
Held-out Concept Detection Accuracy
Retrieval (210M)
Without concepts
3.8% [0.2%, 9.2%]
Full dataset
81.0% [47.3%, 99.4%]
CLIP (ViT-B/16)
79.2% [46.5%, 97.6%]
Generation (210M)
Without concepts
1.2% [0.0%, 3.0%]
Table 2: Held-out concept detection accuracy across settings. In all cases, the trained VLMs fail to detect the held-out concepts when they are not included in their paired image-text finetuning data. Large-scale generation refers to LLaVA-1.5-7B. Values in brackets denote bootstrapped 95% confidence intervals across per-concept accuracies. † ( NLP Connect, 2022 )
Figure 4: Image descriptions given by a LLaVA-1.5-7B model trained without the held-out concepts on images containing them when prompted “What is in this image?”. The corresponding concepts are (a) pizza, (b) umbrella, and (c) toothbrush. The model fails to detect them for (a) and (b), while it succeeds for (c). See Appendix I for more examples.
Figure 5: Vision/language backbone representational alignment is correlated with better performance on benchmarks showcasing VLM failures (left four plots). Each dot represents a LLaVA model trained with a different vision backbone. Representational alignment is measured using the mutual nearest neighbours similarity metric of Huh et al. (2024) . (right two plots) show that on datasets which are difficult due to the model not having other capabilities, e.g. reasoning, the correlation is much smaller.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Additional examples of cases of similar images with dissimilar captions or vice versa. Beneath each pair is the cosine similarity between the images and the average embedding of all of their possible captions, with a random one of the captions being shown. Pairs are sampled uniformly at random from all pairs whose image and caption cosine similarities differ by more than 0.4 .
Figure 7: Randomly sampled VisLingA concepts and some of their images. Each column shows 5 randomly selected images for a randomly selected concept, named below the column. Images are center-cropped for display only and in practice have varying ratios.
Adapter type
Learning rate
Weight decay
Linear
10−2.5
10−2
MLP
10−3
10−2
Attention
10−3.5
10−3
Appendix
Table 3: Optimal learning rates and weight decays for different adapters for the small-scale generative experiments.
Figure 8: Additional examples of generalization failures from the LLaVA-1.5-7B model from Figure 4 , trained without the held-out concepts, when prompted “What is in this image?”. Rows correspond to the concepts pizza, umbrella, toothbrush, zebra, and banana respectively. For each concept we randomly sample two images whose description does not mention the concept (left, center) and one image where the concept was correctly detected (right). Note that this 2:1 ratio is purely illustrative and is not representative of the real detection rate per concept.
The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.
Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.
Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.
Harshvardhan Saini, Samyak Jha, Yiming Tang +1
Indian Institute of Technology Dhanbad · National University of Singapore