Persistent Representational Misalignment Induces Vision-Language Model Failures

Yonatan Gideoni; Yoav Gelberg; Tim G. J. Rudner; Yarin Gal

cs.CVOct 18, 2026

Persistent Representational Misalignment Induces Vision-Language Model Failures

Authors: Yonatan Gideoni, Yoav Gelberg, Tim G. J. Rudner, Yarin Gal

Organizations: University of Oxford · University of Toronto

Abstract

Vision-language models (VLMs) fail at some tasks that are simple for humans---but why? We hypothesize that many such failures stem from VLMs' difficulties translating between vision and language representations, a misalignment the Platonic Representation Hypothesis predicts should disappear with scale. This paper is composed of two parts: in the first, we show that existing vision and language representations are linearly unalignable. This misalignment is an inherent property of the data and hence expected to persist regardless of either scale or the alignment's complexity. In the second, we demonstrate how this misalignment both leads to a predictably induced generalization failure and correlates with existing failure modes, where more representationally aligned VLM backbones accordingly exhibit fewer failures. Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.

Figures & tables

Representations		Cosine sim. RMSE ratio
Text	Text’	1.0
Vis	Vis’	1.3
Vis	Text	3.3
Vis	CLIP Vis	10.9
Text	CLIP Text	11.2
CLIP Vis	CLIP Text	2.9

Table 1: Relative cosine similarity RMSEs for representations from different vision and language models. RMSEs across different modalities are typically at least

3\times

higher than those within modalities. All values are relative to Text,Text’, where

\operatorname{CosSimRMSE}(\text{Text},\text{Text'})=0.05

. RMSEs are calculated using image/caption pairs from the COCO dataset. Models are DINOv3-B and -S for Vis and Vis’, all-MiniLM-L12-v2 and -L6-v2 for Text and Text’, and the CLIP-B/16 vision/text encoders for CLIP Vis/Text.

Figure 2: Vision models do not exhibit the same analogies that language models do . (left) top-10 analogy retrieval accuracies over VisLingA for several model families. The vision models (downwards triangles) do significantly worse than the text models (circles). (right) average retrieval rank. All vision models have a retrieval rank above 100, with a random baseline having a rank of 373.5. In both cases, the CLIP’s text encoders are the worst models in the language family and the opposite for the vision CLIP models, likely due to the intermodal supervision pulling each CLIP encoder towards the other modality. ST=Sentence Transformer ( Reimers and Gurevych, 2019 ) , GTR is from Ni et al. (2022) , CLIP is the LAION CLIP models ( Cherti et al., 2023 ) , ViT are ViT models trained on ImageNet 21k ( Dosovitskiy et al., 2020 ) .

Panel of Figure 3 — Figure 3: Pretrained vision and language models can each know what a zebra is in isolation, while a unified vision-language model fails if its adapter was not trained on zebra image-text pairs. Wavy lines denote feature vectors. Zebra images from Commons (2024) .

Setup (# Parameters)	Held-out Concept Detection Accuracy
Retrieval (210M)
Without concepts	3.8% [0.2%, 9.2%]
Full dataset	81.0% [47.3%, 99.4%]
CLIP (ViT-B/16)	79.2% [46.5%, 97.6%]
Generation (210M)
Without concepts	1.2% [0.0%, 3.0%]

Table 2: Held-out concept detection accuracy across settings. In all cases, the trained VLMs fail to detect the held-out concepts when they are not included in their paired image-text finetuning data. Large-scale generation refers to LLaVA-1.5-7B. Values in brackets denote bootstrapped 95% confidence intervals across per-concept accuracies. † ( NLP Connect, 2022 )

Panel of Figure 4 — Figure 4: Image descriptions given by a LLaVA-1.5-7B model trained without the held-out concepts on images containing them when prompted “What is in this image?”. The corresponding concepts are (a) pizza, (b) umbrella, and (c) toothbrush. The model fails to detect them for (a) and (b), while it succeeds for (c). See Appendix I for more examples.

Figure 5: Vision/language backbone representational alignment is correlated with better performance on benchmarks showcasing VLM failures (left four plots). Each dot represents a LLaVA model trained with a different vision backbone. Representational alignment is measured using the mutual nearest neighbours similarity metric of Huh et al. (2024) . (right two plots) show that on datasets which are difficult due to the model not having other capabilities, e.g. reasoning, the correlation is much smaller.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Additional examples of cases of similar images with dissimilar captions or vice versa. Beneath each pair is the cosine similarity between the images and the average embedding of all of their possible captions, with a random one of the captions being shown. Pairs are sampled uniformly at random from all pairs whose image and caption cosine similarities differ by more than $0.4$ .

Panel of Figure 7 — Figure 7: Randomly sampled VisLingA concepts and some of their images. Each column shows 5 randomly selected images for a randomly selected concept, named below the column. Images are center-cropped for display only and in practice have varying ratios.

Adapter type	Learning rate	Weight decay
Linear	$10^{-2.5}$	$10^{-2}$
MLP	$10^{-3}$	$10^{-2}$
Attention	$10^{-3.5}$	$10^{-3}$

Appendix

Table 3: Optimal learning rates and weight decays for different adapters for the small-scale generative experiments.

Panel of Figure 8 — Figure 8: Additional examples of generalization failures from the LLaVA-1.5-7B model from Figure 4 , trained without the held-out concepts, when prompted “What is in this image?”. Rows correspond to the concepts pizza, umbrella, toothbrush, zebra, and banana respectively. For each concept we randomly sample two images whose description does not mention the concept (left, center) and one image where the concept was correctly detected (right). Note that this 2:1 ratio is purely illustrative and is not representative of the real detection rate per concept.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 355 papers

Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models
Jul 6, 2026Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2VLM Evaluation Vision-Language Models
1MBZUAI · 2The University of Melbourne
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1VLM Hallucination Mitigation Vision-Language Models
Indian Institute of Technology Dhanbad · National University of Singapore

Feb 6, 2026J/K move · Enter open · S save

Same Answer, Different Representations: Hidden instability in VLMs

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema+5

Sapienza University of Rome · University of Edinburgh · University of Oxford+5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Open paper AskPDF arXiv

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Abstract

Vision-Language Alignment

Explore similar work

Same Answer, Different Representations: Hidden instability in VLMs

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models