Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

Jonas Herzog; Yue Wang

cs.CVMar 17, 2026

Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

Authors: Jonas Herzog, Yue Wang

Organizations: Zhejiang University

Abstract

Recent research suggested that the embeddings produced by CLIP-like contrastive language-image training are suboptimal for image-only tasks. The main theory is that the inter-modal (language-image) alignment loss ignores intra-modal (image-image) alignment, leading to poorly calibrated distances between images. In this study, we question this intra-modal misalignment hypothesis. We reexamine its foundational theoretical argument, the indicators used to support it, and the performance metrics affected. For the theoretical argument, we demonstrate that there are no such supposed degrees of freedom for image embedding distances. For the empirical measures, our findings reveal they yield similar results for language-image trained models (CLIP, SigLIP) and image-image trained models (DINO, SigLIP2). This indicates the observed phenomena do not stem from a misalignment specific to the former. Experiments on the commonly studied intra-modal tasks retrieval and few-shot classification confirm that addressing task ambiguity, not supposed misalignment, is key for best results.

Figures & tables

Figure 1: Previous work illustrated an intra-modal misalignment in CLIP space by showing there are cat images closer to a dog ( $d_{\neq}$ ) than to another cat ( $d_{=}$ ). We argue $d_{\neq}<d_{=}$ is no sign of misalignment. For a labeled downstream dataset, intra-class variance of open vocabulary models is expected and desired to capture semantics and style beyond the narrow dataset-specific labels. Classifying and retrieving with frozen CLIP image embeddings still works well when similarities are measured along the dataset-specific semantic axes. Here the horizontal axis captures dog/cat.

Figure 2: Pairwise cosine similarity distributions. Left : Similarities between same class (blue) and opposite class (orange) image feature pairs. A high overlap ratio between the two colors was previously highlighted as an indicator for an intra-modal misalignment issue in CLIP. Right : Similarity distributions of image-text pairs (purple) versus image-image pairs (green). Because CLIP is only supervised on the former, the divergence has previously prompted concerns about whether the latter reflect true similarities. CLIP ViT-B/16. Dataset as in Tab. 1 .

Figure 3: Motivated by the intra-modal misalignment hypothesis, previous work [ 26 ] posited it is necessary to convert image-image comparison (left) into image-text comparison (right).

	Retrieval		Classification
Model	$T\text{-}I$	$I\text{-}I$	$T\text{-}I$ (0-shot)	$I\text{-}I$ (1 \|16-shot)
CLIP ViT-B/16	99.3	87.1	99.6	84.2 \| 99.7
DINOv2 ViT-B/14	-	81.8	-	76.2 \|97.3
DINOv3 ViT-L/16	-	84.3	-	80.2 \|97.8

Table 1: Repeating the demonstrative experiment in [ 26 ] on the simplistic Dogs vs Cats legacy dataset, where near-perfect results are expected. It was suggested in [ 26 ] that poor results with CLIP image-image similarities evidences a misalignment in the image-image space. This hypothesis gets no evidence when swapping model for the uni-modal DINO, a widely acknowledged state-of-the-art image embedder. CLIP scores highest, suggesting that the observed low metrics are not caused by a model weakness and hence neither by a misalignment. Instead, the performance gap between text-image (T-I) and image-image (I-I) can be attributed to the ambiguity [ 21 ] in the way the task is conveyed to the model: Two images with opposite labels might still share enough other concepts to be justifiably similar.

Figure 4: (a)-(c): The previous intra-modal degree-of-freedom argument in [ 40 ] illustrates that two image embeddings can be either close together (a) or far apart (b) while having the same text-image distance $r$ , concluding that two image embeddings can lie on any two arbitrary points on the circumference (c), leaving a degree of freedom for image-image miscalibration. Our interpretation (d)-(f): The previous line of argumentation overlooks that each image embedding is bound to more than one text anchor (f). Moreover, the two different configurations in (a) and (b) are not arbitrary, but have a good reason to exist: Images in (a,d) and (b,e) have equal distance $r$ to the “cat” text, but the two images in (a,d) are much more similar to each other than those in (b,e). Displayed distance values are real measurements.

Panel of Figure 5 — Figure 5: Cosine similarity histograms by class (left) and by modality (right) . The distributions are almost identical for purely text-image trained SigLIP (first row) and SigLIP2 (second row) which includes an image-image self-supervised objective as in the DINO line of work. This indicates intra-class variation (left) and the gap between text and image embeddings (right) are no signs of misalignment brought by pure text-image training, but rather normal behavior. All models are ViT-B. Embeddings are sampled from ImageNet validation set.

Figure 6: Few-shot classification as in training-free VLM adaptation literature. Approaches that attempt to address the intra-modal misalignment issue through avoiding use of image-image similarities (Tip-X, Coder) [ 40 , 44 ] do not win against feature selection in APE [ 53 ] or a basic Linear Discriminant Analysis (LDA) [ 42 ] on image embeddings.

Model	Classifier	1-shot	2-shot	4-shot	8-shot	16-shot
CLIP	Prototype	43.4 (50.7)	55.3 (62.5)	63.8 (69.7)	69.6 (74.7)	73.5 (77.5)
CLIP	LDA	43.4 (50.7)	60.0 (62.5)	69.8 (70.8)	76.1 (76.5)	79.5 (79.6)
SigLIP	Prototype	57.3 (62.1)	68.6 (71.9)	76.3 (78.5)	80.5 (81.8)	82.5 (83.7)
SigLIP	LDA	57.3 (62.1)	71.0 (72.4)	79.0 (79.4)	83.3 (83.3)	85.3 (85.3)
SigLIP2	Prototype	58.0 (63.4)	69.7 (73.2)	77.0 (79.4)	80.8 (82.6)	83.0 (84.5)
SigLIP2	LDA	58.0 (63.4)	73.2 (74.5)	80.5 (81.0)	84.5 (84.7)	86.5 (86.5)

Table 2: Few-shot classification with no use of text prompts. Accuracy is therefore only based on image-to-image similarities. This allows directly to test the intra-modal alignment quality of models with inter-modal loss only (CLIP, SigLIP), both inter- and intra-modal losses (SigLIP2) and image-only models (DINO). Results on

PCA^{\leftarrow}

-projected features are in (

\cdot

) parenthesis. Two best models per classifier are underlined . SigLIP scores higher than DINO , indicating well-aligned image embeddings despite inter-modal training only. All models are ViT-B. Reported values are the average over the 11 datasets as in Fig. 6 .

	ViT	Method	$\langle\cdot,\cdot\rangle$	$\mathcal{R}$ Oxford	$\mathcal{R}$ Paris	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	Caltech	UCF101	ImageNet	Average
CLIP	B/32	Original	$\langle I,I\rangle$	42.4	74.0	24.9	31.2	62.5	14.5	28.3	49.3	33.6	34.6	77.7	46.2	21.6	41.6
		OTI	$\langle T,I\rangle$	43.0	70.3	28.0	37.5	62.6	14.4	31.9	47.2	34.7	36.3	79.9	48.6	23.8	42.9
		$\text{PCA}^{\leftarrow}$	$\langle I^{\leftarrow},I^{\leftarrow}\rangle$	51.4	80.9	34.6	47.7	70.3	16.1	34.0	53.8	43.0	40.0	83.3	53.3	28.6	49.0
	L/14	Original	$\langle I,I\rangle$	57.1	77.8	43.8	47.2	84.2	25.8	33.9	57.8	55.0	39.2	83.8	59.5	33.0	53.7
		OTI	$\langle T,I\rangle$	62.4	77.1	50.5	56.0	86.0	27.1	37.7	56.3	55.9	43.5	87.3	62.8	38.2	57.0
		$\text{PCA}^{\leftarrow}$	$\langle I^{\leftarrow},I^{\leftarrow}\rangle$	64.5	83.0	57.2	62.7	88.9	28.7	39.9	62.8	64.4	46.0	89.5	66.8	42.2	61.3

Table 3: Image-to-image retrieval results (mAP) following [ 26 ] . Rows with

\langle I,I\rangle

\langle I^{\leftarrow},I^{\leftarrow}\rangle

measure intra-modal similarities. We show the performance gain brought by OTI [ 26 ] - an image-to-text conversion attempt motivated by the hypothesis that only

\langle T,I\rangle

is aligned - can be significantly surpassed through simply measuring

\langle I^{\leftarrow},I^{\leftarrow}\rangle

along the main axes of variance of ImageNet class names. This trend is no different between models without intra-modal and with intra-modal training objectives.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Distribution of CLIP embeddings: text (orange), original image (purple), projected image (blue). UMAP (top), PCA (middle), t-SNE (bottom). Left column: projection via Eq. 11 . Right column: mean-adjusted projection via Eq. 12 . Interpretation : (i) Like many previous studies noted, text embeddings and image embeddings lie in two different cones, such that we can find them clearly separated in the figure. We argued in the main paper this “modality gap” does not lead to an intra-modal misalignment. (ii) Projecting via $PCA^{\leftarrow}$ and re-projecting introduces a global shift (see left PCA) in $\mathbb{R}^{512}$ that is orthogonal to the principal components $\in\mathbb{R}^{256}$ . (iii) For comparison in $\mathbb{R}^{512}$ , we can compensate for this shift by adding back the mean of the canceled components (right column). ViT-B/16, ImageNet val.

Panel of Figure 8 — Figure 8: Modality gap: In the original CLIP space, image-image cosine similarities are high (left). With the projection from Eq. 11 , these similarities seem to decrease (middle). Applying mean adjustment as in Eq. 12 removes this effect (right). Interpretation : Because mean adjustment preserves Euclidean distances, the observed changes arises solely from normalization: the projection zeros out components, reducing vector norms such that embeddings lie no more on, but inside the unit hypersphere. Normalization back on the hypersphere then squeezes the projected embeddings apart, leading to a lower cosine similarity value range (middle) compared to original CLIP (left). Adding back the canceled mean before normalization avoids this phenomenon (right). At the same time, there is no significant performance change between (middle) and (right), which once more illustrates that such cosine similarity histograms are insufficient indicators of alignment quality. ViT-B/16, ImageNet validation set.

Figure 9: Captions generated with CLIP Prefix Captioning (Mokady et al . 2021), CLIP ViT-B/32, GPT-2. Swapping out the original CLIP image embedding $x_{i}$ for our projected $x_{i}^{\leftarrow}$ , we can observe the captions still cover the main objects. Samples are from datasets used for retrieval and few-shot classification in the main paper, left to right: SUN, ROxford, ImageNet.

Figure 10: The same experiment as in Fig. 9 , but with the street scene dataset BDD100k used in the ablation in Sec. 4.3 of the main paper to validate our interpretation that the projection increases the “classness” of the image embedding, thereby being beneficial for classification-like tasks, but harmful for tasks such as daytime recognition because some information is lost. In line with the finding of the main paper, here we can see how the captions generated with $PCA^{\leftarrow}$ ignore that it is night (left, middle) and focus on the main objects only (right).

Multimodal Contrastive Learning

Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.

See all 246 papers

IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Mar 20, 2026Simone Magistri, Dipam Goswami, Marco Mistretta +3Latent Space Alignment Modality Gap in VLMs
Media Integration and Communication Center (MICC), University of Florence, Italy · Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain +1
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMs Multimodal Embedding
Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel
AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization
Jul 15, 2026Yiyang Yao, Shanglin Liu, Jianming Lv +4Vision-Language Alignment Multimodal Contrastive Learning
School of Computer Science and Technology, South China University of Technology, Guangzhou, China

Mar 20, 2026J/K move · Enter open · S save

IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment

Simone Magistri, Dipam Goswami, Marco Mistretta, Bartłomiej Twardowski+2

Media Integration and Communication Center (MICC), University of Florence, Italy · Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain+1

Latent Space Alignment Modality Gap in VLMs Image Retrieval Vision-Language Pretraining

Open paper AskPDF arXiv

Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: https://github.com/simomagi/IsoCLIP.

Abstract

Multimodal Contrastive Learning

Explore similar work

IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment

Is the Modality Gap a Bug or a Feature? A Robustness Perspective

AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization