IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment

Simone Magistri; Dipam Goswami; Marco Mistretta; Bartłomiej Twardowski; Joost van de Weijer; Andrew D. Bagdanov

cs.CVMar 20, 2026

IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment

Authors: Simone Magistri, Dipam Goswami, Marco Mistretta, Bartłomiej Twardowski, Joost van de Weijer, Andrew D. Bagdanov

Organizations: Media Integration and Communication Center (MICC), University of Florence, Italy · Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain · IDEAS Research Institute, Warsaw, Poland

Abstract

Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: https://github.com/simomagi/IsoCLIP.

Figures & tables

Figure 1: Overview of intra-modal retrieval with CLIP. (a) The standard approach simply compares the cosine similarities computed after applying projector $W_{i}$ to query and gallery image embeddings, which is sub-optimal due to intra-modal misalignment . (b) To circumvent misalignment, inversion approaches [ 25 ] convert the query image embeddings to text embeddings by iteratively optimizing pseudo-tokens – an expensive operation that incurs high latency – and then computes inter-modal cosine similarities for retrieval. (c) We identify an inter-modal operator $\Psi=W_{i}^{\top}W_{t}$ fundamental to CLIP cosine similarity computations. We propose IsoCLIP, which uses only an isotropic region of the spectrum of $\Psi$ to align the projector weights along well-aligned directions between modalities. Then these aligned projectors are used to map the query and gallery embeddings. IsoCLIP exploits the properties of the CLIP projectors and does not add any latency to process while yielding more optimal intra-modal cosine similarities and significantly improved intra-modal performance.

Figure 2: Spectra of the inter-modal operator

\Psi=W_{i}^{\top}W_{t}

for CLIP ViT-B/16 and ViT-B/32 with OpenAI and DataComp (OpenCLIP) pre-training. Despite variations across models, all spectra show pronounced anisotropy in the extreme top and bottom singular directions, while staying relatively flat in the middle band.

Figure 3: Investigation of different regions of the spectrum (top 50, middle 50, and bottom 50 directions) of the inter-modal operator

\Psi

for aligning the CLIP Projector weights, as defined in Eq. 10 , utilizing the ViT-B/16 model for both image-to-image and text-to-text retrieval tasks. (a) Analysis of cosine similarity distributions showing very high similarities for the top band, and more well-behaved distributions for middle and bottom bands. (b) Overlap between cosine similarities of positive and negative pairs showing better separation for the middle band but highly overlapping distributions for top and bottom bands, implying higher intra-modal misalignment. (c) Performance comparison showing far superior performance using the well-aligned middle band compared to top and bottom bands.

Method	Intra-modal	Backbone	Latency (ms)	Caltech	CUB	ROxford	RParis	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Avg
Image-Image	✓	ViT-B/32	7 $\pm$ 1	77.1	22.9	42.6	67.9	24.6	30.5	62.0	14.5	28.1	47.9	32.3	34.3	47.1	40.9
OTI (I $\rightarrow$ T)	✗		1879 $\pm$ 35	79.9	24.6	43.0	70.3	28.0	37.5	62.6	14.4	31.9	47.2	34.7	36.3	48.6	43.0
IsoCLIP	✓		7 $\pm$ 1	80.8	27.0	47.2	73.8	30.0	40.8	66.5	14.9	30.9	51.5	38.0	36.4	48.4	45.1
Image-Image	✓	ViT-B/16	6 $\pm$ 1	80.6	31.6	46.6	75.3	31.0	36.3	70.8	19.0	30.7	51.2	42.8	35.9	49.8	46.3
OTI (I $\rightarrow$ T)	✗		1856 $\pm$ 56	83.5	33.9	49.9	77.4	37.2	42.9	72.8	20.1	35.1	50.5	47.5	38.7	52.6	49.4
IsoCLIP	✓		6 $\pm$ 1	85.0	38.6	51.8	82.0	41.2	50.7	77.4	20.5	36.0	55.6	53.5	38.5	55.4	52.8

Table 1: Image-to-image retrieval performance for multiple CLIP models on 13 datasets. We also report the query latency for all methods.

Method	Intra-modal	Backbone	Latency (ms)	COCO	Flickr30k	nocaps	Avg
Text-Text	✓	ViT-B/32	6 $\pm$ 1	26.2	51.7	35.1	37.7
OVI (T $\rightarrow$ I)	✗		11549 $\pm$ 192	28.3	54.8	38.8	40.6
IsoCLIP	✓		6 $\pm$ 1	29.1	56.8	39.6	41.9
Text-Text	✓	ViT-B/16	6 $\pm$ 1	26.1	50.7	35.1	37.3
OVI (T $\rightarrow$ I)	✗		11929 $\pm$ 125	28.2	53.1	38.3	39.9
IsoCLIP	✓		6 $\pm$ 1	28.9	55.2	39.4	41.2

Table 2: Text-to-text retrieval performance for multiple CLIP models on three datasets.

Method	Intra-modal	Classifier	Backbone	Caltech	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Average
Image-Text	✗	Zero-Shot	ViT-B/32	91.2	60.4	87.5	67.0	19.1	43.6	45.2	80.5	62.0	62.0	61.9
Image-Image	✓	NCM		91.9	67.0	77.1	93.7	34.5	64.4	77.7	77.3	70.2	76.6	73.0
IsoCLIP	✓	NCM		93.0	73.0	84.0	93.8	35.0	66.8	80.6	78.6	70.5	77.8	75.3
Image-Text	✗	Zero-Shot	ViT-B/16	92.9	65.3	89.1	71.4	24.8	44.4	47.8	86.1	62.6	66.7	65.1
Image-Image	✓	NCM		94.1	74.8	84.2	96.2	43.6	65.8	79.2	84.0	72.1	80.2	77.4
IsoCLIP	✓	NCM		95.4	81.2	90.9	97.1	45.6	68.4	82.6	85.6	72.0	81.7	80.1

Table 3: Image classification performance for multiple CLIP models on 10 datasets. We compare IsoCLIP with intra-modal NCM classification. Zero-shot results are reported only for reference, as they do not require training images to compute class prototypes.

Method	Backbone	Caltech	CUB	ROxford	RParis	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Average
Image-Image	ViT-B/16	80.6	31.6	46.6	75.3	31.0	36.3	70.8	19.0	30.7	51.2	42.8	35.9	49.8	46.3
Image-Image [Pre]		81.6	32.7	49.2	77.8	32.0	39.0	72.4	19.4	32.5	54.5	45.3	38.8	54.2	48.4
$W_{i}^{\text{white}}$		82.1	34.4	48.5	78.6	34.2	40.8	73.7	19.5	32.9	54.3	46.5	38.2	53.9	49.0
IsoCLIP		85.0	38.6	51.8	82.0	41.2	50.7	77.4	20.5	36.0	55.6	53.5	38.5	55.4	52.8
Image-Image	ViT-L/14	83.2	43.0	57.5	76.9	43.3	47.3	84.0	25.8	34.1	59.0	53.0	39.1	60.0	54.3
Image-Image [Pre]		85.4	42.6	59.5	78.1	41.9	48.0	84.7	25.9	35.3	61.7	54.6	42.0	61.9	55.5

Table 4: Ablation Study. We compare IsoCLIP against using pre-projection image features for retrieval (Image–Image [Pre]) and against whitening the CLIP image projection weights (

W_{i}^{\text{white}}

Figure 4: Analysis on Dogs vs. Cats [ 7 ] . (left) IsoCLIP achieves higher precision than CLIP as the number of retrieved dog images

K

increases. (center) CLIP shows significant overlap between intra-class (dog-dog) and inter-class (dog-cat) similarities due to intra-modal misalignment. (right) IsoCLIP reduces this overlap, making image-image similarities more discriminative.

Figure 5: Ablation showing the impact of varying the

k_{t}

and

k_{b}

values for the isotropic middle band selection on Caltech101 (image-to-image retrieval) and COCO (text-to-text retrieval).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S6 : Spectrum of the intra-modal operator in CLIP (

\Sigma_{i}^{2}

) and after applying IsoCLIP (

\hat{\Sigma}_{i}^{2}

) . (Top) IsoCLIP truncates the spectrum of the intra-modal operator according to the retained subspace defined by the middle-band of the inter-modal operator (Eq. ( S9 )), resulting in a lower-rank operator. (Bottom) The normalized eigenvalues (obtained by dividing each eigenvalue by the sum of the spectrum) reveal that, after applying IsoCLIP, the intra-modal operator is distributed across more directions than in standard CLIP.

Figure S7 : Cosine similarity distribution in image-to-image retrieval for positive and negative pairs on the CUB dataset for CLIP (left) and IsoCLIP (right). Positives are obtained by computing the similarity of each image with all images in the gallery that share the same label, while negatives correspond to the cosine similarity with images having different labels. We observe that the CLIP cosine similarities are concentrated in a narrower range with higher mean values, while IsoCLIP, by weighting more directions in cosine similarity computations, spreads the distribution, shifts the mean similarities to lower values and increases positive and negative separation.

Figure S8 : Image-image retrieval mAP performance obtained by iteratively adding top and bottom directions to the 50 middle-band directions when computing IsoCLIP on CUB-2011 dataset using the ViT-B/16.

	Number of Pseudo-Patches $P$
Backbone	1	2	4	8	16
ViT-B/16	52.8	52.9	53.1	51.9	50.8
ViT-B/16-open	60.2	59.1	58.0	57.3	57.2

Appendix

Table S5 : Ablation study on the number of OVI pseudo-patches

P

for text-to-text retrieval on Flickr30K validation set. The highest mAP score for each model is highlighted in bold, with the corresponding value of

P

used in all experiments.

Method	Intra-modal	Backbone	Caltech	CUB	ROxford	RParis	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Avg
Image-Image	✓	ViT-B/32-open	82.3	32.1	50.8	74.7	46.7	44.1	77.0	19.6	36.9	56.4	39.6	36.2	45.7	49.4
OTI (I $\rightarrow$ T)	✗		83.3	34.3	54.4	75.8	50.5	50.5	78.0	20.1	40.9	54.5	42.9	37.8	48.2	51.6
IsoCLIP	✓		83.4	34.2	56.8	75.8	49.9	47.8	78.2	19.8	37.6	57.7	41.5	36.6	46.3	51.2
Image-Image	✓		86.9	55.9	52.1	78.3	49.4	55.4	91.5	24.9	35.8	61.2	55.5	41.1	57.0	57.3
IsoCLIP	✓	EVA-02 B/16	89.7	58.3	53.0	80.4	55.2	62.8	92.7	25.7	40.0	62.3	57.5	42.2	57.9	59.8
Image-Image	✓		89.2	38.1	53.2	76.5	70.8	56.6	89.3	41.8	39.0	50.8	59.2	43.6	59.2	59.0

Appendix

Table S6 : Image-to-image retrieval performance using OpenCLIP ViT-B/32, pre-trained on DataComp dataset, EVA-02 B/16, pretrained on Merged-2B [ 8 ] and SigLIP2 B/16 pre-trained on WebLI [ 42 ] .

Method	Intra-modal	Classifier	Backbone	Caltech	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Average
Image-Text	✗	Zero-Shot	ViT-B/16-open	96.9	89.8	92.8	75.3	29.8	58.3	53.4	87.5	69.8	67.8	72.1
Image-Image	✓	NCM		96.7	90.6	88.9	98.6	54.0	74.5	84.4	86.7	74.9	79.9	82.9
IsoCLIP	✓	NCM		97.2	91.4	90.6	98.8	54.5	75.9	85.5	87.1	74.3	80.9	83.6

Appendix

Table S7 : Image classification performance using OpenCLIP ViT-B/16, pre-trained on DataComp dataset, on 10 datasets. We compare IsoCLIP with intra-modal NCM classification and zero-shot classification.

Method	Backbone	Places365	iNat	Avg
Image-Image		16.72	9.61	13.17
IsoCLIP	PE-Core B/16	17.04	13.07	15.06
Image-Image		16.17	7.56	11.87
IsoCLIP	SigLIP2 B/16	17.78	8.43	13.11
Image-Image		14.54	10.03	12.29
IsoCLIP	EVA-02 B/16	15.01	11.23	13.12

Appendix

Table S8 : Experiments on Places365 & iNaturalist.

Method	Backbone	Caltech	CUB	ROxford	RParis	Cars	Pets	Flowers	Aircraft	DTD	EuroSAT	Food101	SUN397	UCF101	Average
Image-Image	ViT-B/32	77.1	22.9	42.6	67.9	24.6	30.5	62.0	14.5	28.1	47.9	32.3	34.3	47.1	40.9
Image-Image [Pre]		78.3	23.3	44.9	70.5	24.5	32.7	63.4	14.4	29.5	50.8	34.2	36.0	49.7	42.5
$W_{i}^{\text{white}}$		78.4	24.5	45.0	70.7	25.9	33.2	63.7	14.3	29.4	50.3	34.2	35.1	48.9	42.6
IsoCLIP		80.8	27.0	47.2	73.8	30.0	40.8	66.5	14.9	30.9	51.5	38.0	36.4	48.4	45.1
Image-Image	ViT-B/32-open	82.3	32.1	50.8	74.7	46.7	44.1	77.0	19.6	36.9	56.4	39.6	36.2	45.7	49.4
Image-Image [Pre]		83.6	31.6	52.8	73.9	46.1	44.6	76.5	19.5	37.3	57.5	39.9	36.9	46.9	49.8

Appendix

Table S9 : We compare IsoCLIP against using pre-projection image features for retrieval (Image–Image [Pre]) and against whitening the CLIP image projection weights (

W_{i}^{\text{white}}

), using ViT-B32 OpenAI and Open ViT-B32 and ViT-B16 pre-trained on Datacomp dataset.

Task	Backbone	$k_{t}$	$k_{b}$
Image-Image	ViT-B/32	150	50
	ViT-B/32-open	50	50
	ViT-B/16	200	50
	ViT-L/14	250	200
	ViT-B/16-open	100	100
	EVA-02 B/16	150	0

Appendix

Table S10 : Selection of

k_{t}

and

k_{b}

for image-to-image and text-to-text retrieval using IsoCLIP. For image classification tasks, we use the same values selected for image-to-image retrieval.

Figure S9 : Analysis showing the impact of varying

k_{t}

and

k_{b}

for the middle band selection across datasets for text-to-text retrieval. The values used in our reported results based on selection from COCO are denoted with stars.

Panel of Figure 20 — Figure S10 : Analysis showing the impact of varying $k_{t}$ and $k_{b}$ for the middle band selection across datasets for image-to-image retrieval. The values used in our reported results based on selection from Caltech101 are denoted with stars.

Latent Space Alignment

Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.

See all 76 papers

Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP
Mar 17, 2026Jonas Herzog, Yue WangMultimodal Contrastive Learning VLM Evaluation
Zhejiang University
The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal Alignment Modality Gap in VLMs
University of Electronic Science and Technology of China · University of Bristol
Canonicalizing Multimodal Contrastive Representation Learning
Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal Embedding Representation Alignment
Aalto University, YaiYai Ltd

Mar 17, 2026J/K move · Enter open · S save

Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

Jonas Herzog, Yue Wang

Zhejiang University

Multimodal Contrastive Learning VLM Evaluation

Open paper AskPDF arXiv

Recent research suggested that the embeddings produced by CLIP-like contrastive language-image training are suboptimal for image-only tasks. The main theory is that the inter-modal (language-image) alignment loss ignores intra-modal (image-image) alignment, leading to poorly calibrated distances between images. In this study, we question this intra-modal misalignment hypothesis. We reexamine its foundational theoretical argument, the indicators used to support it, and the performance metrics affected. For the theoretical argument, we demonstrate that there are no such supposed degrees of freedom for image embedding distances. For the empirical measures, our findings reveal they yield similar results for language-image trained models (CLIP, SigLIP) and image-image trained models (DINO, SigLIP2). This indicates the observed phenomena do not stem from a misalignment specific to the former. Experiments on the commonly studied intra-modal tasks retrieval and few-shot classification confirm that addressing task ambiguity, not supposed misalignment, is key for best results.

Abstract

Latent Space Alignment

Explore similar work

Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Canonicalizing Multimodal Contrastive Representation Learning