cs.LGFeb 19, 2026

Canonicalizing Multimodal Contrastive Representation Learning

Authors: Sharut Gupta, Sanyam Kansal, Stefanie Jegelka, Phillip Isola, Vikas Garg

Organizations: Aalto University, YaiYai Ltd

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders (f,g)(f, g) and (f~,g~)(\widetilde{f},\widetilde{g})) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map QQ where Q⊤Q=IQ^\top Q = I such that f~(x)≈Qf(x)\widetilde{f}(x)\approx Q f(x) for paired images xx. Strikingly, the same QQ simultaneously aligns the text encoders i.e., g~(y)≈Qg(y)\widetilde{g}(y)\approx Q g(y) for texts yy. Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e. ⟨f(x),g(y)⟩≈⟨f~(x),g~(y)⟩\langle f(x), g(y)\rangle \approx \langle \widetilde{f}(x), \widetilde{g}(y)\rangle, then the two models must be related by a single orthogonal map QQ and the same QQ maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/

Figures & tables

Figure 1: A single orthogonal map aligns two independent contrastive models across modalities. (a) Illustrative PCA schematic on synthetic embeddings from two models (A and B) shows that the spaces are a priori incomparable; (b) A single orthogonal map Q\mathcal{Q} , fit only using image embeddings of CLIP (OpenAI) and CLIP (LAION), almost perfectly aligns image embeddings of the two models (c, left) and simultaneously aligns their text embeddings, as evidenced by a large gain in text-text pointwise cosine similarity (c, right).
Figure 2: (left) PCA visualization of embeddings from Clip (OpenAI) on CIFAR 100, showing a large modality gap. (right) Applying a translation to close this gap distorts the embedding geometry and significantly degrades classification accuracy.
Figure 3: (a) Across CLIP variants, the multimodal kernel ⟨f(x),g(y)⟩\langle f(x),g(y)\rangle (relative angles between image and text embeddings) is strongly preserved (dashed lines), unlike the unimodal kernel ⟨f(x),f~(x′)⟩\langle f(x),\tilde{f}(x^{\prime})\rangle ; (b) CKA on multimodal kernels shows high alignment across models.
Figure 4: Inter-model alignment on Oxford-Pets before and after fitting a single orthogonal map Q\mathcal{Q} from paired images. (a) Image-image cosine similarity; (b) Text-text cosine similarity; (c) Aligned Image and Aligned text class retrieval accuracy. Here, Model A and Model B denote each model’s within-model image-to-text baseline. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text cosine from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 5
Figure 7: Comparison of alignment strategies on Oxford Pets across model pairs in terms of (a) image cosine similarity; (b) text cosine similarity, and (c) aligned-image-to-aligned-text accuracy. Unlike orthogonal maps, linear and non-linear maps distort task-relevant geometry and do not transfer to text, leading to poor downstream performance.
Figure 8: Qualitative k-NN retrieval under two alignment paths for Oxford Pets. We compare two routes from a source image xx into the target model’s image space after alignment by Q\mathcal{Q} : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.
Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: High-level overview of our theoretical results. Background results establish multimodal kernel alignment; additional assumptions progressively strengthen this to linear, then isometric, and finally task-level (classification) alignment.
DatasetClassesTrainValTest
Caltech-101 [ Fei-Fei et al., 2004 ]1004,1281,6492,465
Oxford Pets [ Parkhi et al., 2012 ]372,9447363,669
CIFAR100 [ Krizhevsky et al., 2009 ]10050,000-10,000
STL10 [ Coates et al., 2011 ]1050,000-8,000
DTD [ Cimpoi et al., 2014 ]472,8201,1281,692
Appendix
Table 1: Detailed statistics of the 5 datasets for image classification.
DatasetText Templates (from official CLIP repo)
Oxford Petsa photo of a {class}, a type of pet.
CIFAR-100a photo of a {class}. a blurry photo of a {class}. a black and white photo of a {class}. a low contrast photo of a {class}. a high contrast photo of a {class}. a bad photo of a {class}. a good photo of a {class}. a photo of a small {class}. a photo of a big {class}. a photo of the {class}. a blurry photo of the {class}. a black and white photo of the {class}. a low contrast photo of the {class}. a high contrast photo of the {class}. a bad photo of the {class}. a good photo of the {class}. a photo of the small {class}. a photo of the big {class}.
Caltech-101a photo of a {class}. a painting of a {class}. a plastic {class}. a sculpture of a {class}. a sketch of a {class}. a tattoo of a {class}. a toy {class}. a rendition of a {class}. a embroidered {class}. a cartoon {class}. a {class} in a video game. a plushie {class}. a origami {class}. art of a {class}. graffiti of a {class}. a drawing of a {class}. a doodle of a {class}. a photo of the {class}. a painting of the {class}. the plastic {class}. a sculpture of the {class}. a sketch of the {class}. a tattoo of the {class}. the toy {class}. a rendition of the {class}. the embroidered {class}. the cartoon {class}. the {class} in a video game. the plushie {class}. the origami {class}. art of the {class}. graffiti of the {class}. a drawing of the {class}. a doodle of the {class}.
STL-10a photo of a {class}. a photo of the {class}.
DTDa photo of a {class} texture. a photo of a {class} pattern. a photo of a {class} thing. a photo of a {class} object. a photo of the {class} texture. a photo of the {class} pattern. a photo of the {class} thing. a photo of the {class} object.
Appendix
Table 2: Text templates used to construct class-level text prototypes for each dataset. All templates are taken from the official CLIP repository. For all reported results, we use a single template per class for evaluation
Figure 10: PCA visualization in 2D of generated embeddings of CIFAR-100 from pre-trained models showing pervasive modality gap in multi-modal contrastive representation learning whereby image and text embeddings are located in two completely separate regions of the embedding space.
Figure 11: Across CLIP variants, the multimodal kernel ⟨f(x),g(y)⟩\langle f(x),g(y)\rangle (relative angles between image and text embeddings) is strongly preserved (dashed lines), unlike the unimodal kernel ⟨f(x),f~(x′)⟩\langle f(x),\tilde{f}(x^{\prime})\rangle ;
Figure 12: Cross-model alignment on Oxford Pets across independently trained contrastive models before and after fitting a single orthogonal map R\mathcal{R} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 13: Cross-model alignment on Caltech-101 across independently trained contrastive models before and after fitting a single orthogonal map R\mathcal{R} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 14: Cross-model alignment on CIFAR-100 across independently trained contrastive models before and after fitting a single orthogonal map R\mathcal{R} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 15: Cross-model alignment on STL10 across independently trained contrastive models before and after fitting a single orthogonal map R\mathcal{R} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 16: Cross-model alignment on DTD across independently trained contrastive models before and after fitting a single orthogonal map R\mathcal{R} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. Q\mathcal{Q} aligns images across models with a single orthogonal map, and the same Q\mathcal{Q} learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.
Figure 17: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 18: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 19: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 20: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 21: Generalization of orthogonal alignment under limited supervision (CIFAR-100; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 22: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 23: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 24: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 25: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 26: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 27: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 28: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. R\mathcal{R} learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.
Figure 29: Transfer of Q\mathcal{Q} learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-B/32 and ViT-B/16 (OpenAI). (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.
Figure 30: Transfer of Q\mathcal{Q} learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-B/32 (OpenAI) and ViT-B/32 (LAION) (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.
Figure 31: Transfer of Q\mathcal{Q} learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-L/14 (OpenAI) and FLAVA (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.
Figure 32: Transfer of Q\mathcal{Q} learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-L/14 (OpenAI) and SigLIP. (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.
Figure 33: Cross-model alignment on Caltech-101 learned from text prototypes instead of images, before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 34: Cross-model alignment on CIFAR-100 learned from text prototypes alone before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 35: Cross-model alignment on Oxford Pets learned from text prototypes alone before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 36: Cycle consistency on Caltech-101. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 37: Compositionality on Caltech-101 by comparing QBC∘QABQ_{BC}\circ Q_{AB} with QACQ_{AC} . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 38: Cycle consistency on CIFAR-100. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 39: Compositionality on CIFAR-100 by comparing QBC∘QABQ_{BC}\circ Q_{AB} with QACQ_{AC} . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 40: Cycle consistency on Oxford Pets. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 41: Compositionality on Oxford Pets by comparing QBC∘QABQ_{BC}\circ Q_{AB} with QACQ_{AC} . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.
Figure 42: Cross-model alignment across differing embedding dimensions (Caltech-101; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the 512512 -dimensional source manifold to be isometrically embedded into the 768768 -dimensional target space, maintaining high cross-modal retrieval accuracy.
Figure 43: Cross-model alignment across differing embedding dimensions (CIFAR-100; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the 512512 -dimensional source manifold to be isometrically embedded into the 768768 -dimensional target space, maintaining high cross-modal retrieval accuracy.
Figure 44: Cross-model alignment across differing embedding dimensions (Oxford Pets; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the 512512 -dimensional source manifold to be isometrically embedded into the 768768 -dimensional target space, maintaining high cross-modal retrieval accuracy.
Figure 45: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on Caltech-101 across model pairs. (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).
Figure 46: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on CIFAR-100 across model pairs. (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).
Figure 47: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on Oxford Pets across model pairs (CLIP OpenAI vs. LAION, FLAVA, and SigLIP). (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).
Figure 48: Comparison between an orthogonal map with and without mean adjustment across model pairs on Caltech-101. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).
Figure 49: Comparison between an orthogonal map with and without mean adjustment across model pairs on CIFAR-100. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).
Figure 50: Comparison between an orthogonal map with and without mean adjustment across model pairs on Oxford Pets. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).
Figure 51: Cross-model alignment between identical training datasets with differing architectures (Caltech-101; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying Q\mathcal{Q} yields substantially stronger modality transfer than in settings where the training distributions differ.
Figure 52: Cross-model alignment between identical training datasets with differing architectures (CIFAR-100; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying Q\mathcal{Q} yields substantially stronger modality transfer than in settings where the training distributions differ.
Figure 53: Cross-model alignment between identical training datasets with differing architectures (Oxford Pets; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map Q\mathcal{Q} . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying Q\mathcal{Q} yields substantially stronger modality transfer than in settings where the training distributions differ.
Figure 54: Qualitative k-NN retrieval under two alignment paths for Oxford Pets. We compare two routes from a source image xx into the target model’s image space after alignment by Q\mathcal{Q} : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent target-space neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.
Figure 55: Qualitative k-NN retrieval under two alignment paths for Caltech-101. We compare two routes from a source image xx into the target model’s image space after alignment by Q\mathcal{Q} : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent target-space neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.
Figure 56: Fine-grained semantic preservation under controlled text edits on Oxford Pets. After changing only the class token in the caption (Birman to Russian Blue) while keeping descriptive attributes fixed, alignment via Q\mathcal{Q} retrieves target-model images that match the edited class while preserving caption-specific visual cues (e.g., eye prominence and facial structure), indicating preservation of fine-grained semantics beyond class identity.
Figure 57: Fine-grained semantic preservation under controlled text edits on Oxford Pets. After changing only the class token in the caption (American Bulldog to Basset Hound) while keeping descriptive attributes fixed, alignment via Q\mathcal{Q} retrieves target-model images that match both the edited class and the original attribute cues (e.g., muscular build and posture), indicating preservation of fine-grained semantics beyond class identity.
More in this topic

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 159 papers

Explore similar work

CardsList
Aug 17, 2026

Expressivity In Multimodal Contrastive Learning

Contrastive learning has become a cornerstone of modern representation learning, powering CLIP-style models that underpin text-to-image generation, vision-language models, and retrieval across a rapidly growing range of modalities. Despite this empirical success, the expressive power of these architectures remains poorly understood. To gain insight, we study expressivity by adopting a population-level, density-estimation viewpoint: each architecture comprises a parameterized set of densities whose parameters may be chosen to approximate the joint distribution of the modalities. This isolates a question of pure representational capacity: which joint distributions can a given contrastive family of parameterizations approximate to arbitrary accuracy? We show that expressivity is sharply architecture-dependent. For two modalities, the simple two-tower CLIP architecture is a universal approximator. A natural generalization of CLIP, widely used in practice when three or more modalities are present, is based on a loss found by summing over all pairwise similarities. This provably cannot represent arbitrary joint distributions, although we prove that it remains expressive enough to match all pairwise conditionals. Motivated by this gap, we propose Hadamard-CLIP, which adds a single learned weight vector on top of the existing encoders and restores universal approximation of the joint for any number of modalities while preserving CLIP's fast, precomputable-embedding retrieval.

Andrew Stuart, Florian Wolf

California Institute of Technology

Multimodal Contrastive LearningNeural Network ExpressivityUniversal Approximation
AskPDF
Oct 23, 2026

Closing the Modality Gap via Simplex-Constrained Representations

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, Sd−1S^{d-1}, to the probability simplex, Δd−1Δ^{d-1}. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid ℓ2\ell_2 gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

Shubham Gupta, Siva Reddy, Perouz Taslakian +2

Mila – Québec AI Institute · ServiceNow Research · Université Laval +1

Modality Gap in VLMsCross-Modal AlignmentRepresentation LearningMultimodal Contrastive LearningCross-Modal Retrieval
AskPDF
Mar 30, 2026

Is the Modality Gap a Bug or a Feature? A Robustness Perspective

Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.

Rhea Chowers, Oshri Naparstek, Udi Barzelay +1

Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel

Modality Gap in VLMsMultimodal EmbeddingVLM RobustnessMultimodal Contrastive Learning
AskPDF