Canonicalizing Multimodal Contrastive Representation Learning

Sharut Gupta; Sanyam Kansal; Stefanie Jegelka; Phillip Isola; Vikas Garg

cs.LGFeb 19, 2026

Canonicalizing Multimodal Contrastive Representation Learning

Authors: Sharut Gupta, Sanyam Kansal, Stefanie Jegelka, Phillip Isola, Vikas Garg

Organizations: Aalto University, YaiYai Ltd

Abstract

As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders $(f, g)$ and $(\widetilde{f},\widetilde{g})$ ) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map $Q$ where $Q^\top Q = I$ such that $\widetilde{f}(x)\approx Q f(x)$ for paired images $x$ . Strikingly, the same $Q$ simultaneously aligns the text encoders i.e., $\widetilde{g}(y)\approx Q g(y)$ for texts $y$ . Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e. $\langle f(x), g(y)\rangle \approx \langle \widetilde{f}(x), \widetilde{g}(y)\rangle$ , then the two models must be related by a single orthogonal map $Q$ and the same $Q$ maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/

Figures & tables

Figure 1: A single orthogonal map aligns two independent contrastive models across modalities. (a) Illustrative PCA schematic on synthetic embeddings from two models (A and B) shows that the spaces are a priori incomparable; (b) A single orthogonal map

\mathcal{Q}

, fit only using image embeddings of CLIP (OpenAI) and CLIP (LAION), almost perfectly aligns image embeddings of the two models (c, left) and simultaneously aligns their text embeddings, as evidenced by a large gain in text-text pointwise cosine similarity (c, right).

Figure 2: (left) PCA visualization of embeddings from Clip (OpenAI) on CIFAR 100, showing a large modality gap. (right) Applying a translation to close this gap distorts the embedding geometry and significantly degrades classification accuracy.

Figure 3: (a) Across CLIP variants, the multimodal kernel

\langle f(x),g(y)\rangle

(relative angles between image and text embeddings) is strongly preserved (dashed lines), unlike the unimodal kernel

\langle f(x),\tilde{f}(x^{\prime})\rangle

; (b) CKA on multimodal kernels shows high alignment across models.

Figure 4: Inter-model alignment on Oxford-Pets before and after fitting a single orthogonal map

\mathcal{Q}

from paired images. (a) Image-image cosine similarity; (b) Text-text cosine similarity; (c) Aligned Image and Aligned text class retrieval accuracy. Here, Model A and Model B denote each model’s within-model image-to-text baseline.

\mathcal{Q}

aligns images across models with a single orthogonal map, and the same

\mathcal{Q}

learned only from image embeddings transfers to text, boosting text-text cosine from near-chance to near-oracle, all while preserving strong image classification accuracy.

Figure 7: Comparison of alignment strategies on Oxford Pets across model pairs in terms of (a) image cosine similarity; (b) text cosine similarity, and (c) aligned-image-to-aligned-text accuracy. Unlike orthogonal maps, linear and non-linear maps distort task-relevant geometry and do not transfer to text, leading to poor downstream performance.

Figure 8: Qualitative k-NN retrieval under two alignment paths for Oxford Pets. We compare two routes from a source image $x$ into the target model’s image space after alignment by $\mathcal{Q}$ : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: High-level overview of our theoretical results. Background results establish multimodal kernel alignment; additional assumptions progressively strengthen this to linear, then isometric, and finally task-level (classification) alignment.

Dataset	Classes	Train	Val	Test
Caltech-101 [ Fei-Fei et al., 2004 ]	100	4,128	1,649	2,465
Oxford Pets [ Parkhi et al., 2012 ]	37	2,944	736	3,669
CIFAR100 [ Krizhevsky et al., 2009 ]	100	50,000	-	10,000
STL10 [ Coates et al., 2011 ]	10	50,000	-	8,000
DTD [ Cimpoi et al., 2014 ]	47	2,820	1,128	1,692

Appendix

Table 1: Detailed statistics of the 5 datasets for image classification.

Dataset	Text Templates (from official CLIP repo)
Oxford Pets	a photo of a {class}, a type of pet.
CIFAR-100	a photo of a {class}. a blurry photo of a {class}. a black and white photo of a {class}. a low contrast photo of a {class}. a high contrast photo of a {class}. a bad photo of a {class}. a good photo of a {class}. a photo of a small {class}. a photo of a big {class}. a photo of the {class}. a blurry photo of the {class}. a black and white photo of the {class}. a low contrast photo of the {class}. a high contrast photo of the {class}. a bad photo of the {class}. a good photo of the {class}. a photo of the small {class}. a photo of the big {class}.
Caltech-101	a photo of a {class}. a painting of a {class}. a plastic {class}. a sculpture of a {class}. a sketch of a {class}. a tattoo of a {class}. a toy {class}. a rendition of a {class}. a embroidered {class}. a cartoon {class}. a {class} in a video game. a plushie {class}. a origami {class}. art of a {class}. graffiti of a {class}. a drawing of a {class}. a doodle of a {class}. a photo of the {class}. a painting of the {class}. the plastic {class}. a sculpture of the {class}. a sketch of the {class}. a tattoo of the {class}. the toy {class}. a rendition of the {class}. the embroidered {class}. the cartoon {class}. the {class} in a video game. the plushie {class}. the origami {class}. art of the {class}. graffiti of the {class}. a drawing of the {class}. a doodle of the {class}.
STL-10	a photo of a {class}. a photo of the {class}.
DTD	a photo of a {class} texture. a photo of a {class} pattern. a photo of a {class} thing. a photo of a {class} object. a photo of the {class} texture. a photo of the {class} pattern. a photo of the {class} thing. a photo of the {class} object.

Appendix

Table 2: Text templates used to construct class-level text prototypes for each dataset. All templates are taken from the official CLIP repository. For all reported results, we use a single template per class for evaluation

Figure 10: PCA visualization in 2D of generated embeddings of CIFAR-100 from pre-trained models showing pervasive modality gap in multi-modal contrastive representation learning whereby image and text embeddings are located in two completely separate regions of the embedding space.

Figure 11: Across CLIP variants, the multimodal kernel

\langle f(x),g(y)\rangle

(relative angles between image and text embeddings) is strongly preserved (dashed lines), unlike the unimodal kernel

\langle f(x),\tilde{f}(x^{\prime})\rangle

;

Panel of Figure 12 — Figure 12: Cross-model alignment on Oxford Pets across independently trained contrastive models before and after fitting a single orthogonal map $\mathcal{R}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. $\mathcal{Q}$ aligns images across models with a single orthogonal map, and the same $\mathcal{Q}$ learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.

Panel of Figure 13 — Figure 13: Cross-model alignment on Caltech-101 across independently trained contrastive models before and after fitting a single orthogonal map $\mathcal{R}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. $\mathcal{Q}$ aligns images across models with a single orthogonal map, and the same $\mathcal{Q}$ learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.

Panel of Figure 14 — Figure 14: Cross-model alignment on CIFAR-100 across independently trained contrastive models before and after fitting a single orthogonal map $\mathcal{R}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. $\mathcal{Q}$ aligns images across models with a single orthogonal map, and the same $\mathcal{Q}$ learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.

Panel of Figure 15 — Figure 15: Cross-model alignment on STL10 across independently trained contrastive models before and after fitting a single orthogonal map $\mathcal{R}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. $\mathcal{Q}$ aligns images across models with a single orthogonal map, and the same $\mathcal{Q}$ learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.

Panel of Figure 16 — Figure 16: Cross-model alignment on DTD across independently trained contrastive models before and after fitting a single orthogonal map $\mathcal{R}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Euclidean distance (without normalization) between image embeddings. (f) Euclidean distance (without normalization) between text embeddings. (g) Image–text retrieval using aligned images from model A and text from model B. (h) Image–text retrieval using images from model B and aligned text from model A. (i) Image–text retrieval using aligned images and aligned text from model A. $\mathcal{Q}$ aligns images across models with a single orthogonal map, and the same $\mathcal{Q}$ learned only from image embeddings transfers to text, boosting text-text retrieval from near-chance to near-oracle, all while preserving strong image classification accuracy.

Panel of Figure 17 — Figure 17: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 18 — Figure 18: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 19 — Figure 19: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 20 — Figure 20: Generalization of orthogonal alignment under limited supervision (Caltech-101; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 21 — Figure 21: Generalization of orthogonal alignment under limited supervision (CIFAR-100; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 22 — Figure 22: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 23 — Figure 23: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 24 — Figure 24: Generalization of orthogonal alignment under limited supervision (CIFAR100; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 25 — Figure 25: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-B/32 to CLIP ViT-B/16, OpenAI). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 26 — Figure 26: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-B/32 (OpenAI) to CLIP ViT-B/32 (LAION). (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 27 — Figure 27: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-L/14 (OpenAI) to FLAVA (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 28 — Figure 28: Generalization of orthogonal alignment under limited supervision (Oxford Pets; CLIP ViT-L/14 (OpenAI) to SigLIP. (a) Image-image class retrieval, (b) text-text class retrieval, and (c) mean text-text cosine similarity, each reported on seen and unseen classes. (d-f) Downstream transfer: (d) aligned images from model A with text from model B, (e) images from model B with aligned text from model A, and (f) aligned images with aligned text from model A. $\mathcal{R}$ learned from few classes transfers across modalities and generalizes to unseen classes, achieving near-oracle cross-model retrieval and classification.

Panel of Figure 29 — Figure 29: Transfer of $\mathcal{Q}$ learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-B/32 and ViT-B/16 (OpenAI). (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 30 — Figure 30: Transfer of $\mathcal{Q}$ learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-B/32 (OpenAI) and ViT-B/32 (LAION) (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 31 — Figure 31: Transfer of $\mathcal{Q}$ learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-L/14 (OpenAI) and FLAVA (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 32 — Figure 32: Transfer of $\mathcal{Q}$ learnt on Oxford Pets to Caltech-101 and vice-versa across CLIP ViT-L/14 (OpenAI) and SigLIP. (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 33 — Figure 33: Cross-model alignment on Caltech-101 learned from text prototypes instead of images, before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 34 — Figure 34: Cross-model alignment on CIFAR-100 learned from text prototypes alone before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 35 — Figure 35: Cross-model alignment on Oxford Pets learned from text prototypes alone before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval (c) Mean text-text cosine similarity. (d) Mean image-image cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 36 — Figure 36: Cycle consistency on Caltech-101. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 37 — Figure 37: Compositionality on Caltech-101 by comparing $Q_{BC}\circ Q_{AB}$ with $Q_{AC}$ . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 38 — Figure 38: Cycle consistency on CIFAR-100. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 39 — Figure 39: Compositionality on CIFAR-100 by comparing $Q_{BC}\circ Q_{AB}$ with $Q_{AC}$ . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 40 — Figure 40: Cycle consistency on Oxford Pets. (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 41 — Figure 41: Compositionality on Oxford Pets by comparing $Q_{BC}\circ Q_{AB}$ with $Q_{AC}$ . (a) Image–image class retrieval. (b) text–text class retrieval (c) Mean image-image cosine similarity. (c) Mean text-text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A.

Panel of Figure 42 — Figure 42: Cross-model alignment across differing embedding dimensions (Caltech-101; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the $512$ -dimensional source manifold to be isometrically embedded into the $768$ -dimensional target space, maintaining high cross-modal retrieval accuracy.

Panel of Figure 43 — Figure 43: Cross-model alignment across differing embedding dimensions (CIFAR-100; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the $512$ -dimensional source manifold to be isometrically embedded into the $768$ -dimensional target space, maintaining high cross-modal retrieval accuracy.

Panel of Figure 44 — Figure 44: Cross-model alignment across differing embedding dimensions (Oxford Pets; CLIP ViT-B/32 to CLIP ViT-L/14, OpenAI) before and after fitting an orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. Enforcing semi-orthogonality via Stiefel manifold projection allows the $512$ -dimensional source manifold to be isometrically embedded into the $768$ -dimensional target space, maintaining high cross-modal retrieval accuracy.

Panel of Figure 45 — Figure 45: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on Caltech-101 across model pairs. (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).

Panel of Figure 46 — Figure 46: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on CIFAR-100 across model pairs. (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).

Panel of Figure 47 — Figure 47: Comparison of alignment strategies (Non-Linear, Linear, and Orthogonal) on Oxford Pets across model pairs (CLIP OpenAI vs. LAION, FLAVA, and SigLIP). (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean image–image cosine similarity. (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Unlike linear and non-linear maps, orthogonal transformations preserve the underlying image-text geometry favorable for downstream tasks and transfers across modalities (text in this case).

Panel of Figure 48 — Figure 48: Comparison between an orthogonal map with and without mean adjustment across model pairs on Caltech-101. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).

Panel of Figure 49 — Figure 49: Comparison between an orthogonal map with and without mean adjustment across model pairs on CIFAR-100. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).

Panel of Figure 50 — Figure 50: Comparison between an orthogonal map with and without mean adjustment across model pairs on Oxford Pets. (a) Image–image class retrieval (b) text–text class retrieval (c) Mean image–image cosine similarity (d) Mean text–text cosine similarity. (e) Image–text retrieval using aligned images from model A and text from model B. (f) Image–text retrieval using images from model B and aligned text from model A. (g) Image–text retrieval using aligned images and aligned text from model A. Mean adjustment boosts pointwise image-image (c) and text-text (d) cosine and downstream performance (e,f), but can slightly underperform on aligned-image-to-aligned-text and text-text retrieval (b,g).

Panel of Figure 51 — Figure 51: Cross-model alignment between identical training datasets with differing architectures (Caltech-101; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying $\mathcal{Q}$ yields substantially stronger modality transfer than in settings where the training distributions differ.

Panel of Figure 52 — Figure 52: Cross-model alignment between identical training datasets with differing architectures (CIFAR-100; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying $\mathcal{Q}$ yields substantially stronger modality transfer than in settings where the training distributions differ.

Panel of Figure 53 — Figure 53: Cross-model alignment between identical training datasets with differing architectures (Oxford Pets; CLIP ViT-B/32 to ViT-B/16, OpenAI) before and after fitting a single orthogonal map $\mathcal{Q}$ . (a) Image–image class retrieval and (b) text–text class retrieval. (c) Mean text–text cosine similarity. (d) Image–text retrieval using aligned images from model A and text from model B. (e) Image–text retrieval using images from model B and aligned text from model A. (f) Image–text retrieval using aligned images and aligned text from model A. When both models are trained on the same data distribution and differ only in architecture, applying $\mathcal{Q}$ yields substantially stronger modality transfer than in settings where the training distributions differ.

Figure 54: Qualitative k-NN retrieval under two alignment paths for Oxford Pets. We compare two routes from a source image $x$ into the target model’s image space after alignment by $\mathcal{Q}$ : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent target-space neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.

Figure 55: Qualitative k-NN retrieval under two alignment paths for Caltech-101. We compare two routes from a source image $x$ into the target model’s image space after alignment by $\mathcal{Q}$ : a direct image-alignment path (top row) and a text-mediated path (bottom row). Both routes yield highly consistent target-space neighborhoods, showing that the two alignment paths approximately close as a commuting diagram.

Figure 56: Fine-grained semantic preservation under controlled text edits on Oxford Pets. After changing only the class token in the caption (Birman to Russian Blue) while keeping descriptive attributes fixed, alignment via $\mathcal{Q}$ retrieves target-model images that match the edited class while preserving caption-specific visual cues (e.g., eye prominence and facial structure), indicating preservation of fine-grained semantics beyond class identity.

Figure 57: Fine-grained semantic preservation under controlled text edits on Oxford Pets. After changing only the class token in the caption (American Bulldog to Basset Hound) while keeping descriptive attributes fixed, alignment via $\mathcal{Q}$ retrieves target-model images that match both the edited class and the original attribute cues (e.g., muscular build and posture), indicating preservation of fine-grained semantics beyond class identity.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 159 papers

Expressivity In Multimodal Contrastive Learning
Aug 17, 2026Andrew Stuart, Florian WolfMultimodal Contrastive Learning Neural Network Expressivity
California Institute of Technology
Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMs Multimodal Embedding
Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel

Aug 17, 2026J/K move · Enter open · S save

Expressivity In Multimodal Contrastive Learning

Andrew Stuart, Florian Wolf

California Institute of Technology

Multimodal Contrastive Learning Neural Network Expressivity Universal Approximation

Open paper AskPDF arXiv

Contrastive learning has become a cornerstone of modern representation learning, powering CLIP-style models that underpin text-to-image generation, vision-language models, and retrieval across a rapidly growing range of modalities. Despite this empirical success, the expressive power of these architectures remains poorly understood. To gain insight, we study expressivity by adopting a population-level, density-estimation viewpoint: each architecture comprises a parameterized set of densities whose parameters may be chosen to approximate the joint distribution of the modalities. This isolates a question of pure representational capacity: which joint distributions can a given contrastive family of parameterizations approximate to arbitrary accuracy? We show that expressivity is sharply architecture-dependent. For two modalities, the simple two-tower CLIP architecture is a universal approximator. A natural generalization of CLIP, widely used in practice when three or more modalities are present, is based on a loss found by summing over all pairwise similarities. This provably cannot represent arbitrary joint distributions, although we prove that it remains expressive enough to match all pairwise conditionals. Motivated by this gap, we propose Hadamard-CLIP, which adds a single learned weight vector on top of the existing encoders and restores universal approximation of the joint for any number of modalities while preserving CLIP's fast, precomputable-embedding retrieval.

Abstract

Multimodal Embedding

Explore similar work

Expressivity In Multimodal Contrastive Learning

Closing the Modality Gap via Simplex-Constrained Representations

Is the Modality Gap a Bug or a Feature? A Robustness Perspective