Expressivity In Multimodal Contrastive Learning

Andrew Stuart; Florian Wolf

stat.MLAug 17, 2026

Expressivity In Multimodal Contrastive Learning

Authors: Andrew Stuart, Florian Wolf

Organizations: California Institute of Technology

Abstract

Contrastive learning has become a cornerstone of modern representation learning, powering CLIP-style models that underpin text-to-image generation, vision-language models, and retrieval across a rapidly growing range of modalities. Despite this empirical success, the expressive power of these architectures remains poorly understood. To gain insight, we study expressivity by adopting a population-level, density-estimation viewpoint: each architecture comprises a parameterized set of densities whose parameters may be chosen to approximate the joint distribution of the modalities. This isolates a question of pure representational capacity: which joint distributions can a given contrastive family of parameterizations approximate to arbitrary accuracy? We show that expressivity is sharply architecture-dependent. For two modalities, the simple two-tower CLIP architecture is a universal approximator. A natural generalization of CLIP, widely used in practice when three or more modalities are present, is based on a loss found by summing over all pairwise similarities. This provably cannot represent arbitrary joint distributions, although we prove that it remains expressive enough to match all pairwise conditionals. Motivated by this gap, we propose Hadamard-CLIP, which adds a single learned weight vector on top of the existing encoders and restores universal approximation of the joint for any number of modalities while preserving CLIP's fast, precomputable-embedding retrieval.

Figures & tables

Panel of Figure 1 — Figure 1: Graphical Abstract. We frame contrastive learning as learning a tilting function ${\Phi}^{\mathsf{a}}$ , a score for how much more likely a tuple $(x_{1},\dots,x_{m})$ is to co-occur than under independence (the log-density of the joint relative to the product of marginals). Top row: Four ways to build ${\Phi}^{\mathsf{a}}$ from per-modality encoders, with the induced similarity score below each ( $\tau$ : learnable temperature, $\omega$ : learned fusion weights). Bottom table: Which distributions each architecture can approximate arbitrarily well. The full joint ${\boldsymbol{x}}$ , one modality given all others ( $x_{i}\mid x_{-i}$ ), and one given just one other ( $x_{i}\mid x_{j}$ ); ✔ = provably can, ✘ = provably cannot.

Multimodal Contrastive Learning

Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.

See all 246 papers

Canonicalizing Multimodal Contrastive Representation Learning
Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal Embedding Representation Alignment
Aalto University, YaiYai Ltd
Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
On the modality gap and the contrastive loss in multi-modal representation learning
Jul 12, 2026Fabian Mager, Hiba Nassar, Lars Kai HansenCross-Modal Alignment Multimodal Contrastive Learning
Department of Applied Mathematics and Computer Science Technical University of Denmark

Feb 19, 2026J/K move · Enter open · S save

Canonicalizing Multimodal Contrastive Representation Learning

Sharut Gupta, Sanyam Kansal, Stefanie Jegelka, Phillip Isola+1

Aalto University, YaiYai Ltd

Multimodal Embedding Representation Alignment Multimodal Contrastive Learning Cross-Modal Alignment

Open paper AskPDF arXiv

As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders

(f, g)

and

(\widetilde{f},\widetilde{g})

) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map

Q

where

Q^\top Q = I

such that

\widetilde{f}(x)\approx Q f(x)

for paired images

x

. Strikingly, the same

Q

simultaneously aligns the text encoders i.e.,

\widetilde{g}(y)\approx Q g(y)

for texts

y

. Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e.

\langle f(x), g(y)\rangle \approx \langle \widetilde{f}(x), \widetilde{g}(y)\rangle

, then the two models must be related by a single orthogonal map

Q

and the same

Q

maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/

Abstract

Multimodal Contrastive Learning

Explore similar work

Canonicalizing Multimodal Contrastive Representation Learning

Closing the Modality Gap via Simplex-Constrained Representations

On the modality gap and the contrastive loss in multi-modal representation learning