Closing the Modality Gap via Simplex-Constrained Representations

Shubham Gupta; Siva Reddy; Perouz Taslakian; Valentina Zantedeschi; Cem Subakan

cs.AIOct 23, 2026

Closing the Modality Gap via Simplex-Constrained Representations

Authors: Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi, Cem Subakan

Organizations: Mila – Québec AI Institute · ServiceNow Research · Université Laval · McGill University

Abstract

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, $S^{d-1}$ , to the probability simplex, $Δ^{d-1}$ . Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid $\ell_2$ gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

Figures & tables

Figure 1: (a) Expected

\ell_{2}

distance vs. dimension. Sphere: plateau at

\sqrt{2}

. Dense simplex (

\alpha=1

): shrinks as

\mathcal{O}(1/\sqrt{d})

. Sparse simplex (

\alpha=c/d

): plateau at

\sqrt{2/(c+1)}

. (b) Dirichlet

\hat{\alpha}

estimated from learned ReTreever embeddings (SoundDescs). As dimension grows,

\hat{\alpha}\ll 1

, placing embeddings in the sparse regime where distances plateau rather than contract.

	Clotho					SoundDescs					AudioCaps
	Gap			Retrieval		Gap			Retrieval		Gap			Retrieval
Adapter	$\ell_{2}$ $\downarrow$	RG $\downarrow$	Sil $\downarrow$	T2A $\uparrow$	A2T $\uparrow$	$\ell_{2}$ $\downarrow$	RG $\downarrow$	Sil $\downarrow$	T2A $\uparrow$	A2T $\uparrow$	$\ell_{2}$ $\downarrow$	RG $\downarrow$	Sil $\downarrow$	T2A $\uparrow$	A2T $\uparrow$
CLAP	.640	.748	.197	.259	.436	.488	.521	.117	.273	.250	.363	.380	.067	.596	.507
Linear	.514	.689	.167	.373	.453	.458	.554	.133	.410	.418	.366	.468	.139	.584	.439
MLP	.323	.353	.064	.358	.436	.220	.264	.033	.520	.514	.266	.285	.049	.587	.448
XAttn	.450	.443	.126	.353	.432	.309	.331	.051	.464	.456	.297	.321	.053	.592	.452

Table 1: Audio-text modality gap and retrieval. Sfx = softmax. Simplex methods shaded . Modality gap measured as Centroid

\ell_{2}

, RelGap (RG, Eq. 1 ), silhouette (Sil) (lower is better (

\downarrow

)), Retrieval as NDCG@10 (higher is better (

\uparrow

)). Best per column in bold . † Lin.+Sfx and MLP+Sfx: per dataset, the better of the paper’s configuration (L2-normalised adapter input, no logit norm) and the same adapter trained on un-normalised input with a LayerNorm on the logits before the softmax; the other rows are unchanged.

Figure 3: Coarse-to-fine modality gap: MRL (Euclidean, red ) vs. ReTreever (simplex, blue ) across representation dimensions. Euclidean gaps grow; simplex gaps stay flat.

Figure 4: Relative gap (RelGap, Eq. 1 ) across adapter types on SoundDescs (left) and AudioCaps (right). Non-softmax methods ( red ) vs. softmax methods ( blue ). Lower is better. Softmax methods consistently reduce RelGap.

Modality Gap in VLMs

Modality gaps in vision-language models are mismatches between visual and textual representations or their use in downstream tasks.

See all 59 papers

Canonicalizing Multimodal Contrastive Representation Learning
Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal Embedding Representation Alignment
Aalto University, YaiYai Ltd
Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildGeometric Representation Learning Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

Feb 19, 2026J/K move · Enter open · S save

Canonicalizing Multimodal Contrastive Representation Learning

Sharut Gupta, Sanyam Kansal, Stefanie Jegelka, Phillip Isola+1

Aalto University, YaiYai Ltd

Multimodal Embedding Representation Alignment Multimodal Contrastive Learning Cross-Modal Alignment

Open paper AskPDF arXiv

As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders

(f, g)

and

(\widetilde{f},\widetilde{g})

) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map

Q

where

Q^\top Q = I

such that

\widetilde{f}(x)\approx Q f(x)

for paired images

x

. Strikingly, the same

Q

simultaneously aligns the text encoders i.e.,

\widetilde{g}(y)\approx Q g(y)

for texts

y

. Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e.

\langle f(x), g(y)\rangle \approx \langle \widetilde{f}(x), \widetilde{g}(y)\rangle

, then the two models must be related by a single orthogonal map

Q

and the same

Q

maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/

Abstract

Modality Gap in VLMs

Explore similar work

Canonicalizing Multimodal Contrastive Representation Learning

Closing the gap in multimodal medical representation alignment

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning