Representation Alignment from a Few Paired Data via Distilled Optimal Transport

Julien Kordes; Reuben Dorent

cs.AIOct 8, 2026

Representation Alignment from a Few Paired Data via Distilled Optimal Transport

Authors: Julien Kordes, Reuben Dorent

Organizations: Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, 75013 Paris, France

Abstract

Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only $N \sim 100$ -- $500$ pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the $N$ paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.

Figures & tables

Figure 1: Overview of AnchorDOT. Step 1: modality-specific representations are expressed by their similarity profile to a small set of shared anchors, embedding them in a common anchor-relative coordinate space . Step 2: an entropic optimal transport coupling is solved between the two anchor-relative embedding spaces, recovering a soft cross-modal correspondence between points. Step 3: this coupling is used as a pseudo-pairing signal to train two lightweight encoders

f_{\theta}

and

g_{\psi}

, via contrastive learning directly on the original pretrained embeddings, distilling the transport plan into a shared aligned space that requires no anchors and no optimal transport at inference time.

	Flickr30k						MS-COCO
	Image $\rightarrow$ Text			Text $\rightarrow$ Image			Image $\rightarrow$ Text			Text $\rightarrow$ Image
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Contrastive	22.05	49.10	60.45	22.30	48.05	59.60	14.60	40.65	54.95	15.00	40.55	54.30
ASIF	10.35	29.10	42.45	9.55	29.80	43.05	15.15	48.65	69.95	14.20	48.80	69.70
CSA	0.90	3.15	5.60	0.95	3.55	6.10	1.30	5.55	9.40	1.35	5.70	8.95
SUE	7.80	24.50	37.50	7.35	26.80	38.05	9.90	36.60	55.80	9.10	37.25	56.35

Table 1: Cross-modal retrieval at

N=500

anchors, recall@

k

(%). All baselines are re-run under our protocol, with the same frozen backbones, anchors and test split, averaged over five seeds. Our re-run of SUE exceeds its published Flickr30k results at this budget (I2T R@1

7.80

vs.

4.25

Table 3

Figure 2: Recall@10 (I2T/T2I mean) under two independent ablations on the pool composition. (a)–(b) Recall@

10

vs. number of anchor pairs

N

, at fixed pool and test split, over the full and low-

N

ranges respectively: (a) restricted to our method and the contrastive baseline, (b) all six methods of Table 1 . (c) Recall@

10

vs. unpaired pool size at fixed

N=500

and test split.

Figure 3: Recall@

10

on Flickr30k as a function of the overlap

o

, for SUE, SOTAlign and our method.

Figure 4: (a)

N_{10}

skewness (lower is better) and (b) recall@

10

of nearest neighbor, the inverted softmax, CSLS and our converged coupling, on the same anchor-relative cost between the

400

held-out Flickr30k test pairs, in both retrieval directions. Error bars denote

\pm 1

std over

5

seeds.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Recall comparison of the transductive OT teacher with how much is retained after distillation to the inductive encoders

Panel of Figure 6 — Figure 6: Participation ratio as a function of $\gamma$ evaluated on Flickr30k for three anchor budgets $N$ .

Figure 7: Recall comparison of our method to existing unsupervised baselines on Flickr30k with the same usual setting

Figure 8: Hubness and recall@10 comparison on Flickr30k and MS-COCO for the Bridge cost against our method. The

N_{10}

skewness gives similar results while the recall@10 drops considerably when using the Bridge cost suggesting that our method benefits from representing a point from its similarity to all anchors and not only the closest one.

Figure 9: Hubness and recall@10 comparison on Flickr30k and MS-COCO for the KPG-RL cost against our method. Both

N_{10}

skewness and recall@10 give similar results with our method getting slightly better performances on the retrieval task

Figure 10: Recall@k as a function of the weight

\alpha

in the FGW setting

Figure 11: Examples of images generated from a custom caption following Yacobi et al. [2025] ’s generation protocol

Figure 12: Examples of zero-shot classification on ImageNet comparing our method to SUE, SOTAlign (all trained on Flickr30k) and a pretrained CLIP checkpoint

Figure 13: Comparison of our method, SUE and SOTAlign on zero-shot retrieval on Flickr30k after being trained on

30k

samples from MS-COCO

Figure 14: Text to Image retrieval examples on the dataset Flickr30k

Figure 15: Image to Text retrieval examples on the dataset MS-COCO

Figure 16: Text to Molecule retrieval examples on the dataset ChEBI-20

	Image $\rightarrow$ Text			Text $\rightarrow$ Image
Method	R@1	R@5	R@10	R@1	R@5	R@10
Contrastive	22.05 (2.21)	49.10 (2.48)	60.45 (2.99)	22.30 (2.76)	48.05 (3.07)	59.60 (3.20)
ASIF	10.35 (0.57)	29.10 (2.68)	42.45 (3.24)	9.55 (0.82)	29.80 (2.17)	43.05 (1.56)
CSA	0.90 (0.45)	3.15 (1.28)	5.60 (1.27)	0.95 (0.65)	3.55 (0.84)	6.10 (0.96)
SUE	7.80 (1.51)	24.50 (1.71)	37.50 (0.85)	7.35 (0.70)	26.80 (1.00)	38.05 (1.08)
SOTAlign	35.60 (2.05)	65.00 (1.92)	76.20 (3.29)	35.20 (2.31)	65.10 (2.06)	75.55 (2.27)

Appendix

Table 4: Complement of Table 1 with std over 5 seeds for Flickr30k.

	Image $\rightarrow$ Text			Text $\rightarrow$ Image
Method	R@1	R@5	R@10	R@1	R@5	R@10
Contrastive	14.60 (0.58)	40.65 (2.41)	54.95 (1.45)	15.00 (1.43)	40.55 (2.69)	54.30 (2.64)
ASIF	15.15 (1.04)	48.65 (2.93)	69.95 (2.42)	14.20 (2.28)	48.80 (2.26)	69.70 (2.00)
CSA	1.30 (0.73)	5.55 (1.47)	9.40 (1.39)	1.35 (1.01)	5.70 (1.68)	8.95 (1.91)
SUE	9.90 (0.72)	36.60 (1.34)	55.80 (3.95)	9.10 (1.07)	37.25 (1.56)	56.35 (2.31)
SOTAlign	30.50 (1.93)	70.10 (1.86)	84.50 (2.46)	30.90 (1.19)	69.70 (2.78)	84.50 (2.08)

Appendix

Table 5: Complement of Table 1 with std over 5 seeds for MS-COCO.

	Molecule $\rightarrow$ Text			Text $\rightarrow$ Molecule
Method	R@1	R@5	R@10	R@1	R@5	R@10
Contrastive	16.10	37.90	48.20	15.10	36.15	47.05
ASIF	8.40 (0.91)	28.20 (1.52)	40.90 (2.28)	14.55 (0.60)	35.85 (1.46)	49.40 (0.91)
CSA	0.20 (0.20)	1.10 (0.84)	2.55 (0.51)	0.65 (0.52)	2.45 (1.08)	4.40 (1.52)
SUE	3.25 (0.77)	14.85 (2.71)	25.05 (3.83)	3.35 (1.01)	15.10 (1.65)	26.10 (3.52)
SOTAlign	20.30 (1.02)	47.15 (2.33)	60.60 (0.76)	20.30 (1.43)	47.80 (2.03)	62.20 (0.93)

Appendix

Table 6: Complement of Table 3 with std over 5 seeds for ChEBI-20.

Varying number of anchors ( $M=2000$ )
Setting	Value	Ours	SOTAlign	Speedup
	$N=100$	5.9 $\pm$ 3.4	77.1 $\pm$ 3.5	13.07 $\times$
	$N=500$	5.0 $\pm$ 0.4	75.0 $\pm$ 0.2	15.00 $\times$
	$N=1000$	7.9 $\pm$ 0.5	76.0 $\pm$ 0.2	9.62 $\times$
	$N=5000$	29.2 $\pm$ 6.8	99.6 $\pm$ 0.3	3.41 $\times$
Varying number of unpaired points ( $N=500$ )

Appendix

Table 7: Training wall-clock time (seconds) on Flickr30k, averaged over 3 seeds. We vary either the number of anchors

N

while fixing

M=2000

, or the number of unpaired points

M

while fixing

N=500

. Speedup is measured relative to SOTAlign.

Hyperparameter	Value
Projection dimension	1024
Optimizer	Adam
Learning rate	$10^{-3}$
Weight decay	$10^{-2}$
Warmup steps	200
Epochs	200

Appendix

Table 8: Training hyperparameters, fixed across datasets and seeds.

Figure 17: Held-out rank (mean, both retrieval directions),

N_{10}

skewness and recall@1 as a function of the epsilon multiplier

\eta

, one curve per

\gamma

value. The selected

(\gamma^{\star},\eta^{\star})

minimizes this held-out rank.

Dataset	$\gamma^{\star}$	$\eta^{\star}$
Flickr30k	0.75	0.02
MS-COCO	0.75	0.02
ChEBI-20	1.00	0.02

Appendix

Table 9: Selected profile hyperparameters per dataset (seed 0)

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
Feb 26, 2026Simon Roschmann, Paul Krzakala, Sonia Mazelet +2Cross-Modal Learning Multimodal Embedding
Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning +3
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
Aug 4, 2026Yixuan Florence Wu, Yilun Zhu, Naichen ShiCross-Modal Alignment Multimodal Embedding
Department of Engineering Sciences and Applied Mathematics Northwestern University Evanston, IL 60208, USA · Deparment of Electrical Engineering and Computer Science University of Michigan Ann Arbor, MI 48109, USA · Department of Industrial Engineering and Management Sciences Department of Mechanical Engineering Northwestern University Evanston, IL 60208, USA

Feb 26, 2026J/K move · Enter open · S save

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot+1

Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning+3

Cross-Modal Learning Multimodal Embedding Vision-Language Alignment Semi-Supervised Learning Latent Space Alignment Multimodal Understanding

Open paper AskPDF arXiv

The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.

Abstract

Cross-Modal Alignment

Explore similar work

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport