Aligning two models without paired data requires a distance that both share, yet existing methods use Euclidean or spherical geometry. This is either an arbitrary choice or imposed by their loss function. We show that, under a precise form of the manifold hypothesis, the pullback of the data metric gives such a distance for any two models of one data-generating process. This holds whether the models are encoders, decoders or both, and whatever their architecture, loss, prior or training data. The alignment map preserves this distance and the distribution of representations, and is identifiable if and only if the data distribution breaks every symmetry of the data manifold. We show that this symmetry breaking holds generically. Identifiability makes the alignment coupling the unique optimum of Gromov--Wasserstein and of its weighted combinations with a linear program. The linear program alone has the same optimum if almost every data point has a distinct distribution of distances. The coupling, and with it the alignment map, is recovered exactly from permuted samples, even without identifiability, and consistently from independent ones. Our method, Neural Gromov--Monge Alignment (NGMA), uses no pairs or labels and outperforms every unpaired and label-informed baseline. It relies on pullback distances to align image models with different priors and single-cell models trained on disjoint cells, where Euclidean distances fail.
Figures & tables
Figure 1: Outline of Sections 2 – 4 : the setting, identifiability and recovery from samples.
Figure 2: Three data mm-spaces on the round sphere, with the symmetries S that preserve each distribution (olive), the vMF centres (teal, area proportional to weight) and iso-density contours. O(3) consists of all isometries of the sphere, and O(2) of those that fix the vMF centre.
Figure 3: Data on the sphere with A=R . (a) Rotation error of the LDD-LP on independent samples against the number of landmarks k (median and range over 10 seeds; at k=2000 every point is a landmark), and zero on permuted samples (black). (b) The LDD-LP, NGMA’s coupling and entropic GW alone, annealed as in NGMA, on Pmix . (c) The margin test: the estimated margin over its DKW accuracy, ξ^(t) , mean ± s.d. over 3 seeds, which rejects equal LDDs where ξ^(t)>1 .
Best baseline
Paired ceiling
Prior
NGMA
unpaired
label-informed
NGMA
Rigid map
MNIST
Gaussian
0.967 ± 0.021
0.941 ± 0.028
0.525 ± 0.069 ‡
0.998 ± 0.001
0.955 ± 0.021
VampPrior
0.989 ± 0.012
0.930 ± 0.012
0.552 ± 0.235
1.000 ± 0.000
0.936 ± 0.008
cross-prior
0.955 ± 0.008
0.571 ± 0.021
0.319 ± 0.152
0.999 ± 0.000
0.512 ± 0.021
Fashion-
Gaussian
0.719 ± 0.112
0.710 ± 0.040
0.159 ± 0.134
0.997 ± 0.001
0.782 ± 0.032
MNIST
VampPrior
0.979 ± 0.027
0.656 ± 0.045
0.355 ± 0.131
1.000 ± 0.000
0.747 ± 0.029
Table 1: R@1 among the 10,000 test points ( 19,962 on CelebA) of VAEs aligned on disjoint samples ( n=3000 per side, 7500 on CelebA), as mean ± s.d. over model pairings and draws. The best unpaired baseline is Rosetta, or mini-vec2vec where marked † . The best label-informed baseline is RelGeo, or RelReps where marked ‡ . The paired ceilings fit NGMA’s map and a rigid map on the true pairs. On CelebA, parentheses count draws reaching R@1 of 0.3 .
Figure 4: scVI models of lung cells trained on different cells, aligned on disjoint samples of 3000 cells per side, against the number of training cells per model. (a) R@1 among 10,000 held-out cells (chance 10−4 ). (b) Cell-type accuracy of the nearest cell. Lines are means over two model pairings and three draws, faint markers are single draws, and dashed lines are maps fitted on the true pairs.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Gaussian
VampPrior
cross-prior
Method
R@1
CSLS@1
R@1
CSLS@1
R@1
CSLS@1
Given the true pairs
NGMA, fitted on the true pairs
0.998 ± 0.001
0.999 ± 0.000
1.000 ± 0.000
1.000 ± 0.000
0.999 ± 0.000
1.000 ± 0.000
Rigid map, fitted on the true pairs
0.955 ± 0.021
0.975 ± 0.014
0.936 ± 0.008
0.954 ± 0.012
0.512 ± 0.021
0.441 ± 0.032
Label-informed (class labels or anchors)
RelReps
0.525 ± 0.069
0.585 ± 0.069
0.297 ± 0.082
0.338 ± 0.096
0.004 ± 0.001
0.007 ± 0.001
Appendix
Table 2: Instance retrieval on MNIST from disjoint samples ( n=3,000 per side), every method, scored as in Table 1 . Mean ± s.d. is the mean over model pairings (six for Gaussian and VampPrior, four for cross-prior) of each pairing’s mean, with the s.d. over all draws. NGMA, Rosetta and mini-vec2vec use five draws per pairing, the other rows at least two. CSLS@1 is top-1 retrieval under the hubness correction of Conneau et al. [2018] . RelReps uses class-centroid anchors, so RelReps-Centroid is the same computation as RelReps a→b here and is not repeated. NGMA is ahead of every baseline not given the true pairs in all three columns: 0.967 against 0.941 (Rosetta), 0.989 against 0.930 (Rosetta), 0.955 against 0.571 (Rosetta). With the cross-prior pair, NGMA from GW on the Euclidean cost reaches 0.146 and from the LDD-LP coupling 0.894.
Gaussian
VampPrior
cross-prior
Method
R@1
CSLS@1
R@1
CSLS@1
R@1
CSLS@1
Given the true pairs
NGMA, fitted on the true pairs
0.997 ± 0.001
0.999 ± 0.000
1.000 ± 0.000
1.000 ± 0.000
0.998 ± 0.001
0.999 ± 0.000
Rigid map, fitted on the true pairs
0.782 ± 0.032
0.846 ± 0.031
0.747 ± 0.029
0.847 ± 0.024
0.071 ± 0.004
0.062 ± 0.005
Label-informed (class labels or anchors)
RelReps
0.151 ± 0.053
0.180 ± 0.059
0.062 ± 0.013
0.073 ± 0.016
0.002 ± 0.001
0.002 ± 0.001
Appendix
Table 3: Instance retrieval on FashionMNIST from disjoint samples ( n=3,000 per side), every method, scored as in Table 1 . Mean ± s.d. is the mean over model pairings (six for Gaussian and VampPrior, four for cross-prior) of each pairing’s mean, with the s.d. over all draws. NGMA, Rosetta and mini-vec2vec use five draws per pairing, the other rows at least two. CSLS@1 is top-1 retrieval under the hubness correction of Conneau et al. [2018] . RelReps uses class-centroid anchors, so RelReps-Centroid is the same computation as RelReps a→b here and is not repeated. NGMA is ahead of every baseline not given the true pairs in all three columns: 0.719 against 0.710 (Rosetta), 0.979 against 0.656 (Rosetta), 0.871 against 0.098 (RelGeo). With the cross-prior pair, NGMA from GW on the Euclidean cost reaches 0.643 and from the LDD-LP coupling 0.614.
Gaussian
VampPrior
cross-prior
Method
R@1
CSLS@1
R@1
CSLS@1
R@1
CSLS@1
Given the true pairs
NGMA, fitted on the true pairs
0.995 ± 0.000
0.995 ± 0.000
0.996 ± 0.000
0.997 ± 0.000
0.996 ± 0.000
0.996 ± 0.000
Rigid map, fitted on the true pairs
0.993 ± 0.000
0.997 ± 0.000
0.995 ± 0.000
0.998 ± 0.000
0.990 ± 0.000
0.996 ± 0.000
Label-informed (class labels or anchors)
RelReps-Centroid
0.270 ± 0.028
0.352 ± 0.031
0.249 ± 0.065
0.305 ± 0.077
0.000 ± 0.000
0.000 ± 0.000
Appendix
Table 4: Instance retrieval on CelebA from disjoint samples ( n=7,500 per side), every method, scored as in Table 1 . Mean ± s.d. is the mean over model pairings (three for Gaussian, two for VampPrior and two for cross-prior) of each pairing’s mean, with the s.d. over all draws. NGMA, Rosetta and mini-vec2vec use five draws per pairing, the other rows at least two. CSLS@1 is top-1 retrieval under the hubness correction of Conneau et al. [2018] . NGMA is ahead of every baseline not given the true pairs in all three columns: 0.804 against 0.567 (Rosetta), 0.818 against 0.616 (mini-vec2vec), 0.818 against 0.757 (Rosetta). With the cross-prior pair, NGMA from GW on the Euclidean cost reaches 0.006 and from the LDD-LP coupling 0.643. Where a draw of NGMA has R@1 below 0.3 , the parentheses count the draws that reach it. From NGMA’s coupling, 28 of the 35 draws reach it; from GW alone, 25.
Figure 5: NGMA on CelebA, every draw of every model pairing (35 draws): R@1 from NGMA’s coupling against R@1 from entropic GW alone, on the same samples with the same map. Dotted lines mark R@1 0.3 . 28 draws reach it with NGMA’s coupling and 25 with GW; the coupling recovers 3 draws on which GW fails and loses 0, and no draw is lower by more than 0.004.
IWAE (nats)
Generator MMD 2 ( ×103 )
Setting
pairs
a−b
s.e.
a vs. b
a vs. data
b vs. data
p
MNIST, Gaussian
6
−4.3 – +0.5
0.6–0.8
0.4–1.3
8.0–10.5
8.1–8.7
0.005–0.010
MNIST, VampPrior
6
−0.6 – +3.9
0.6–0.8
0.8–1.6
2.5–2.7
2.3–2.9
0.005
MNIST, cross-prior
4
+22.3 – +25.2
0.7–0.9
2.6–5.2
2.5–2.7
7.4–8.7
0.005
FashionMNIST, Gaussian
6
−1.0 – +1.6
0.3–0.4
0.5–1.2
13.7–16.5
12.7–13.7
0.005–0.055
FashionMNIST, VampPrior
6
−1.1 – +0.2
0.3–0.4
0.0–0.7
3.1–3.3
2.6–3.0
0.030–0.428
Appendix
Table 5: Likelihood and generated distributions of the two models of every pair: the difference of their importance-weighted (IWAE) bounds on 2000 test images ( K=1000 importance samples), and the squared maximum mean discrepancy (MMD 2 ) between their generated images, and of each against the data, with the permutation p -value of the first. Ranges over pairings; in the cross-prior settings a is the VampPrior model. The cross-prior pairs differ by 10.2–25.2 nats, the same-prior pairs by at most 4.3. The two generators of a cross-prior pair are further apart than the VampPrior model is from the data on 7 of 8 pairs, and than the Gaussian model is from the data on 0. The MMD test also tells 18 of 24 same-prior pairs apart ( p<0.05 ; 0.005 is the smallest p that 200 permutations can return), so the IWAE gap, not the MMD, is what sets the cross-prior pairs apart. They are isometric (Table 6 ) without being equivalent, as Proposition 2.3 allows.
Pullback distances
Euclidean distances
Setting
pairs
corr.
scale
same neighbour
corr.
scale
same neighbour
MNIST, Gaussian
6
0.987–0.991
0.98–0.99
0.84–0.87
0.903–0.941
0.99–1.00
0.58–0.63
MNIST, VampPrior
6
0.992–0.994
0.99–1.00
0.86–0.90
0.961–0.973
0.98–1.02
0.74–0.75
MNIST, cross-prior
4
0.976–0.983
1.00–1.01
0.84–0.86
0.696–0.714
2.89–2.95
0.43–0.44
FashionMNIST, Gaussian
6
0.979–0.989
1.00
0.81–0.84
0.905–0.928
0.99–1.00
0.57–0.60
FashionMNIST, VampPrior
6
0.994–0.996
0.98–1.01
0.85–0.86
0.972–0.978
1.00–1.02
0.72–0.77
Appendix
Table 6: Pairwise distances in one model against the same pairs in the other, under the true correspondence, on the samples with 100% overlap (one draw, 1500 representations). Pullback distances are segment lengths under the decoder ensemble (Equation 15 ). corr. is the Pearson correlation of the two distance matrices, scale the factor by which one exceeds the other, and same neighbour the share of images whose nearest neighbour is the same image in both models. Ranges are over pairings. Pullback distances agree across all 32 pairs (corr. 0.973–0.996, scale 0.98–1.01). Euclidean distances do not on the cross-prior pairs, with corr. 0.71, 0.61 and scale 2.92, 5.01 (MNIST, FashionMNIST). A random pairing gives ∣ corr. ∣≤0.01 .
Figure 6: NGMA’s R@1 against the share of images the two samples have in common, fitted from five couplings, every MNIST and FashionMNIST setting. Lines join the means over pairings and draws, and dots are single draws. At 0% the samples are the disjoint samples of Table 1 , with five draws per pairing; every other share has two. The three GW couplings are plain entropic GW, without NGMA’s LDD-LP stages. With all images shared, NGMA’s coupling reaches 0.990–0.999 and GW on the pullback cost 0.991–0.999 in every setting, and GW on the Euclidean cost reaches 0.353 on the MNIST cross-prior pair and 0.679 on the FashionMNIST cross-prior pair. The LDD-LP needs shared images where LDDs are poorly separated, as on FashionMNIST with Gaussian priors (0.268 at 0% , 0.874 at 15% and 0.980 at 33% ).
Share of images in common (%)
Method
0
5
10
15
33
66
100
MNIST, Gaussian ; rigid map on the true pairs 0.955
NGMA
0.967
0.963
0.969
0.982
0.989
0.999
0.999
RelReps
0.525
0.531
0.531
0.539
0.549
0.573
0.600
RelGeo
0.512
0.491
0.485
0.482
0.451
0.496
0.476
Rosetta
0.941
0.945
0.950
0.952
0.956
0.957
0.958
Appendix
Table 7: R@1 against the share of images the two samples have in common, every method on every MNIST and FashionMNIST pairing: the mean over a setting’s pairings of the mean over draws. At 0% the samples are the disjoint samples of Table 1 , with five draws per pairing (two for RelReps and RelGeo); every other share has two draws per pairing. RelReps and RelGeo are label-informed. Rosetta and mini-vec2vec use no pairs and no labels, like NGMA. The rigid map fitted on the true pairs does not depend on the share, so each setting’s header gives its R@1. Rosetta and mini-vec2vec centre and normalise the codes before fitting their rotation, so they can exceed it. The best R@1 in each column of a setting is in bold. NGMA has the highest R@1 in every column of every setting. On single pairings, a baseline matches or exceeds NGMA at 4 of 896 points (method, pairing, share): Rosetta on FashionMNIST, Gaussian ac at 0% (0.652 against 0.537); Rosetta on MNIST, Gaussian ac at 5% (0.921 against 0.917); Rosetta on MNIST, Gaussian ac at 10% (0.926 against 0.919); Rosetta on MNIST, Gaussian ad at 15% (0.934 against 0.932).
a→b
b→a
Draws
n
true R@1
mirror R@1
true R@1
mirror R@1
self-mirror rate
CelebA, failed draws
NGMA’s coupling
7
0.054–0.113
0.905–0.992
0.055–0.111
0.910–0.992
0.101–0.116
GW alone
10
0.052–0.113
0.870–0.993
0.053–0.111
0.893–0.993
0.101–0.116
CelebA, successful draws
NGMA’s coupling
28
0.948–0.995
0.057–0.106
0.947–0.994
0.057–0.103
0.101–0.116
Appendix
Table 8: The mirror test. Every test image is flipped left to right and encoded by both models. True R@1 scores the image of a representation under a map against the representations of the unflipped images, and mirror R@1 against those of the flipped images, with the image’s own mirror as the partner. The self-mirror rate is the fraction of images whose own mirror is the nearest flipped representation in the same model. Ranges are over draws. The 70 CelebA alignments are the seven pairings with draws 1–5, each from NGMA’s coupling and from GW alone, and a draw fails when its R@1 is below 0.3 . The controls are MNIST and FashionMNIST, Gaussian pairing ab , draws 1–5.
different cells
same cells
Training size
4,900
9,800
19,600
4,900
9,800
19,600
R@1
NGMA
0.221 ± 0.035
0.369 ± 0.026
0.378 ± 0.015
0.274 ± 0.023
0.401 ± 0.031
0.480 ± 0.016
NGMA, Euclidean
0.119 ± 0.064
0.002 ± 0.002
0.047 ± 0.066
0.062 ± 0.058
0.181 ± 0.118
0.146 ± 0.146
NGMA, one decoder
0.195 ± 0.031
0.348 ± 0.020
0.294 ± 0.075
0.257 ± 0.013
0.372 ± 0.050
0.389 ± 0.073
unpaired baselines
Appendix
Table 9: scVI models of Tabula Sapiens lung cells (Appendix E.6 ): NGMA against the unpaired baselines, on the disjoint samples ( n=3000 cells per side) of two models trained on different cells or on the same cells with different seeds, by training size. Retrieval of the true partner among the 10,000 evaluation cells (R@1, chance 10−4 ) and cell-type accuracy of the nearest cell (chance 0.124), scored as in Table 1 . GENOT-Q runs on its own GW coupling with the Euclidean cost or with the pullback cost NGMA uses. NGMA, one decoder, reads the pullback metric of the scVI decoder alone instead of the ensemble of eight. The last two rows of each block are not methods. They are NGMA’s map class and the rigid map, both fitted on the true pairs and scored a→b . Mean ± s.d. over two pairings and three draws; the best method in each column is in bold. † RelGeo builds its anchors from the cell-type labels of both models.
Stage
MNIST, FashionMNIST
CelebA
VAE training, per model
3
16–25
Pullback distances, per model
9–18
810
Coupling, per draw
0.02–0.2
0.05–2.3
Map fit and evaluation, per draw
0.1–0.25
0.35–0.75
One pair of models, one draw
25–43
1 700
Appendix
Table 10: Compute per stage, in hours on one GPU: one GCD of an AMD Instinct MI250X, which has two and which the software sees as two GPUs. Distances are pullback distances under the 8-decoder ensemble between all n images a model encodes ( n=6000 for MNIST and FashionMNIST, n=15000 for CelebA, of which each draw uses 7500); jobs split by rows compute each pair twice, the upper end of the MNIST/FashionMNIST range. The whole project, development included, used about 50 000 GPU-hours.
The Platonic Representation Hypothesis suggests that independently trained neural networks converge to increasingly similar latent spaces. However, current strategies for mapping these representations are inherently pairwise, scaling quadratically with the number of models and failing to yield a consistent global reference. In this paper, we study the alignment of M≥3 models. We first adapt Generalized Procrustes Analysis (GPA) to construct a shared orthogonal universe that preserves the internal geometry essential for tasks like model stitching. We then show that strict isometric alignment is suboptimal for retrieval, where agreement-maximizing methods like Canonical Correlation Analysis (CCA) typically prevail. To bridge this gap, we finally propose Geometry-Corrected Procrustes Alignment (GCPA), which establishes a robust GPA-based universe followed by a post-hoc correction for directional mismatch. Extensive experiments demonstrate that GCPA consistently improves any-to-any retrieval while retaining a practical shared reference space.
Akshit Achara, Tatiana Gaintseva, Mateo Mahaut +5
King’s College London, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Universitat Pompeu Fabra, Barcelona, Spain +5
Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only N∼100--500 pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the N paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.
Julien Kordes, Reuben Dorent
Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, 75013 Paris, France
Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization (GeoWarP) aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.