Connecting Neural Models Latent Geometries with Relative Geodesic Representations

Hanlin Yu; Berfin Inal; Georgios Arvanitidis; Søren Hauberg; Francesco Locatello; Marco Fumero

cs.LGJun 2, 2025

Connecting Neural Models Latent Geometries with Relative Geodesic Representations

Authors: Hanlin Yu, Berfin Inal, Georgios Arvanitidis, Søren Hauberg, Francesco Locatello, Marco Fumero

Organizations: University of Helsinki · University of Amsterdam · DTU · IST Austria

Abstract

Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential-geometric structure of latent spaces of neural models, it is possible to capture precisely the transformations between representational spaces trained on similar data distributions. Specifically, we assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, and pretraining schemes.

Figures & tables

Figure 1: Neural models trained on similar data learn parametrizations of the same manifold. NNs learn parametrizations ( $D_{1},D_{2}$ ) of the same underlying manifold $\mathcal{Y}$ up to isometries $T$ . Pulling back the metric from $\mathcal{Y}$ makes relative geodesic representations invariant to transformations $T$ between latent spaces $\mathcal{Z}_{1}$ and $\mathcal{Z}_{2}$ .

Panel of Figure 2 — Figure 2: Pairwise latent-space energy matrices for (a) MNIST and (b) CIFAR-10. In each subfigure, the left heatmap shows the straight-line energy approximation and the right shows the geodesic energies of the ground truth geodesic curve. The Spearman rank correlations between the two measures are $\rho=0.99$ for MNIST and $\rho=1.00$ for CIFAR-10, demonstrating near-perfect agreements.

Panel of Figure 3 — Figure 3: Aligning latent spaces of autoencoders : MRR score as a function of the number of anchors on pairs of autoencoders trained with different initializations on the MNIST (left), FashionMNIST (center), CIFAR10 (right) datasets, respectively. In green, we plot the performance of Moschella et al. (2023) ; in red and orange the linear and orthogonal baselines respectively; in blue, our method. The shaded area indicates standard deviation across 5 different random sets of anchors. Relative geodesic consistently outperforms baselines, obtaining peak performance.

Figure 4: Stitching on Autoencoders : We visualize qualitative reconstructions of samples, stitching autoencoders of models trained with different initializations on MNIST (left), FashionMNIST (center), CIFAR10 (right). The first two columns show reconstructions from the original models; middle three columns represent baselines ( Maiorca et al., 2024 ; Moschella et al., 2023 ) ; the rightmost column is our method. Relative geodesic yields the best stitching results using just 5 anchors.

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Rel(Cosine) ( Moschella et al., 2023 )	$0.129\pm 0.135$	$0.166\pm 0.162$	$0.221\pm 0.178$	$0.135\pm 0.148$	$0.068\pm 0.08$
RelGeo( $L2$ )	$0.047\pm 0.013$	$0.112\pm 0.031$	$0.412\pm 0.09$	$0.28\pm 0.129$	$0.025\pm 0.012$
RelGeo(Diet)	$\mathbf{0.387}\pm 0.145$	$\mathbf{0.445}\pm 0.142$	$\mathbf{0.566}\pm 0.111$	$\mathbf{0.523}\pm 0.177$	$\mathbf{0.314}\pm 0.188$

Table 1: Average MRR cosine results for different methods across different datasets. Relative representations pulling back from diet decoder ( RelGeo(Diet) ) consistently provides better retrievals.

Panel of Figure 5 — Figure 5: CUB Accuracies (top) and symmetricized MRR cosine (bottom). RelGeo(Diet) and especially RelGeo( $L2$ ) provide strong stitching accuracies, while RelGeo(Diet) maintains strong instance identifiability.

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Rel(Cosine) ( Moschella et al., 2023 )	$0.907\pm 0.09$	$0.775\pm 0.132$	$\mathbf{0.549}\pm 0.152$	$0.531\pm 0.188$	$0.384\pm 0.115$
RelGeo( $L2$ )	$\mathbf{0.955}\pm 0.03$	$\mathbf{0.874}\pm 0.055$	$0.501\pm 0.159$	$\mathbf{0.595}\pm 0.163$	$\mathbf{0.59}\pm 0.054$
RelGeo(Diet)	$0.915\pm 0.074$	$0.775\pm 0.115$	$0.479\pm 0.17$	$0.559\pm 0.171$	$0.416\pm 0.079$

Table 2: Average stitching performances across different settings. RelGeo(

L2

) often outperforms Rel(Cosine) , while RelGeo(Diet) remains competitive.

Figure 6: Matching multimodal models. Symmetricized MRR cosine on Flickr30k. RelGeo(Diet) substantially improves upon Rel(Cosine) in aligning multimodal models.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder

3\times 3

conv. 32 stride 2-ReLu

3\times 3

conv. 64 stride 2-ReLu

Flatten

(64*k*k)\times h

Linear

Latents

Decoder

Appendix

Table 3: Architecture of the convolutional autoencoders.

Table 10

Encoder	Activation
Conv2d(1, 32, kernel = 3, stride=2, pad=1)	ReLU
Conv2d(32, 64, kernel = 3, stride=2, pad=1)	ReLU
Flatten	—
Linear(6477, $d$ )	—
Decoder	Activation
Linear( $d$ , 6477)	ReLU

Appendix

Table 6: ConvAutoencoder architecture (latent dim

d

Panel of Figure 7 — Figure 7: Pairwise latent-space energy matrices for (a) MNIST and (b) CIFAR-10, with latent dimensionality 2. In each subfigure, the left heatmap shows the straight-line energy proxy and the right shows the full Riemannian geodesic energies. The Spearman rank correlation between the two measures is 0.99 for MNIST and $\rho=1.00$ for CIFAR-10, demonstrating near-perfect agreement.

Panel of Figure 8 — Figure 8: Impact of varying discretization levels on similarity and energy metrics for (a) MNIST and (b) CIFAR - 10 datasets. Each subplot shows how Spearman’s $\rho$ , Pearson’s $r$ , and Euclidean distance change as the number of discretization levels increases.

Panel of Figure 9 — Figure 9: Results on CIFAR-10. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist

Panel of Figure 10 — Figure 10: Results on CIFAR-100. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist

Panel of Figure 11 — Figure 11: Results on ImageNet-1k. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist

Panel of Figure 12 — Figure 12: Results on CUB. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist

Panel of Figure 13 — Figure 13: Results on SVHN. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Rel(Cosine) [ Moschella et al., 2023 ]	$0.098\pm 0.133$	$0.122\pm 0.164$	$0.103\pm 0.146$	$0.046\pm 0.055$	$0.046\pm 0.081$
RelGeo(L2)	$0.046\pm 0.013$	$0.105\pm 0.031$	$0.179\pm 0.173$	$0.187\pm 0.141$	$0.04\pm 0.021$
RelGeo(Diet)	$\mathbf{0.252}\pm 0.189$	$\mathbf{0.278}\pm 0.211$	$\mathbf{0.462}\pm 0.148$	$\mathbf{0.433}\pm 0.212$	$\mathbf{0.306}\pm 0.188$

Appendix

Table 7: Aggregated results of MRR CDist Sym.

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Rel(Cosine) [ Moschella et al., 2023 ]	$0.08\pm 0.077$	$0.122\pm 0.109$	$0.21\pm 0.149$	$0.089\pm 0.094$	$0.035\pm 0.034$
RelGeo(L2)	$0.019\pm 0.005$	$0.046\pm 0.016$	$0.236\pm 0.08$	$0.156\pm 0.089$	$0.013\pm 0.005$
RelGeo(Diet)	$\mathbf{0.189}\pm 0.108$	$\mathbf{0.241}\pm 0.117$	$\mathbf{0.358}\pm 0.126$	$\mathbf{0.327}\pm 0.184$	$\mathbf{0.131}\pm 0.107$

Appendix

Table 8: Aggregated results of MRR Cosine.

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Rel(Cosine) [ Moschella et al., 2023 ]	$0.051\pm 0.072$	$0.071\pm 0.107$	$0.078\pm 0.105$	$0.023\pm 0.02$	$0.02\pm 0.032$
RelGeo(L2)	$0.019\pm 0.005$	$0.04\pm 0.015$	$0.106\pm 0.11$	$0.108\pm 0.092$	$0.012\pm 0.005$
RelGeo(Diet)	$\mathbf{0.127}\pm 0.118$	$\mathbf{0.151}\pm 0.138$	$\mathbf{0.298}\pm 0.141$	$\mathbf{0.269}\pm 0.195$	$\mathbf{0.123}\pm 0.103$

Appendix

Table 9: Aggregated results of MRR CDist.

Method	ResNet-50	ViT-16	ViT-32	DINOv2
Rel(Cosine) [ Moschella et al., 2023 ]	$0.507\pm 0.2$	$0.669\pm 0.229$	$0.664\pm 0.218$	$0.678\pm 0.24$
RelGeo(L2)	$\mathbf{0.646}\pm 0.209$	$\mathbf{0.709}\pm 0.208$	$\mathbf{0.72}\pm 0.194$	$\mathbf{0.737}\pm 0.209$
RelGeo(Diet)	$0.529\pm 0.194$	$0.658\pm 0.229$	$0.661\pm 0.219$	$0.668\pm 0.237$

Appendix

Table 10: Alternatively aggregated results of Accuracy.

Method	ResNet-50	ViT-16	ViT-32	DINOv2
Rel(Cosine) [ Moschella et al., 2023 ]	$0.032\pm 0.023$	$0.212\pm 0.173$	$0.208\pm 0.175$	$0.124\pm 0.104$
RelGeo(L2)	$0.143\pm 0.132$	$0.197\pm 0.186$	$0.205\pm 0.189$	$0.154\pm 0.137$
RelGeo(Diet)	$\mathbf{0.336}\pm 0.143$	$\mathbf{0.506}\pm 0.2$	$\mathbf{0.526}\pm 0.187$	$\mathbf{0.42}\pm 0.106$

Appendix

Table 11: Alternatively aggregated results of MRR Cosine Sym.

Method	ResNet-50	ViT-16	ViT-32	DINOv2
Rel(Cosine) [ Moschella et al., 2023 ]	$0.009\pm 0.005$	$0.141\pm 0.156$	$0.134\pm 0.158$	$0.049\pm 0.047$
RelGeo(L2)	$0.052\pm 0.033$	$0.144\pm 0.146$	$0.147\pm 0.146$	$0.103\pm 0.085$
RelGeo(Diet)	$\mathbf{0.204}\pm 0.13$	$\mathbf{0.432}\pm 0.235$	$\mathbf{0.437}\pm 0.232$	$\mathbf{0.313}\pm 0.108$

Appendix

Table 12: Alternatively aggregated results of MRR CDist Sym.

Method	ResNet-50	ViT-16	ViT-32	DINOv2
Rel(Cosine) [ Moschella et al., 2023 ]	$0.011\pm 0.005$	$0.138\pm 0.11$	$0.133\pm 0.112$	$0.147\pm 0.128$
RelGeo(L2)	$0.074\pm 0.077$	$0.107\pm 0.118$	$0.116\pm 0.126$	$0.079\pm 0.074$
RelGeo(Diet)	$\mathbf{0.182}\pm 0.107$	$\mathbf{0.299}\pm 0.184$	$\mathbf{0.316}\pm 0.182$	$\mathbf{0.201}\pm 0.076$

Appendix

Table 13: Alternatively aggregated results of MRR Cosine.

Method	ResNet-50	ViT-16	ViT-32	DINOv2
Rel(Cosine) [ Moschella et al., 2023 ]	$0.007\pm 0.001$	$0.079\pm 0.086$	$0.089\pm 0.112$	$0.019\pm 0.015$
RelGeo(L2)	$0.023\pm 0.016$	$0.075\pm 0.096$	$0.078\pm 0.099$	$0.051\pm 0.05$
RelGeo(Diet)	$\mathbf{0.121}\pm 0.094$	$\mathbf{0.253}\pm 0.193$	$\mathbf{0.258}\pm 0.194$	$\mathbf{0.143}\pm 0.065$

Appendix

Table 14: Alternatively aggregated results of MRR CDist.

Panel of Figure 14 — Figure 14: Accuracies on CUB with varying number of anchors. From top to bottom: 50, 100, 200, 500, 1000 anchors.

Panel of Figure 15 — Figure 15: MRR Cosine Sym on CUB with varying number of anchors. From top to bottom: 50, 100, 200, 500, 1000 anchors.

Panel of Figure 16 — Figure 16: Results of RelGeo(Diet) on CUB with varying number of diet points. Top: accuracies; bottom: MRR Cosine Sym.

Panel of Figure 17 — Figure 17: Results of RelGeo(L2) on CUB with varying number of discretization steps. Top: accuracies; bottom: MRR Cosine Sym.

Panel of Figure 18 — Figure 18: Results of RelGeo(Diet) on CUB with varying number of discretization steps. Top: accuracies; bottom: MRR Cosine Sym.

Panel of Figure 19 — Figure 19: Results of RelGeo(Diet) on CUB with varying diet augmentation strengths. Results with strength 3 can be seen above. Top: accuracies; bottom: MRR Cosine Sym.

Panel of Figure 20 — Figure 20: Results using the fps scheme to select the anchors. From top to bottom: accuracies, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.

Panel of Figure 21 — Figure 21: Results using the kmeans scheme to select the anchors. From top to bottom: accuracies, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.

Panel of Figure 22 — Figure 22: Stitching results for multimodal scenario. From top to bottom: MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.

Method	MRR Cosine Sym	MRR CDist Sym	MRR Cosine	MRR CDist
Rel(Cosine) [ Moschella et al., 2023 ]	$0.298\pm 0.395$	$0.293\pm 0.389$	$0.283\pm 0.382$	$0.252\pm 0.373$
RelGeo(Diet)	$\mathbf{0.413}\pm 0.353$	$\mathbf{0.384}\pm 0.359$	$\mathbf{0.317}\pm 0.372$	$\mathbf{0.302}\pm 0.378$

Appendix

Table 15: Average MRR results for different methods across different datasets. Relative representations pulling back from diet decoder ( RelGeo(Diet) ) consistently provides better retrievals.

Decoder	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Abs	$10.467\pm 0.765$	$10.71\pm 0.655$	$8.804\pm 0.581$	$1.593\pm 0.615$	$15.57\pm 0.843$
Diet(0)	$3.462\pm 0.03$	$3.501\pm 0.045$	$3.556\pm 0.018$	$3.509\pm 0.028$	$3.506\pm 0.044$
Diet(1)	$718.818\pm 216.179$	$730.446\pm 226.91$	$1651.49\pm 211.037$	$1937.656\pm 192.894$	$705.929\pm 202.844$
Diet(2)	$735.144\pm 219.211$	$739.671\pm 222.36$	$2034.677\pm 276.309$	$2428.942\pm 196.388$	$727.477\pm 206.929$
Diet(3)	$771.299\pm 213.022$	$767.453\pm 222.62$	$2645.914\pm 260.069$	$3201.588\pm 242.339$	$754.965\pm 202.021$

Appendix

Table 16: Times in seconds for training the decoders.

Num Anchors	Time (s) $\pm$ Std	MRR
2	$0.8480\pm 0.0407$	$0.0168$
3	$0.8348\pm 0.0367$	$0.0807$
5	$0.8377\pm 0.0435$	$0.3503$
8	$0.9450\pm 0.0285$	$0.7004$
10	$1.0969\pm 0.0297$	$0.8384$
15	$1.2853\pm 0.0264$	$0.9296$

Appendix

Table 17: Running time / accuracy tradeoff of RelGeo(L2).

Num Anchors	Time (s)	Accuracy
200	$0.088\pm 0.14$	$0.425\pm 0.189$
500	$0.175\pm 0.053$	$0.531\pm 0.188$
1000	$0.062\pm 0.091$	$0.559\pm 0.179$

Appendix

Table 18: Running time / accuracy tradeoff of Rel(Cosine).

Num Anchors	Time (s)	Accuracy
200	$8.004\pm 3.304$	$0.5\pm 0.184$
500	$21.297\pm 8.774$	$0.595\pm 0.163$
1000	$47.218\pm 19.486$	$0.619\pm 0.154$

Appendix

Table 19: Running time / accuracy tradeoff of RelGeo(L2).

Num Anchors	Time (s)	Accuracy
200	$7.274\pm 3.28$	$0.459\pm 0.177$
500	$19.427\pm 8.771$	$0.559\pm 0.171$
1000	$43.108\pm 19.502$	$0.585\pm 0.163$

Appendix

Table 20: Running time / accuracy tradeoff of RelGeo(Diet).

Method	CIFAR-10	CIFAR-100	ImageNet-1k	CUB
Rel(Cosine) [ Moschella et al., 2023 ]	$0.071\pm 0.113$	$0.184\pm 0.065$	$0.085\pm 0.124$	$0.05\pm 0.071$
RelGeo(L2)	$85.414\pm 40.61$	$86.079\pm 40.652$	$258.788\pm 70.064$	$139.998\pm 66.509$
RelGeo(Diet)	$89.899\pm 40.599$	$89.902\pm 40.588$	$155.991\pm 70.136$	$147.336\pm 66.505$

Appendix

Table 21: Times in seconds of Rel(Cosine) , RelGeo(L2) and RelGeo(Diet) for generating the representations.

Metric	CIFAR-10	CIFAR-100	ImageNet-1k	CUB	SVHN
Accuracy	$0.959\pm 0.026$	$0.894\pm 0.043$	$0.623\pm 0.103$	$0.729\pm 0.133$	$0.625\pm 0.033$
MRR Cosine Sym	$0.012\pm 0.0$	$0.021\pm 0.003$	$0.075\pm 0.014$	$0.211\pm 0.082$	$0.034\pm 0.01$
MRR CDist Sym	$0.012\pm 0.001$	$0.025\pm 0.004$	$0.13\pm 0.046$	$0.201\pm 0.08$	$0.028\pm 0.008$
MRR Cosine	$0.012\pm 0.001$	$0.019\pm 0.003$	$0.066\pm 0.022$	$0.152\pm 0.069$	$0.011\pm 0.002$
MRR CDist	$0.012\pm 0.001$	$0.023\pm 0.005$	$0.111\pm 0.052$	$0.144\pm 0.073$	$0.011\pm 0.003$

Appendix

Table 22: Results of RelGeo(Fisher).

Panel of Figure 23 — Figure 23: Results of RelGeo(Fisher) on CIFAR-10.

Panel of Figure 24 — Figure 24: Results of RelGeo(Fisher) on CIFAR-100.

Panel of Figure 25 — Figure 25: Results of RelGeo(Fisher) on ImageNet-1k.

Panel of Figure 26 — Figure 26: Results of RelGeo(Fisher) on CUB.

Panel of Figure 27 — Figure 27: Results of RelGeo(Fisher) on SVHN.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 264 papers

Multi-Way Representation Alignment
Feb 5, 2026Akshit Achara, Tatiana Gaintseva, Mateo Mahaut +5Neural Representation Geometry Latent Space Alignment
King’s College London, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Universitat Pompeu Fabra, Barcelona, Spain +5
Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching
Aug 28, 2026Cameron Ryan, Vivek Sivaraman Narayanaswamy, Kowshik Thopalli +1Model Stitching Latent Space Alignment
Northeastern University, Boston, MA 02115. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94551-0808.
Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds
Mar 30, 2026N Alex Cayco-Gajic, Arthur PellegrinoNeural Representation Geometry Representational Similarity Analysis
Gatsby Unit, University College London

Feb 5, 2026J/K move · Enter open · S save

Multi-Way Representation Alignment

Akshit Achara, Tatiana Gaintseva, Mateo Mahaut, Pritish Chakraborty+4

King’s College London, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Universitat Pompeu Fabra, Barcelona, Spain+5

Neural Representation Geometry Latent Space Alignment

Open paper AskPDF arXiv

The Platonic Representation Hypothesis suggests that independently trained neural networks converge to increasingly similar latent spaces. However, current strategies for mapping these representations are inherently pairwise, scaling quadratically with the number of models and failing to yield a consistent global reference. In this paper, we study the alignment of

M \ge 3

models. We first adapt Generalized Procrustes Analysis (GPA) to construct a shared orthogonal universe that preserves the internal geometry essential for tasks like model stitching. We then show that strict isometric alignment is suboptimal for retrieval, where agreement-maximizing methods like Canonical Correlation Analysis (CCA) typically prevail. To bridge this gap, we finally propose Geometry-Corrected Procrustes Alignment (GCPA), which establishes a robust GPA-based universe followed by a post-hoc correction for directional mismatch. Extensive experiments demonstrate that GCPA consistently improves any-to-any retrieval while retaining a practical shared reference space.

Abstract

Neural Representation Geometry

Explore similar work

Multi-Way Representation Alignment

Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching

Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds