Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching
Authors: Cameron Ryan, Vivek Sivaraman Narayanaswamy, Kowshik Thopalli, Shusen Liu
Organizations: Northeastern University, Boston, MA 02115. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94551-0808.
Independently trained neural networks tend to encode the same data with similar latent geometries. These latent geometries are not directly compatible, yet they can be nearly the same up to some class of transformations. While there exists many methods for alignment between different latent spaces, it is typically done using a set of shared sample correspondences, known as anchors. This leaves a fundamental question: are the geometric signatures of different latent spaces representing similar data sufficient to recover an alignment between them? To that end, we introduce HGA (Hyperspherical Gaussian Alignment), a method that directly optimizes a transformation between two latent spaces by maximizing a geometric measure of "fit" between them. Since it is driven by the geometry of the latent spaces rather than paired data, HGA can operate in both an unsupervised and weakly supervised regime. On tasks such as model stitching or multilingual word embedding correspondence recovery, HGA manages to match supervised results with minimal or no supervision.
Figures & tables
Dataset
Supervised Ortho
Unsupervised Latent Functional Maps
Unsupervised HGA
MNIST
96.79%
27.73±14.39%
96.55±0.09%
CIFAR10
73.48%
19.62±9.54%
72.80±0.30%
CIFAR100
34.51%
1.42±0.33%
31.20±0.96%
Table 1: Stitching accuracy (%) of fully supervised and unsupervised methods on test sets for each dataset. HGA optimization is done with 50 restarts for each dataset, and the rotation matrix with the highest objective function score is used for model stitching. Results show mean and standard deviation for 3 runs of Latent Functional Maps and HGA with different random seeds.
Figure 1: Word correspondence accuracy vs number of anchors for various alignment methods on word embeddings from FastText
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Objective function score at σ=0.2 vs optimization step for 10 random orthogonal matrix initializations. Color denotes the final objective function score reached. Noticeable jumps are due to a decrease in σ for the objective function Jσ(R) being used to compute optimization gradient.
Layer
σ
σstart
LR
LR final
Stitching Acc. (HGA unsupervised)
Stitching Acc. (Ortho Supervised)
4
0.1
4
0.1
0.0005
70%
70%
5
0.1
4
0.1
0.0005
62%
71%
6
0.1
4
0.1
0.001
75%
73%
7
0.2
4
0.01
0.0005
75%
76%
8
0.1
4
0.01
0.0005
76%
77%
Appendix
Table 2: CIFAR10 best unsupervised stitching accuracies and corresponding hyperparameters for each model layer. Results from an experiment running the optimization algorithm with a single restart and different hyperparameters for each run. Far right column shows stitching accuracy for orthogonal least squares solution with 1024 anchor points.
Figure 3: Percent of Anchors Corrupted vs. Model Stitching Accuracy for the latent spaces of two neural networks trained on the MNIST dataset.
Figure 4: Scatter plots show objective function scores vs stitching accuracies for various optimization trials with slightly different optimization hyperparameters on CIFAR10.
Figure 5: Implied kernel subsets and model stitching confusion matrices for unsupervised stitching in MNIST, CIFAR10, and CIFAR100
LFM
HGA
R
Word
Score
R
Word
Score
dog (in anchor list)
1
chien
0.050
1
chien
0.929
2
chat
0.040
2
chiens
0.692
3
animaux
0.031
3
chat
0.683
4
peau
0.028
4
animal
0.609
Appendix
Figure 6: Qualitative comparison of nearest French neighbours retrieved by LFM and HGA for English query words, given 100 anchor words. Correct translations are shown in bold . For anchor words ( dog , house ) both methods rank the correct translation first, but for words absent from the anchor list ( bird , horse , flower , forest ) LFM largely returns unrelated or noisy tokens, whereas HGA still recovers the correct translation.
Figure 7: Subset of kernel between a model trained on MNIST excluding the “0” class, and a regular model trained on MNIST
Figure 8: Kernel sum distributions for each class between a model trained on MNIST excluding the “0” class, and a regular model trained on MNIST
Rank
Orthogonal Procrustes
Latent Functional Maps
Spherical Alignment
1
banner
pennant
banner
2
pennant
banner
pennant
3
minivan
tag
scoreboard
4
scoreboard
jersey
tag
5
tag
game
jersey
6
woman
visor
board game
Appendix
Table 3: Concepts from the THINGS dataset with the lowest average kernel sums. We show results using three different alignment methods (ranked ascending).
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.
The Platonic Representation Hypothesis suggests that independently trained neural networks converge to increasingly similar latent spaces. However, current strategies for mapping these representations are inherently pairwise, scaling quadratically with the number of models and failing to yield a consistent global reference. In this paper, we study the alignment of M≥3 models. We first adapt Generalized Procrustes Analysis (GPA) to construct a shared orthogonal universe that preserves the internal geometry essential for tasks like model stitching. We then show that strict isometric alignment is suboptimal for retrieval, where agreement-maximizing methods like Canonical Correlation Analysis (CCA) typically prevail. To bridge this gap, we finally propose Geometry-Corrected Procrustes Alignment (GCPA), which establishes a robust GPA-based universe followed by a post-hoc correction for directional mismatch. Extensive experiments demonstrate that GCPA consistently improves any-to-any retrieval while retaining a practical shared reference space.
Akshit Achara, Tatiana Gaintseva, Mateo Mahaut +5
King’s College London, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Universitat Pompeu Fabra, Barcelona, Spain +5
Recent Joint-Embedding Predictive Architectures (JEPAs) prevent representation collapse by constraining learned representations to follow a prescribed target distribution, such as an isotropic Gaussian or the uniform distribution on a hypersphere. Klindt et al. (2026) showed that, under their Euclidean assumptions, matching a Gaussian target can recover Gaussian latent variables up to a linear transformation, and that the Gaussian is the unique distribution with this guarantee. We extend their analysis to latent variables supported on embedded Riemannian manifolds and derive conditions on the latent geometry and positive-pair dynamics under which alignment and exact distribution matching guarantee linear recovery. In particular, when the latent variables are uniformly distributed on a sphere and the representations are matched to the same spherical distribution, every optimal representation recovers the latent state up to an orthogonal transformation. This shows that Gaussian uniqueness is not a universal property of distribution-matched JEPAs: non-Euclidean latent geometries can admit other linearly recoverable distributions. We further derive an approximate-recovery bound that is strictly tighter for the spherical world than for the Gaussian world. Experiments on Gaussian, spherical, and toroidal latent spaces show that geometrically compatible targets yield better linear recovery when optimization succeeds, whereas mismatched targets distort the latent structure. This advantage persists in high-dimensional Clifford-torus worlds.
Léo Nicollier, Enric Meinhardt-Llopis, Marc Pic +2
CB, ATT · Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · Advanced Track and Trace +5
Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA
Shuwen Yu, Zhanxuan Hu, Yi Zhao +2
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China