Latent Space Geometry

Follow this topic Ask about this topic

Latest papers

29 papers8 papers in the last four weeks, up 60% on the four weeks before. 0.0% of all new papers.

Oct 24, 2026

Stable Dimensionality Reduction via Isometric Centroid-Encoder

Many supervised dimensionality reduction methods promote intra-class compactness and inter-class separation but do not explicitly constrain class geometry, leading to variability in the relative arrangement of classes across independent runs. Such variability limits the reproducibility and interpretability of low-dimensional embeddings. We propose the Isometric Centroid-Encoder (ICE), a supervised dimensionality reduction framework in which class centroids are learned directly in the latent space while preserving a prescribed pairwise class-centroid geometry. Unlike existing centroid- or prototype-based methods that primarily encourage intra-class compactness, ICE explicitly regularizes the relative arrangement of latent class centroids. Unlike multidimensional scaling and many metric-learning approaches that preserve sample-level relationships, ICE operates on class centroids, preserving inter-class geometry without requiring large numbers of sample-pair or triplet constraints. We further show that the intrinsic dimensionality required to represent the centroid-distance configuration of

M

classes is at most

M-1

, explaining the observed saturation of distance-preservation measures as the latent dimensionality increases. To quantify embedding reproducibility, we introduce the Centroid Neighborhood Stability (CNS) metric, which measures the consistency of class-level neighborhood topology across independent runs and complements existing embedding quality measures. Experiments on image and biological datasets demonstrate that ICE produces stable embeddings with improved preservation of class-centroid geometry while remaining computationally efficient. Additional analyses of optimization dynamics, sensitivity, and preprocessing further provide practical insights into latent dimensionality and explicit geometry modeling for supervised dimensionality reduction.

Tomojit Ghosh, Siva Rama Krishna Prasad Changala, Prathyusha Kanakamalla +1

Colonel Glenn Hwy, Dayton, OH 45435-0001, USA Department of Computer Science and Engineering Wright State University

Dimensionality Reduction Latent Space Geometry

Oct 20, 2026

When Do Gaussian Priors Fail? A Latent Geometry Investigation of RL Fine-Tuning

Latent diffusion models almost universally place a Gaussian prior on the VAE latent space. For base generation this default is unproblematic; under reinforcement-learning fine-tuning, however, it carries a structural limitation. The standard VAE's KL term does not merely ignore mode structure - it actively taxes separation and tightness. When data are multimodal, the ELBO optimum compresses distinct modes into an overlapping latent. RL fine-tuning amplifies this overlap: the score network is under-resolved on the saddle region between modes, dynamic-to-variational amplification rises sharply, and the collapse boundary beta_c is pushed far above its variational scale. On the real alanine-dipeptide Boltzmann distribution, a Gaussian-prior VAE collapses at beta_c = 0.224 while a mode-tracked Gaussian-mixture prior collapses at 0.00316, a 71x gap from the same pipeline. We trace this gap to a single geometric mediator, d_min/(2 sigma_0), and verify the causal chain through controlled interventions: moving geometry at fixed prior, and switching prior at matched geometry (full mediation; prior-label coefficient 0.995x, 95% CI [0.79, 1.25]). A three-source decomposition localises the amplification to the learned policy (zero ODE-solver contribution, bounded anchor mismatch); the coverage plateau is an unconditional benefit of mode-separating priors that no amount of RL capacity recovers. The remedy is conditional: a non-Gaussian prior is necessary precisely when the latent geometry would fall into overlap and the task is coverage-sensitive. A machine-checked theory skeleton supports the KL pricing structure - a divergent compactness tax under a Gaussian prior versus a finite assignment fee under a mixture - and the position-versus-plateau distinction delimits what is finite-resource from what requires changing the prior.

Wang Ru

Department of Chemical Engineering, Tsinghua University

Latent Space Geometry Diffusion Model Fine-Tuning RL for Diffusion Models Neural Representation Geometry

Oct 16, 2026

Learning to Be Guided: Shaping Latent Geometry for Classifier-Free Guidance

Classifier-free guidance (CFG) is a standard component of conditional diffusion models, and guided generation performance has become the de facto measure of state-of-the-art models. Recent work on latent diffusion models seeks to learn latent spaces that are easier to model with diffusion. However, under CFG, generation additionally depends on how conditions are arranged in the latent space. We study this arrangement under discrete class conditions, termed class geometry, and investigate what class geometry allows a latent space to work well with CFG. Under a Gaussian class model, we show that shifting variance from intra- to inter-class at fixed total variance makes classes more distinguishable under noise, and derive a noisy posterior objective whose per-sample descent direction is proportional to the oracle CFG correction. Based on the analysis, we propose Prototype Regularization (PR), a lightweight surrogate for the intractable noisy posterior objective, which applies a cosine prototype classifier to spatially pooled latents. With a controlled training budget (20 epochs for VAE, 80 epochs for DiT), adding PR to an LDM baseline reduces guided gFID on ImageNet from 3.31 to 2.28, surpassing a DINOv3-aligned VA-VAE (2.44) with negligible overhead. We further demonstrate that PR generalizes to pseudo-labels derived from clustering, continuous text-embedding conditions, and video latents. Together, our findings highlight class geometry as a critical factor in guided generation and take a first step toward CFG-compatible representation learning. Our project page is available at: https://maps-research.github.io/cfg-geometry/.

Haoming Liu, Junhao Zhu, Shaoyi Zheng +4

New York University · New York University Shanghai · Georgia Institute of Technology

Latent Space Geometry Diffusion-Based Image Generation Classifier-Free Guidance Latent Diffusion Models Neural Representation Geometry

Oct 15, 2026

Native Geometry vs. Linear Recoverability: How Alignment Changes Platonic Scaling

Do larger models learn increasingly similar representations of the same world? The Platonic Representation Hypothesis (PRH) predicts that they should, but similar can mean native cosine geometry---nearest-neighbour structure induced by cosine similarity before any learned alignment---or linear recoverability: correspondence revealed by a supervised linear map. Using matched Legacy Survey and Hyper Suprime-Cam (HSC) embeddings across five model families, we find that Ridge alignment strengthens the relationship between model size and cross-dataset neighbourhood correspondence relative to native cosine geometry. The effect appears in all five families and persists when both representations are reduced to dimension, so it is not simply that wider embeddings give alignment more dimensions to work with. The learned maps remain far from rotations or uniform scaling, and Sparse Autoencoder (SAE) and Block-Sparse Featuriser (BSF) features give no advantage over the same alignment applied to dense embeddings. Model scale thus improves linear recoverability more clearly than native cosine-neighbourhood correspondence, without establishing convergence toward a shared native metric geometry.

Angus Hoy, Austin Lutterbach, Jasper Chong +2

EleutherAI SOAR · UniverseTBD · University of Queensland +4

Neural Representation Geometry Latent Space Geometry Platonic Representation Hypothesis Representation Alignment Latent Space Alignment Scaling Laws

Oct 13, 2026

Order-Consistent Latent Geometry for Reliable Short-Horizon Planning in World Models

Latent planners often rank candidate action sequences by terminal Euclidean distance to a goal latent. Exact agreement with transition distance would make this score ideal, but is stronger than planning requires: along a transition-shortest rollout, the essential requirement is that latent distance does not reverse true transition progress. We show that this pathwise no-reversal requirement is precisely the classical self-approaching condition. To the best of our knowledge, we are the first to identify self-approachingness as a planner-facing compatibility condition for terminal Euclidean latent planning and to connect its violation to an exact finite-horizon candidate-order reversal. Because visual world models predict discrete latent sequences, we analyze their polygonal interpolation. We prove that an acute bend creates a nonempty region in which terminal Euclidean distance ranks a transition-worse candidate ahead of one that makes genuine progress, even under exact rollouts. We further interpret latent dimension as angular capacity through a classical spherical-code bound. A controlled six-branch experiment, including a zero-padded three-dimensional baseline, validates the predicted ordering failure and separates angular arrangement from ambient dimensionality.

Junlin Wu, Wen Fan, Dandan Zhang

Imperial College London, London, United Kingdom

Latent Space Geometry Latent-Space Planning World Model-Based Planning Latent World Models Neural Representation Geometry

Oct 4, 2026

Do Neural Networks Learn Structure-Preserving Maps? A Case Study in Latent-to-Hilbert Embeddings

We ask whether a neural network can learn a structure-preserving map from a compressed latent space to a Hilbert-space representation. Using an 8-dimensional autoencoder bottleneck on MNIST and

n

-qubit product-state targets from PCA-based angle encoding, we report four findings. Although the target angles are generated by a nonlinear sigmoid transformation of the latent projections, the resulting mapping is well approximated by a linear function over the observed latent distribution: linear regression from

z

to the true target angles achieves

R^{2} = 0.98

, while regression to the MLP's recovered angles achieves

R^{2} = 0.91

. The learned map's primary direction is strongly aligned with the target-induced direction, with cosine similarity

0.989

, while remaining nearly orthogonal to the input's principal direction, with cosine similarity

0.002

. The map is genuinely rank-4: removing any singular direction degrades inner-product preservation by

2.5

--

5.9\times

despite a singular-value spectrum with two dominant and two small values. The learned subspace does not coincide with the PCA basis used to construct the target, and different random seeds recover the same primary direction but diverge in higher ranks. Finally, kernel ridge regression with an RBF kernel outperforms a tuned MLP (IP error

0.0144

vs.\

0.0197

), suggesting that for approximately linear structure-preserving mappings, classical kernel methods may be a simpler and more effective alternative.

Muhammad Adnan Shahzad

Department of Computer Science and Software Engineering (CSSE) Montreal, QC, Canada

Representation Learning Latent Space Alignment Structure-Preserving Neural Networks Latent Space Geometry

Oct 1, 2026

GeoLatent: Geometry-Guided Latent Structuring with Routed Optimization for 3D Reasoning

Despite progress in vision-language models, 3D spatial reasoning from 2D images remains challenging. Text-based methods describe intermediate geometry with discrete tokens, limiting fidelity for continuous spatial relations. Continuous latents offer richer representations, but a single latent type does not explicitly separate the cues needed across spatial tasks. Decomposed spatial latents address this by representing position, direction, and global geometry separately under geometric supervision. Yet the geometry representation can still collapse toward one dominant direction, and unrestricted attention can leave the latents underused during answer learning. We introduce GeoLatent, combining Common--Residual Geometry Alignment (CR-GEO) with routed optimization to structure the geometry states while promoting latent-mediated answer learning. CR-GEO separates shared from residual teacher geometry; routed optimization jointly trains geometry and language, temporarily directs visual answer learning through the latents, and restores full attention with geometry supervision. In controlled comparisons, CR-GEO raises geometry effective rank from 1.00 to 3.87, while blocking latent readout at the bottleneck lowers direction accuracy from 89.1% to 25.8% on 128 fixed questions. After recovery, the differentiated geometry representation and latent-mediated visual route remain available alongside direct image access. GeoLatent achieves 73.0% on SPAR-Bench and 72.1% on SPBench, outperforming previously reported methods on both.

Yakun Zhu, Yi Bin, Yujuan Ding +5

Tongji University · The Hong Kong Polytechnic University

3D Spatial Reasoning Geometric Representation Learning Latent Visual Reasoning 3D VLMs Latent Space Geometry

Sep 30, 2026

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to

\textit{translation symmetry}

in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group

G

, the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of

G

. Circles and Pringles arise when

G

is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group

\mathbb{Z}_{12}

: for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group (

T/I

) acting on chords (music theory), which

\textit{implies}

that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Liam Storan, Andreas Tolias, Nina Miolane

Stanford University · University of California, Santa Barbara

Word Embeddings Representation Geometry in Language Models Latent Space Geometry

Sep 26, 2026

Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis

Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by

\ell_2

objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.

Haowen Xue, Hao Chen, Hexuan Hu +6

Hohai University · University of Cambridge · National University of Defense Technology

Latent Space Geometry Medical Image Synthesis Latent Variable Models 3D Medical Imaging Neural Representation Geometry

Sep 18, 2026

Beyond Gaussian Worlds: Latent Geometry Matters for JEPAs

Recent Joint-Embedding Predictive Architectures (JEPAs) prevent representation collapse by constraining learned representations to follow a prescribed target distribution, such as an isotropic Gaussian or the uniform distribution on a hypersphere. Klindt et al. (2026) showed that, under their Euclidean assumptions, matching a Gaussian target can recover Gaussian latent variables up to a linear transformation, and that the Gaussian is the unique distribution with this guarantee. We extend their analysis to latent variables supported on embedded Riemannian manifolds and derive conditions on the latent geometry and positive-pair dynamics under which alignment and exact distribution matching guarantee linear recovery. In particular, when the latent variables are uniformly distributed on a sphere and the representations are matched to the same spherical distribution, every optimal representation recovers the latent state up to an orthogonal transformation. This shows that Gaussian uniqueness is not a universal property of distribution-matched JEPAs: non-Euclidean latent geometries can admit other linearly recoverable distributions. We further derive an approximate-recovery bound that is strictly tighter for the spherical world than for the Gaussian world. Experiments on Gaussian, spherical, and toroidal latent spaces show that geometrically compatible targets yield better linear recovery when optimization succeeds, whereas mismatched targets distort the latent structure. This advantage persists in high-dimensional Clifford-torus worlds.

Léo Nicollier, Enric Meinhardt-Llopis, Marc Pic +2

CB, ATT · Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · Advanced Track and Trace +5

Joint-Embedding Predictive Architecture Latent Space Geometry Neural Representation Geometry Latent Variable Models Hyperspherical Representation Learning

Sep 16, 2026

Learning Submanifolds for Subsequent Inference on Random Dot Product Graphs, Part 1: Theory

We propose a framework for restricted inference on random dot product graphs whose latent positions lie on an unknown low-dimensional support manifold. For general decision problems, we propose semisupervised decision rules that use auxiliary data to learn the support manifold. Specifically, our rules use the Isomap manifold learning procedure to construct a low-dimensional Euclidean representation of the observed graph, in which space an isometrically invariant function maps configurations of points to actions. We study the behavior of the proposed rules as the quantity of auxiliary data sampled from the unknown support manifold increases. We show that, as the auxiliary sample size increases, the risk of the semisupervised rule converges to the risk of an oracle rule that relies on the maximal amount of low-dimensional Euclidean structure that can be extracted from the support manifold. Examples, applications, and simulation studies are deferred to a sequel.

Michael W. Trosset, Carey E. Priebe

Department of Statistics, Indiana University. · Department of Applied Mathematics & Statistics, Johns Hopkins University.

Semi-Supervised Learning Decision-Making under Uncertainty Latent Space Geometry

Sep 14, 2026

$\mathbb{SL}(n)$ Representation Learning: An Intrinsic Mixed-Curvature Space with Higher Curvature Capacities and Deeper Order-Aware Composition

Mixed-curvature representation learning seeks to capture rich geometric structures that cannot be adequately modeled by a single curvature regime. Existing approaches largely rely on product manifolds, which require manually specifying how different curvature spaces are combined and separate their curvature contributions across factors. We introduce the

\mathbb{SL}(n)

space, a representation geometry defined by the simple

\det(A)=1

constraint and a left invariant Schatten-

p

Finsler structure. Despite this minimal construction,

\mathbb{SL}(n)

exhibits pointwise negative, zero, and positive flag curvature around a common flagpole, while its mixed-curvature and curvature-coupling capacities are asymptotically maximal relative to the intrinsic geometric upper bound. Beyond geometry, its noncommutative group structure provides inherent order sensitivity, and its non-nilpotent Lie algebra admits nonzero nested Lie brackets at arbitrary depth, enabling deep order-aware composition. Empirically,

\mathbb{SL}(n)

consistently outperforms a broad range of representation manifold baselines across graph benchmarks at different scales. It reduces average distortion over the strongest baselines by

44.3\%

on KEGG and

40.5\%

on HumanCyc, and improves Hits@20 by

42.8\%

on OGBL-PPA. Experiments on Flickr30k-Order further support its ability to capture higher order dependencies from ordered composition. Together, these results show how a seemingly simple structural constraint can yield unexpectedly rich geometry, capacity, and composition within a unified representation space.

Xingrun Li, Yusuke Mukuta, Xin Yang +2

The University of Tokyo · Stanford University

Finsler Geometry Graph Representation Learning Geometric Representation Learning Lie Group Methods Latent Space Geometry

Aug 31, 2026

Latent-Space No-Arbitrage Geometry of Generative Models for Implied Volatility Surfaces

Generative models for implied volatility surfaces must produce outputs that satisfy static no-arbitrage constraints. We study these constraints in latent space. For a fixed generator, we assign each latent code a scalar margin determined by the no-arbitrage conditions of the generated surface. The codes with nonnegative margin form the admissible latent set. We establish conditions under which strictly admissible codes remain admissible under small perturbations and the boundary of the admissible set is characterized by zero margin. For regular boundary components, we formulate a level-set equation whose local dynamics are directed toward the zero-margin set. The analysis treats the generator as a map from latent variables to surfaces and is therefore not restricted to a particular architecture. It applies to variational autoencoders, generative adversarial networks, and other generative models with a deterministic realization map. Numerical tests recover known boundaries in analytic examples. Experiments with a variational autoencoder trained on Heston surfaces show that similar reconstruction errors can correspond to different admissible regions and that the latent prior may be concentrated inside such a region. The computed boundary can also be used to modify latent codes that generate violating surfaces.

Jing Wang, Shuaiqiang Liu, Cornelis Vuik

Delft Institute of Applied Mathematics, Delft University of Technology

Constrained Generative Modeling Neural Representation Geometry Quantitative Finance Latent Space Geometry

Aug 28, 2026

Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching

Independently trained neural networks tend to encode the same data with similar latent geometries. These latent geometries are not directly compatible, yet they can be nearly the same up to some class of transformations. While there exists many methods for alignment between different latent spaces, it is typically done using a set of shared sample correspondences, known as anchors. This leaves a fundamental question: are the geometric signatures of different latent spaces representing similar data sufficient to recover an alignment between them? To that end, we introduce HGA (Hyperspherical Gaussian Alignment), a method that directly optimizes a transformation between two latent spaces by maximizing a geometric measure of "fit" between them. Since it is driven by the geometry of the latent spaces rather than paired data, HGA can operate in both an unsupervised and weakly supervised regime. On tasks such as model stitching or multilingual word embedding correspondence recovery, HGA manages to match supervised results with minimal or no supervision.

Cameron Ryan, Vivek Sivaraman Narayanaswamy, Kowshik Thopalli +1

Northeastern University, Boston, MA 02115. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94551-0808.

Latent Space Alignment Latent Space Geometry Cross-Lingual Representation Alignment

Aug 25, 2026

Hyperbolic Latent Geometry for Tree-Structured Prototype Networks: A Local-vs-Global Trade-off

We study a tree-structured regularizer over class-prototype layouts in a hierarchical-classification model and ask whether the choice of latent manifold for the prototypes (Euclidean R^d vs. the Poincare ball B^d_c) affects how well that regularizer can be satisfied without distorting the data likelihood. The two manifolds differ only in their volume growth: hyperbolic space grows exponentially with radius and embeds trees with provably lower distortion than R^d of matched dimension, so the structured regularizer should be cheaper to satisfy on B^d_c. Across 150 seed-replicated regularized maximum-likelihood fits spanning embedding dimension, curvature, and regularizer strength on WikiArt (27 styles, 81,446 paintings, frozen CLIP ViT-B/16 features), we find a single robust effect: Poincare prototypes preserve the topology of the nearest-neighbor graph in latent space substantially better than matched Euclidean prototypes (sibling recall@5 +8.7 pp, cousin recall +15.2 pp; paired-t p < 10^-4, sign agreement 0.94), and the gap holds across three reference-tree definitions (hand-built lineage, CLIP-derived, and DINOv2-derived). On classification, Euclidean prototypes are tied with logistic regression on raw encoder features, indicating no detectable contribution from the latent geometry; only the hyperbolic fit improves on a k-NN encoder baseline for local retrieval. Global tree-fidelity comparisons are unstable across reference trees and we do not claim a winner. The results give an empirical separation, on a real hierarchical-classification problem, between two natural latent geometries for a class-structured regularizer.

Peter Flo, Luca Grossmann

Harvard University, Cambridge, MA, USA.

Latent Space Geometry Neural Representation Geometry Hyperbolic Representation Learning Prototype-Based Classification Hierarchical Classification

Aug 25, 2026

Beyond Uniform Local Isometry and Topology: FactoMap for Disentangled Representations

Many disentanglement methods represent generative factors using Euclidean product coordinates, although the underlying factor spaces may wrap, collapse, or have position-dependent geometry. We introduce factor-space structure, which combines factor domains, generator-induced identifications, and position-dependent scales to distinguish topologically equivalent spaces with different factor geometries. We show that statistical independence does not imply uniform product geometry: independent hue and scale can affect observations at different, scale-dependent rates, producing anisotropy that no fixed rescaling can remove. We propose the Factor-Space Topographic Map (FactoMap), which learns interpretable prototypes indexed by a structured factor-space lattice. Topographic learning transfers the lattice's periodicity, collapses boundaries, and non-uniform extent to the representation. Experiments show that matching this structure better preserves factor continuity and improves disentanglement of the underlying factors.

Sohini Gupta, Michael Gao, Bahareh Tolooshams

University of Alberta Alberta Machine Intelligence Institute (Amii) · University of Alberta Alberta Machine Intelligence Institute (Amii) Canada CIFAR AI Chair

Disentangled Representation Learning Latent Space Geometry Topological Representation Learning Topological Mapping

Aug 4, 2026

On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs

Recent audio restoration increasingly relies on large-scale conditional latent generative modeling, including diffusion, Schrodinger Bridges, and Flow Matching variants, to invert degradations such as bandwidth limitation or noise. We present an analysis of the performance of various state-of-the-art methods compared to simple arithmetic transformations in the latent spaces of multiple neural codecs for musical bandwidth extension. We show that estimating a single transport vector between the clean and degraded latent centroids on a reference set, and adding it to degraded latents, can yield restoration performance competitive with large diffusion models. This suggests, first, that some neural codec latent spaces exhibit structure aligned with audio bandwidth; and second, that in such cases complex conditional models may offer only limited gains over a simple vector addition. We argue that these findings reveal an interesting avenue for future research whereby models could take advantage of the latent space structure in order to offer greater training and parameter efficiency, and overall better performance. Additionally, we propose to consider this simple arithmetic transformation as a baseline for music bandwidth extension research, as it allows an assessment of the contribution of learnable parameters towards restoration performance.

Hendrik Vincent Koops, Hao Hao Tan, Elio Quinton

Music & Audio Machine Learning Lab · Universal Music Group, London, U.K.

Neural Audio Codecs Neural Representation Geometry Audio Diffusion Models Latent Space Geometry

Jun 29, 2026

I-BBS: Coordinate-Free Inference of Latent Sub-Manifolds Using Random Distance Matrix Theory

Bogomolny, Bohigas and Schmit (BBS) found that the spectrum of the pairwise distance matrix on N points sampled from a smooth d-dimensional manifold encodes a signature of the underlying geometry. We develop I-BBS (Inference-BBS), a coordinate-free method that identifies a low-dimensional latent sub-manifold embedded in a high-dimensional ambient distance matrix alone, without accessing an ambient high-dimensional vector space. It therefore applies even when that space is only partly observable or undefined. We model the ambient embedding by two classes of generative noise, model-based and model-free. The noise mixes the latent signal with off-manifold components, so the eigenvalues reorganise collectively and the latent geometry cannot be read off eigenvalue by eigenvalue. We recover it instead from two integer-stable signatures that survive the noise: the multiplicity of the top non-Perron multiplet, which fixes

d

, and a parameter-free law for how the multiplet positions shrink as the noise grows. On synthetic spheres

S^1

,

S^2

and

S^3

these integer signatures are far more stable under noise than the continuous spectral slope, and a blind test recovers both the manifold and the noise model from a single distance matrix. Applications to neural-network representations and to the dynamic training regime are developed in two companion papers.

Igor Halperin

Intrinsic Dimensionality Random Matrix Theory Manifold Learning Latent Space Geometry

Jun 5, 2026

Constructing VAE Latent Spaces with Prescribed Topology

Variational autoencoders (VAEs) learn low-dimensional latent representations of high-dimensional data. When the data lies on a manifold with non-Euclidean topology, the standard Gaussian prior introduces a topological mismatch that degrades reconstruction quality and prevents faithful representation. We present a constructive mathematical framework that resolves this mismatch for all manifolds that admit a product covering space. These are manifolds expressible as products of elementary factors (circles, intervals, or lines) or as quotients of such products by a finite symmetry group. The class includes cylinders, tori, Möbius strips, Klein bottles, and real projective spaces. Factorized distributions over the elementary factors yield product topologies with closed-form, decoupled KL divergences, so that each latent factor can be shaped independently while keeping training tractable. We catalogue reparametrizable encoder-prior pairs for periodic, bounded, and unbounded supports, and provide coordinate transformations that allow standard neural networks to output non-Euclidean parameters with smooth gradients. For quotient manifolds, the decoder receives group-invariant features of the covering-space coordinates, so that identified points produce identical outputs. Anchor constraints fix the coordinate system relative to the data or create soft topological holes. Experiments on synthetic manifolds and real-image datasets (rotated and cyclically shifted MNIST) confirm that a topology-matched prior aligns KL regularization with the data manifold. The resulting topology-aware models outperform the Gaussian baseline at all practically relevant regularization strengths. The code is available at https://github.com/JvHulst/VAE-Topology.

Jilles S. van Hulst, Jakub M. Tomczak, W. P. M. H. Heemels +1

Control Systems Technology Section, Department of Mechanical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands. · The research is carried out as part of the ITEA4 20216 ASIMOV project.

Variational Autoencoders Manifold Learning Topological Representation Learning Topological Mapping Latent Space Geometry

May 26, 2026

SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings

A fundamental open question in self-supervised learning (SSL) is the explicit characterization of the optimal geometry of the learned representations. Recently, LeJEPA identified isotropic Gaussian embeddings as optimal for minimizing downstream prediction risk in Euclidean spaces. However, the corresponding problem for distributions supported on lower-dimensional manifolds, such as the hypersphere, remains unexplored. In this work, we demonstrate that extending this minimax analysis to smooth distributions on Riemannian manifolds fundamentally changes the optimal solution. We show that, under a worst-case formulation, both k-nearest neighbors and kernel ridge regression induce hyperspherical uniformity. More precisely, we show that uniform distributions on manifolds are optimal for k-nearest neighbors, and that the uniform distribution on the sphere is optimal for kernel ridge regression with both the exponential dot-product kernel and the linear kernel. This theoretical insight reveals a fundamental limitation of Gaussian embeddings: their non-uniform density induces anisotropic k-NN neighborhoods, severely biasing the estimator. To correct this, we introduce SPHERE-JEPA, a theoretically grounded SSL framework. We adapt LeJEPA's Cram{é}r-Wold projection mechanism to enforce hyperspherical uniformity rather than a Gaussian prior. Empirically, SPHERE-JEPA yields significant improvements, boosting texture retrieval mAP by over 6%, while consistently matching or outperforming LeJEPA on standard benchmarks-including a +1.8% linear probing gain on ImageNet-1K (ViT-B/14).

L{é}o Nicollier, Max Dunitz, Marc Pic +3

CB,ATT · ATT · CB,IFUMI +1

Self-Supervised Learning Joint-Embedding Predictive Architecture Hyperspherical Representation Learning Geometric Representation Learning Manifold Learning Latent Space Geometry

May 21, 2026

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance

In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings.

Amanda Myntti, Jenna Kanerva, Veronika Laippala +1

TurkuNLP, University of Turku, Finland · ELLIS Institute Finland

Text Embedding Evaluation Text Embedding Models Latent Space Geometry

May 16, 2026

Scale Determines Whether Language Models Organize Representation Geometry for Prediction

In language models, what a representation encodes is determined by the geometry of its representation space: distances, not activations, carry meaning. Existing tools characterize the shape of this geometry but do not ask what that shape is organized for. We introduce Subspace PGA, a metric that tests whether a layer's distance structure aligns with the readout subspace of the unembedding matrix

W_U

more than with random subspaces of equal size. Across seven Pythia models (70M--6.9B) and three cross-family models, intermediate geometry is significantly organized for prediction (peak

z = 9

--

24

), but the degree is scale-dependent: small models (

d \leq 1024

) progressively lose it at late layers during training -- even as loss keeps improving -- while large models (

d \geq 2048

) preserve it throughout. We trace this to a capacity trade-off: a few dominant directions migrate away from

W_U

's readout, masking rather than destroying the predictive structure beneath, and removing them restores alignment. Neither spectral metrics nor loss curves capture this distinction. Scale thus determines not only how well a model predicts, but how its representation geometry is organized to do so.

Weilun Xu

School of Computer and Communication Sciences ´Ecole Polytechnique F´ed´erale de Lausanne

Language Model Pretraining Neural Representation Geometry Language Model Scaling Representation Geometry in Language Models Latent Space Geometry

May 14, 2026

Aligning Latent Geometry for Spherical Flow Matching in Image Generation

Latent flow matching for image generation usually transports Gaussian noise to variational autoencoder latents along linear paths. Both endpoints, however, concentrate in thin spherical shells, and a Euclidean chord leaves those shells even when preprocessing aligns their radii. By decomposing each latent token into radial and angular components, we show through component-swap probes that decoded perceptual and semantic content is carried predominantly by direction, with radius contributing much less. We therefore project data latents onto a fixed token radius, use the radial projection of Gaussian noise as the spherical prior, finetune the decoder with the encoder frozen, and replace linear interpolation with spherical linear interpolation. The resulting geodesic paths stay on the sphere at every timestep, and their velocity targets are purely angular by construction. Under matched training, the method consistently improves class-conditional ImageNet-256 FID across different image tokenizers, leaves the diffusion architecture unchanged, and requires no auxiliary encoder or representation-alignment objective.

Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe +2

Virginia Tech · fal

Flow Matching Neural Representation Geometry Conditional Image Generation Image Generation Flow-Based Generative Modeling Latent Space Geometry

Apr 20, 2026

Geometric Decoupling: Diagnosing the Structural Instability of Latent

Latent Diffusion Models (LDMs) achieve high-fidelity synthesis but suffer from latent space brittleness, causing discontinuous semantic jumps during editing. We introduce a Riemannian framework to diagnose this instability by analyzing the generative Jacobian, decomposing geometry into Local Scaling (capacity) and Local Complexity (curvature). Our study uncovers a **

Geometric Decoupling"**: while curvature in normal generation functionally encodes image detail, OOD generation exhibits a functional decoupling where extreme curvature is wasted on unstable semantic boundaries rather than perceptible details. This geometric misallocation identifies

Geometric Hotspots" as the structural root of instability, providing a robust intrinsic metric for diagnosing generative reliability.

Yuanbang Liang, Zhengwen Chen, Yu-Kun Lai

School of Computer Science and Informatics, Cardiff University, Cardiff, United Kingdom

Neural Representation Geometry Latent Diffusion Models Generative Modeling Latent Space Geometry Riemannian Geometry

Apr 3, 2026

Understanding Latent Diffusability via Fisher Geometry

Diffusion models often degrade in latent spaces, yet the formal causes remain poorly understood. We quantify latent-space diffusability via the rate of change of the Minimum Mean Squared Error (MMSE) along the diffusion trajectory. Our framework decomposes this MMSE rate into contributions from Fisher Information (FI) and Fisher Information Rate (FIR). We show that isometric embeddings preserve intrinsic FI and establish quantitative intrinsic-FI bounds for a broader class of bi-Lipschitz encoders with controlled weak volume distortion, whereas FIR is governed by the interplay between encoder and data geometries. Our analysis separates four geometric contributions in local stability bounds for Gaussian-smoothed FIR: dimensional compression, tangential distortion, high-frequency encoder curvature, and curvature of data manifold. Experiments across diverse autoencoding architectures provide qualitative support for the geometric mechanisms identified by the theory and show that empirical FI and FIR track several measures of generation quality and latent-space geometry in the settings tested. We establish FI and FIR as a comprehensive analytical framework for understanding latent diffusability.

Jing Gu, Morteza Mardani, Wonjun Lee +2

School of Mathematics, University of Minnesota · NVIDIA · Department of Mathematics, The Ohio State University +1

Information Geometry Neural Representation Geometry Latent Diffusion Models Diffusion Models Latent Space Geometry

Apr 1, 2026

Denoising distances beyond the volumetric barrier

We study the problem of reconstructing the latent geometry of a

d

-dimensional Riemannian manifold from a random geometric graph. While recent works have made significant progress in manifold recovery from random geometric graphs, and more generally from noisy distances, the precision of pairwise distance estimation has been fundamentally constrained by the volumetric barrier, namely the natural sample-spacing scale

n^{-1/d}

coming from the fact that a generic point of the manifold typically lies at distance of order

n^{-1/d}

from the nearest sampled point. In this paper, we introduce a novel approach, Orthogonal Ring Distance Estimation Routine (ORDER), which achieves a pointwise distance estimation precision of order

n^{-2/(d+5)}

up to polylogarithmic factors in

n

in polynomial time. This strictly beats the volumetric barrier for dimensions

d > 5

. As a consequence of obtaining pointwise precision better than

n^{-1/d}

, we prove that the Gromov--Wasserstein distance between the reconstructed metric measure space and the true latent manifold is of order

n^{-1/d}

. This matches the Wasserstein convergence rate of empirical measures, demonstrating that our reconstructed graph metric is asymptotically as good as having access to the full pairwise distance matrix of the sampled points. Our results are proven in a very general setting which includes general models of noisy pairwise distances, sparse random geometric graphs, and unknown connection probability functions.

Han Huang, Pakawut Jiradilok, Elchanan Mossel

Manifold Learning Latent Space Geometry

Mar 17, 2026

Manifold-Matching Autoencoders

We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).

Laurent Cheret, Vincent Létourneau, Isar Nejadgholi +3

Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada

Multidimensional Scaling Autoencoders Latent-Space Regularization Dimensionality Reduction Latent Space Geometry Latent Space Alignment+2

Feb 16, 2026

Symmetry in language statistics shapes the geometry of model representations

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2

1UC Berkeley · 2EPFL · 3Johns Hopkins +1

Representation Learning Neural Representation Geometry Word Embeddings Representation Geometry in Language Models Latent Space Geometry

Nov 4, 2025

Geometry as a Missing Axis of Representation Quality: The Variational Geometric Information Bottleneck under Data Scarcity

We study latent geometry as an explicit component of representation quality in data-scarce learning. For an encoder (φ), we define (Q_{β,γ}(φ)=I(φ(X);Y)-β\mathcal C(φ)-γd_{\mathrm{int}}(φ)), combining task-relevant information with penalties for curvature and intrinsic latent dimension. Thus geometry becomes part of the bottleneck criterion, not only a post hoc diagnostic. Under smooth-manifold, loss-transfer, and estimator-concentration assumptions, we derive non-asymptotic low-label generalization bounds where intrinsic dimension and covering complexity enter explicitly. We characterize the information--geometry frontier and prove empirical-surrogate consistency. The analysis links encoder geometry to learning through latent covering numbers, loss-class entropy, and uniform deviation. We instantiate the theory as V-GIB, adding curvature and dimension penalties to variational bottleneck training. Real low-label benchmarks compare V-GIB with ERM, VIB, and ablations across (1%)--(20%) label fractions. Results show improved performance and reduced geometric complexity in several regimes, especially FashionMNIST and CIFAR-10, while confirming that no fixed regularizer is universally dominant.

Ronald Katende

Department of Mathematics, Kabale University, Katuna Road, Kikungiri Hill, Kabale, 317, Uganda

Intrinsic Dimensionality Information Bottleneck Geometric Representation Learning Variational Information Bottleneck Data-Efficient Learning Latent Space Geometry