Beyond Unimodal Bases: Pullback Geometry for Multimodal Data

Honglei Brinkmann; Lucas Ng; Georgios Batzolis; Mark Girolami; Carola-Bibiane Schönlieb; Willem Diepeveen

cs.LGSep 30, 2026

Beyond Unimodal Bases: Pullback Geometry for Multimodal Data

Authors: Honglei Brinkmann, Lucas Ng, Georgios Batzolis, Mark Girolami, Carola-Bibiane Schönlieb, Willem Diepeveen

Organizations: University of Cambridge Cambridge, UK · University of California Los Angeles, USA

Abstract

Data-driven Riemannian geometry provides nonlinear interpolation and geometric representations of high-dimensional data. For these operations to be statistically meaningful, paths between observations should preferentially traverse high-likelihood regions. Existing scalable pullback constructions typically use a unimodal Gaussian latent distribution, assuming that the data reside close to a single manifold. For multimodal data, mapping separated modes or local structures into one Gaussian region can require substantial transport deformation and compromise the resulting geometry. We introduce a pullback geometry for data supported on mixtures of manifolds. Using a latent Gaussian mixture, we define its Riemannian metric as the matrix square of the responsibility-weighted expected component precision. The metric is smooth and positive definite and recovers the existing Gaussian construction in the single-component limit. For structured overlapping mixtures, we establish conditions under which the log-density is concave along geodesics, providing a formal connection between the proposed geometry and paths through high-likelihood regions, and derive the corresponding local curvature relations. We instantiate this geometry in a normalizing flow with adaptive mixture learning, allowing the number of active components to emerge from the data and supporting component-wise reconstruction and local effective-dimension estimation. Experiments on synthetic geometric data, a controlled multi-view image setting with a known reference trajectory, and MNIST show reduced transport distortion, competitive path support, close reference-trajectory recovery, and improved interpolation realism. These results extend scalable pullback geometry beyond datasets that reside close to a single manifold while retaining tractable and interpretable local structure.

Figures & tables

Figure 1: Motivating Wavy-X example. Learned latent representations (top) and mapped discrete geodesic approximations (bottom) for two shared endpoint pairs. SBPRG and the fixed single-Gaussian control concentrate the intersecting structures into one latent region. A fixed two-component mixture is more expressive but does not consistently recover paths along the observed branches. The adaptive mixture retains two active components, organizes the data into distinct latent directions, and follows the displayed branches more closely.

Figure 2: For this $d=2,K=3$ mixture of low central covariance $(-0.823)$ , a large (non-red) region satisfies geodesic log-concavity; a geodesic follows the arms.

Model	NLL $\downarrow$	$D_{\mathrm{iso}}\downarrow$	Q95 $\kappa_{J}\downarrow$	$K_{\mathrm{active}}$	$L_{\star}\downarrow$	$\lambda_{\max}(\mathrm{Ric}_{\star}(0))$	Path 1 support $\downarrow$	Path 2 support $\downarrow$
SBPRG ( $K=1$ )	1.5777	2.9788	40.9951	1	1.3863	0.0000	0.3262	0.3320
Fixed $m=1$	1.6078	9.1065	28.2596	1	1.3863	$\approx 0$	0.3396	0.2905
Fixed $m=2$	1.6893	10.2015	28.0523	2	1.3863	$\approx 0$	1.1756	2.5638
Adaptive $m=20$	1.5590	1.8943	4.1490	2	$\mathbf{6.47\!\times\!10^{-17}}$	-37.2766	0.2794	0.3158

Table 1: Wavy-X quantitative comparison. Lower is better for NLL, distortion, conditioning,

L_{\star}

, and support deviation. Paths are densely sampled discrete geodesic approximations under

g_{Z}=P^{2}

Panel of Figure 3 — Figure 3: BlueWhale viewpoint paths. Image sequences between the same endpoints: (a) the metadata-defined reference trajectory; (b) the adaptive-GMM support-graph path; (c) the decoded SBPRG path; and (d) straight latent interpolation. Panel (b) consists of observed samples, whereas panels (c) and (d) are continuously decoded.

Path	MSE $\downarrow$	MAE $\downarrow$	Mean SSIM $\uparrow$	Worst-step MSE $\downarrow$
Adaptive-GMM support graph	0.000306	0.003897	0.993647	0.001257
SBPRG decoded path	0.010547	0.032986	0.835206	0.022814
Straight latent path	0.007188	0.026982	0.859611	0.013496

Table 2: BlueWhale path comparison. Image-space agreement with the metadata-defined reference sequence. The adaptive path is constrained to observed samples and is therefore not directly comparable to the continuously decoded alternatives.

Panel of Figure 4 — Figure 4: MNIST path interpolation ( $T=16$ ; rows: straight latent / SBPRG / adaptive GMM). Top: $4\to 6$ . The adaptive sequence remains recognizable, whereas the alternatives contain visibly noisy intermediate frames. Bottom: $6\to 7$ . The adaptive path passes through a digit-4-like region of the learned representation.

Method	Feature FD $\downarrow$	PPL $\downarrow$	PDV rel $\downarrow$
Straight latent	265.218	104.667	0.166
SBPRG	311.615	101.823	0.120
Adaptive GMM (ours)	144.432	164.663	0.426

Table 3: MNIST path interpolation. Classifier-feature Fréchet distance is the primary measure of interior-sample realism. The adaptive GMM improves this measure at the cost of greater perceptual path length and step-size variation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

r_{nk}^{(T_{e})}=\operatorname{softmax}_{k}\left(\frac{\log w_{k}+\log\mathcal{N}(z_{n};\mu_{k},A_{k})}{T_{e}}\right)

Appendix

Algorithm 1 Stabilized Training of an Adaptive Multimodal Flow

z_{i}=\left(1-\frac{i}{T}\right)z_{0}+\frac{i}{T}z_{T},\qquad i=1,\ldots,T-1;

Appendix

Algorithm 2 Metric-Based Latent Geodesic Computation

Figure 5: Component-wise linear reconstruction on Wavy-X. Original samples and responsibility-weighted linear reconstructions for the two retained components. One principal direction explains $99.41\%$ and $99.29\%$ of their respective weighted data-space variance.

Figure 6: Layer-wise transport on Wavy-X. Matched samples are traced from the data space at flow depth zero (blue), through the intermediate flow layers (black), to the final latent representation (red). From left to right: SBPRG with $K=1$ , the fixed $m=1$ single-Gaussian model, the fixed $m=2$ Gaussian-mixture model, and the adaptive model initialized with $m=20$ candidate components. The visualization shows how each learned flow reorganizes the same Wavy-X observations and complements the quantitative transport-distortion comparison.

Panel of Figure 7 — Figure 7: Tetrahedral-star metric paths. Top: latent representations and paths. Bottom: corresponding paths mapped to data space. Red and purple curves denote the two prescribed endpoint pairs.

Panel of Figure 8 — Figure 8: Additional tetrahedral-star diagnostics. (a) Component-wise reconstructions summarize the dominant local linear structure. (b) Layer-wise trajectories visualize how each flow transports the data toward its latent base.

Model	NLL $\downarrow$	$D_{\mathrm{iso}}\downarrow$	Q95 $\kappa_{J}\downarrow$	$K_{\mathrm{eff}}$	$L_{\star}\downarrow$	$\lambda_{\max}(\mathrm{Ric}_{\star}(0))$	Support 1 $\downarrow$	Support 2 $\downarrow$
SBPRG ( $K=1$ )	-1.7634	4.6871	61.5166	1.0000	2.0794	0.0000	0.3970	0.4137
Fixed $m=1$	-1.1533	21.3192	12.2771	1.0000	2.0794	$\approx 0$	0.4329	1.1875
Fixed $m=2$	-1.1440	23.5916	11.3507	2.0000	2.0794	$\approx 0$	0.5142	0.6308
Adaptive $m=20$	-1.7787	2.8724	5.9996	1.9988	$<10^{-6}$	-14.8535	0.4329	0.4067

Appendix

Table 4: Tetrahedral-star quantitative comparison. Lower is better for NLL, isometry distortion, upper-tail Jacobian conditioning, the star–Ricci diagnostic

L_{\star}

, and support deviation.

Figure 9: Component-wise local reconstruction on MNIST. Top: one representative sample per digit class. Bottom: oracle best-subspace reconstruction with pixel MSE. The model retains $K_{\mathrm{active}}=13$ active components from an initial pool of 30; digit labels are not used during training or subspace fitting.

Explore similar work

Sep 20, 2026cs.CV

GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning

Multimodal classifiers can converge to modality-dominant solutions in which one modality dominates the joint prediction, suppressing the learning of others. Existing balancing methods mainly adjust losses, gradients, or modality contributions, largely treating modality imbalance as an optimization problem while implicitly treating the weak modality as under-optimized but representationally intact. In this work, we find that this assumption does not always hold, as persistent modality dominance can induce a representation-level collapse of the weak modality, which we term \emph{manifold modality collapse} (MMC). MMC manifests as a coupled geometric degradation in which weak-modality representations collapse onto fewer directions within each class and become less separable across classes. Motivated by this observation, we propose \emph{GeoBalance}, a geometry-aware framework that monitors these two geometric properties and reconstructs the weak modality representation only when it exhibits signs of MMC. Once triggered, GeoBalance uses a fixed Simplex-ETF class scaffold and spectral regularization to restore class separation while preventing collapse onto a few feature directions. To preserve reconstruction during joint training, asymmetric gradient projection removes the joint-gradient component conflicting with reconstruction, leaving non-conflicting optimization unchanged. Extensive experiments across six multimodal benchmarks demonstrate great improvements over competitive balancing methods, validating its effectiveness.

Zechang Xiong, Da Li, Rong Yin +7

Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Amap, Alibaba Group, Beijing, China +3

Multimodal Learning Multimodal Classification Modalities Geogs-Slam

Ask PDF

Jun 12, 2026cs.LG

Riemannian Metric Matching for Scalable Geometric Modeling of Distributions

High-dimensional datasets often concentrate near low-dimensional structures, but estimating their geometry from samples typically relies on graphs and kernels that scale poorly with dataset size and dimension. We propose Riemannian metric matching: a denoising probabilistic framework for learning the Riemannian geometry of data using neural networks. Specifically, we learn the carré du champ operator, which, using diffusion geometry, gives us access to the Riemannian geometry toolkit for downstream machine learning and statistical tasks. Our key observation is that the carré du champ operator can be formulated as a conditional expectation over random perturbations of the data, which can be exploited for sample-wise training and constant cost, amortized inference without explicit kernel construction. Empirically, metric matching rivals or improves the accuracy of

k

-NN-based diffusion geometry estimators, while enabling amortized inference that is up to

400\times

faster, and supports graph-free geometric analysis on high-dimensional images where nearest neighbors break down.

Jacob Bamberger, Adam Gosztolai, Pierre Vandergheynst +2

Department of Computer Science, University of Oxford, Oxford, UK · Institute of Artificial Intelligence, Medical University of Vienna, Vienna, Austria · EPFL, Lausanne, Switzerland +1

Riemannian Manifolds Riemannian Flow Matching High-Dimensional Distributions

Ask PDF

May 22, 2026cs.LG

Riemannian Archetypal Analysis: Interpretable non-linear data analysis on deformed star distributions

Classical archetypal analysis is appealing for its interpretability, but its linear geometry can limit performance on data with strongly non-linear structure; at the same time, existing neural extensions improve flexibility while often weakening the geometric meaning of archetypes and interpolations. In this work, we develop a Riemannian version of archetypal analysis based on data-driven pullback geometry for real-valued data, with the goal of combining the interpretability of classical archetypal analysis with the expressive power of modern non-linear models. We introduce a class of deformed star distributions together with associated pullback Riemannian geometry to provide a statistical interpretation of the resulting manifold mappings, define the Riemannian archetypal mapping (RAM) as a projection onto the manifold of geodesically convex combinations of archetypes, and propose a practical optimization scheme based on convex relaxation followed by non-convex refinement. We further propose a learning scheme that yields reasonable, albeit generally suboptimal, deformed star distributions from data. Experiments on synthetic examples and MNIST show that the resulting framework produces meaningful geodesics, useful denoising projections, and geometry-aware classifications, while also clarifying where current optimization limitations remain.

Willem Diepeveen, Deanna Needell

Department of Mathematics University of California, Los Angeles Los Angeles, CA 90095, USA

Riemannian Manifolds Nonlinearity Distributions Interpretability

Ask PDF