Beyond Unimodal Bases: Pullback Geometry for Multimodal Data

Honglei Brinkmann; Lucas Ng; Georgios Batzolis; Mark Girolami; Carola-Bibiane Schönlieb; Willem Diepeveen

cs.LGSep 30, 2026

Beyond Unimodal Bases: Pullback Geometry for Multimodal Data

Authors: Honglei Brinkmann, Lucas Ng, Georgios Batzolis, Mark Girolami, Carola-Bibiane Schönlieb, Willem Diepeveen

Organizations: University of Cambridge Cambridge, UK · University of California Los Angeles, USA

Abstract

Data-driven Riemannian geometry provides nonlinear interpolation and geometric representations of high-dimensional data. For these operations to be statistically meaningful, paths between observations should preferentially traverse high-likelihood regions. Existing scalable pullback constructions typically use a unimodal Gaussian latent distribution, assuming that the data reside close to a single manifold. For multimodal data, mapping separated modes or local structures into one Gaussian region can require substantial transport deformation and compromise the resulting geometry. We introduce a pullback geometry for data supported on mixtures of manifolds. Using a latent Gaussian mixture, we define its Riemannian metric as the matrix square of the responsibility-weighted expected component precision. The metric is smooth and positive definite and recovers the existing Gaussian construction in the single-component limit. For structured overlapping mixtures, we establish conditions under which the log-density is concave along geodesics, providing a formal connection between the proposed geometry and paths through high-likelihood regions, and derive the corresponding local curvature relations. We instantiate this geometry in a normalizing flow with adaptive mixture learning, allowing the number of active components to emerge from the data and supporting component-wise reconstruction and local effective-dimension estimation. Experiments on synthetic geometric data, a controlled multi-view image setting with a known reference trajectory, and MNIST show reduced transport distortion, competitive path support, close reference-trajectory recovery, and improved interpolation realism. These results extend scalable pullback geometry beyond datasets that reside close to a single manifold while retaining tractable and interpretable local structure.

Figures & tables

Figure 1: Motivating Wavy-X example. Learned latent representations (top) and mapped discrete geodesic approximations (bottom) for two shared endpoint pairs. SBPRG and the fixed single-Gaussian control concentrate the intersecting structures into one latent region. A fixed two-component mixture is more expressive but does not consistently recover paths along the observed branches. The adaptive mixture retains two active components, organizes the data into distinct latent directions, and follows the displayed branches more closely.

Figure 2: For this $d=2,K=3$ mixture of low central covariance $(-0.823)$ , a large (non-red) region satisfies geodesic log-concavity; a geodesic follows the arms.

Model	NLL $\downarrow$	$D_{\mathrm{iso}}\downarrow$	Q95 $\kappa_{J}\downarrow$	$K_{\mathrm{active}}$	$L_{\star}\downarrow$	$\lambda_{\max}(\mathrm{Ric}_{\star}(0))$	Path 1 support $\downarrow$	Path 2 support $\downarrow$
SBPRG ( $K=1$ )	1.5777	2.9788	40.9951	1	1.3863	0.0000	0.3262	0.3320
Fixed $m=1$	1.6078	9.1065	28.2596	1	1.3863	$\approx 0$	0.3396	0.2905
Fixed $m=2$	1.6893	10.2015	28.0523	2	1.3863	$\approx 0$	1.1756	2.5638
Adaptive $m=20$	1.5590	1.8943	4.1490	2	$\mathbf{6.47\!\times\!10^{-17}}$	-37.2766	0.2794	0.3158

Table 1: Wavy-X quantitative comparison. Lower is better for NLL, distortion, conditioning,

L_{\star}

, and support deviation. Paths are densely sampled discrete geodesic approximations under

g_{Z}=P^{2}

Panel of Figure 3 — Figure 3: BlueWhale viewpoint paths. Image sequences between the same endpoints: (a) the metadata-defined reference trajectory; (b) the adaptive-GMM support-graph path; (c) the decoded SBPRG path; and (d) straight latent interpolation. Panel (b) consists of observed samples, whereas panels (c) and (d) are continuously decoded.

Path	MSE $\downarrow$	MAE $\downarrow$	Mean SSIM $\uparrow$	Worst-step MSE $\downarrow$
Adaptive-GMM support graph	0.000306	0.003897	0.993647	0.001257
SBPRG decoded path	0.010547	0.032986	0.835206	0.022814
Straight latent path	0.007188	0.026982	0.859611	0.013496

Table 2: BlueWhale path comparison. Image-space agreement with the metadata-defined reference sequence. The adaptive path is constrained to observed samples and is therefore not directly comparable to the continuously decoded alternatives.

Panel of Figure 4 — Figure 4: MNIST path interpolation ( $T=16$ ; rows: straight latent / SBPRG / adaptive GMM). Top: $4\to 6$ . The adaptive sequence remains recognizable, whereas the alternatives contain visibly noisy intermediate frames. Bottom: $6\to 7$ . The adaptive path passes through a digit-4-like region of the learned representation.

Method	Feature FD $\downarrow$	PPL $\downarrow$	PDV rel $\downarrow$
Straight latent	265.218	104.667	0.166
SBPRG	311.615	101.823	0.120
Adaptive GMM (ours)	144.432	164.663	0.426

Table 3: MNIST path interpolation. Classifier-feature Fréchet distance is the primary measure of interior-sample realism. The adaptive GMM improves this measure at the cost of greater perceptual path length and step-size variation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

r_{nk}^{(T_{e})}=\operatorname{softmax}_{k}\left(\frac{\log w_{k}+\log\mathcal{N}(z_{n};\mu_{k},A_{k})}{T_{e}}\right)

Appendix

Algorithm 1 Stabilized Training of an Adaptive Multimodal Flow

z_{i}=\left(1-\frac{i}{T}\right)z_{0}+\frac{i}{T}z_{T},\qquad i=1,\ldots,T-1;

Appendix

Algorithm 2 Metric-Based Latent Geodesic Computation

Figure 5: Component-wise linear reconstruction on Wavy-X. Original samples and responsibility-weighted linear reconstructions for the two retained components. One principal direction explains $99.41\%$ and $99.29\%$ of their respective weighted data-space variance.

Figure 6: Layer-wise transport on Wavy-X. Matched samples are traced from the data space at flow depth zero (blue), through the intermediate flow layers (black), to the final latent representation (red). From left to right: SBPRG with $K=1$ , the fixed $m=1$ single-Gaussian model, the fixed $m=2$ Gaussian-mixture model, and the adaptive model initialized with $m=20$ candidate components. The visualization shows how each learned flow reorganizes the same Wavy-X observations and complements the quantitative transport-distortion comparison.

Panel of Figure 7 — Figure 7: Tetrahedral-star metric paths. Top: latent representations and paths. Bottom: corresponding paths mapped to data space. Red and purple curves denote the two prescribed endpoint pairs.

Panel of Figure 8 — Figure 8: Additional tetrahedral-star diagnostics. (a) Component-wise reconstructions summarize the dominant local linear structure. (b) Layer-wise trajectories visualize how each flow transports the data toward its latent base.

Model	NLL $\downarrow$	$D_{\mathrm{iso}}\downarrow$	Q95 $\kappa_{J}\downarrow$	$K_{\mathrm{eff}}$	$L_{\star}\downarrow$	$\lambda_{\max}(\mathrm{Ric}_{\star}(0))$	Support 1 $\downarrow$	Support 2 $\downarrow$
SBPRG ( $K=1$ )	-1.7634	4.6871	61.5166	1.0000	2.0794	0.0000	0.3970	0.4137
Fixed $m=1$	-1.1533	21.3192	12.2771	1.0000	2.0794	$\approx 0$	0.4329	1.1875
Fixed $m=2$	-1.1440	23.5916	11.3507	2.0000	2.0794	$\approx 0$	0.5142	0.6308
Adaptive $m=20$	-1.7787	2.8724	5.9996	1.9988	$<10^{-6}$	-14.8535	0.4329	0.4067

Appendix

Table 4: Tetrahedral-star quantitative comparison. Lower is better for NLL, isometry distortion, upper-tail Jacobian conditioning, the star–Ricci diagnostic

L_{\star}

, and support deviation.

Figure 9: Component-wise local reconstruction on MNIST. Top: one representative sample per digit class. Bottom: oracle best-subspace reconstruction with pixel MSE. The model retains $K_{\mathrm{active}}=13$ active components from an initial pool of 30; digit labels are not used during training or subspace fitting.

Explore similar work

GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning
Sep 20, 2026Zechang Xiong, Da Li, Rong Yin +7Multimodal Learning Multimodal Classification
Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Amap, Alibaba Group, Beijing, China +3
Riemannian Metric Matching for Scalable Geometric Modeling of Distributions
Jun 12, 2026Jacob Bamberger, Adam Gosztolai, Pierre Vandergheynst +2Riemannian Manifolds Riemannian Flow Matching
Department of Computer Science, University of Oxford, Oxford, UK · Institute of Artificial Intelligence, Medical University of Vienna, Vienna, Austria · EPFL, Lausanne, Switzerland +1
Riemannian Archetypal Analysis: Interpretable non-linear data analysis on deformed star distributions
May 22, 2026Willem Diepeveen, Deanna NeedellRiemannian Manifolds Nonlinearity
Department of Mathematics University of California, Los Angeles Los Angeles, CA 90095, USA

Sep 20, 2026, cs.CVJ/K move · Enter open · S save

GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning

Zechang Xiong, Da Li, Rong Yin, Kexin Tang+6

Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Amap, Alibaba Group, Beijing, China+3

Multimodal Learning Multimodal Classification Modalities Geogs-Slam

Open paper AskPDF arXiv

Multimodal classifiers can converge to modality-dominant solutions in which one modality dominates the joint prediction, suppressing the learning of others. Existing balancing methods mainly adjust losses, gradients, or modality contributions, largely treating modality imbalance as an optimization problem while implicitly treating the weak modality as under-optimized but representationally intact. In this work, we find that this assumption does not always hold, as persistent modality dominance can induce a representation-level collapse of the weak modality, which we term \emph{manifold modality collapse} (MMC). MMC manifests as a coupled geometric degradation in which weak-modality representations collapse onto fewer directions within each class and become less separable across classes. Motivated by this observation, we propose \emph{GeoBalance}, a geometry-aware framework that monitors these two geometric properties and reconstructs the weak modality representation only when it exhibits signs of MMC. Once triggered, GeoBalance uses a fixed Simplex-ETF class scaffold and spectral regularization to restore class separation while preventing collapse onto a few feature directions. To preserve reconstruction during joint training, asymmetric gradient projection removes the joint-gradient component conflicting with reconstruction, leaving non-conflicting optimization unchanged. Extensive experiments across six multimodal benchmarks demonstrate great improvements over competitive balancing methods, validating its effectiveness.

Abstract

Explore similar work

GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning

Riemannian Metric Matching for Scalable Geometric Modeling of Distributions

Riemannian Archetypal Analysis: Interpretable non-linear data analysis on deformed star distributions