cs.LGOct 24, 2026

Variational Atlas Autoencoders and Self-Supervised Chart Selection

Authors: Arno Coomans, Merlin Nimier-David, Eugene d'Eon, Boris Bonev

Organizations: NVIDIA · Graz University of Technology, Austria

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Complex, high-dimensional data lie on manifolds that a single global latent coordinate system cannot represent without distortion. We introduce Variational Atlas Autoencoders, a mixture-of-charts framework that learns an atlas of local latent representations, each responsible for one region of the data manifold. The central difficulty in learning an atlas is deciding which chart should be responsible for which data point without labels. We show that this decision need not be a heuristic. By treating the chart index as a latent variable, the evidence lower bound yields the training objective and casts chart assignment as posterior inference, valid for any routing distribution. Within this framework, we propose latent-evidence routing, which assigns a point to a chart in proportion to the evidence of its encoding under that chart's latent prior. This has a simple interpretation: the closer a point lands to the center of a chart's latent space, and the more confidently it is placed there, the more likely it is to belong to that chart. Chart assignment thus emerges as posterior inference, depends on the encoder alone, and is fully self-supervised, so the charts discover structure in the data on their own, separating classes and subclasses such as distinct types of shoes or faces without access to labels. We further recover an atlas from the trained charts by identifying their overlaps and transition maps, enabling continuous interpolation within and across charts that stays close to the data manifold rather than traversing unsupported regions of latent space. Compared to capacity-matched baselines, Variational Atlas Autoencoders yields better-structured latent spaces and higher sample quality on image datasets including CelebA, while sampling requires evaluating only a single chart's decoder.

Figures & tables

Figure 1: Variational Atlas Autoencoders specialize to meaningful subsets of the data manifold with no label supervision. Left : a genus-2 surface covered by four 22 D charts, colored by the chart claiming each region, shows that multiple charts avoid the distortion of a single global 2D latent space. Right : generated samples per chart on three image datasets. On MNIST ( LeCun et al., 1998 ) and Fashion-MNIST ( Xiao et al., 2017 ) ( 1616 -D latents), a chart typically covers one class or a related pair; on CelebA ( Liu et al., 2015 ) ( 128128 -D latents), charts split by facial features, illumination, and background.
Figure 2: Top : we train CC encoder–decoder pairs to model the data manifold using independent charts. Our training objective, the ELBO, decomposes into per-chart losses weighted by a routing probability. The KL divergence on the chart selection prior encourages all charts to be used with equal probability. Bottom : sampling from a chart is then as simple as selecting a chart cc uniformly at random, sampling a latent zc\mathbf{z}_{c} from the standard normal prior, and decoding it. Moment matching can optionally be added, see Sec. 3.2 .
Figure 3: Illustration of latent-evidence routing. Two inputs (left) are encoded by every chart; the evidence of each encoded point under its chart’s latent prior induces a distribution over charts, which is used for routing. For the histogram (right), the temperature parameter was artificially raised to T=10T=10 to make the distribution easier to visualize.
Figure 4: An atlas of chart autoencoders can create smooth latent interpolation within each chart as well as across multiple charts. After training, we precompute a transition graph (right). Interpolation between any two examples in any two charts can then be achieved by following the shortest path in that graph. Within each chart’s latent space, we interpolate linearly (left).
Figure 5: Sampling a model trained on Fashion-MNIST ( Xiao et al., 2017 ) along the two dominant latent PCA axes generates fewer degenerate samples than a parameter-matched vanilla VAE ( Kingma and Welling, 2014 ) . The charts also specialize without supervision, e.g. by placing bags without handles in a separate chart. For readability, we show 16 out of the 20 charts.
Figure 6: Interpolations between sequences of randomly selected training samples (green). Our method may walk through one or more charts via portals ( Sec. 3.3 ), denoted by orange vertical bars. The vanilla VAE traverses its single global latent space in a straight line, which leads to several nonsensical generations along the way. In contrast, our method consistently stays on the data manifold, even if it entails covering more distance in latent space.
 Sample FIDReconstruction FID
DatasetPlain VAEVaDEVAA (ours)Plain VAEVaDEVAA (ours)
MNIST19.84 ±\pm 0.3414.95 ±\pm 0.5413.74 ±\pm 0.5516.81 ±\pm 0.2320.54 ±\pm 0.1218.25 ±\pm 0.13
Fashion83.27 ±\pm 1.6245.07 ±\pm 1.6237.73 ±\pm 1.1231.27 ±\pm 0.3243.49 ±\pm 0.1532.83 ±\pm 0.38
CelebA30.43 ±\pm 0.5928.01 ±\pm 0.2627.84 ±\pm 0.4216.66 ±\pm 0.1822.09 ±\pm 0.0717.24 ±\pm 0.25
Table 1: Sampling and reconstruction quality for a plain VAE ( Kingma and Welling, 2014 ) , VaDE ( Jiang et al., 2017 ) , and our atlas model (10 charts), all using the same number of parameters and training iterations. Quality is measured by the Fréchet Inception Distance (Inception-v3, Szegedy et al. (2016) ), averaged over three runs (mean ±\pm std). Lower is better.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

 MNISTFashion-MNISTCelebA
Convolutional stages224
Channel widths64, 12864, 12864, 128, 256, 512
Bottleneck channels13651
Bottleneck resolution7×77\times 77×77\times 74×44\times 4
Appendix
Table 2: Resolved DCGAN dimensions per dataset: channel widths of the shared convolutional network and the size of the feature map passed to the per-chart linear layers.
HyperparameterMNISTFashion-MNISTCelebA
Charts102010
Latent dimension1616128
Reconstruction lossBernoulli (BCE)Bernoulli (BCE)L1 + LPIPS, scale ×3\times 3
TT (routing temperature)Fixed at 1.0Fixed at 1.0Annealed 6.0→2.06.0\to 2.0
β\beta (KL weight)0.50.10.1
γ\gamma (load-balancing weight)50, entire run50, entire run50, entire run
Appendix
Table 3: Tuned hyperparameters for MNIST, Fashion-MNIST, and CelebA.
HyperparameterValue
BackboneMLP, hidden width 512, depth 5
Charts4
Latent dimension2
Reconstruction lossSquared error, no output squashing, scale ×200\times 200
TT (routing temperature)annealed 1.0→0.21.0\to 0.2
β\beta (KL weight)1.5
Appendix
Table 4: Tuned hyperparameters for the double torus.
  Latent-evidence routing
DatasetELBOμ\mu -basedResamplingExpected log-priorPrior-marginal
MNIST11.80 ±\pm 0.1012.99 ±\pm 0.2213.79 ±\pm 1.0012.77 ±\pm 0.3312.62 ±\pm 0.53
Fashion-MNIST43.64 ±\pm 0.3438.42 ±\pm 0.4939.77 ±\pm 1.4638.03 ±\pm 0.6438.53 ±\pm 0.45
CelebA47.14 ±\pm 0.1943.46 ±\pm 0.5344.05 ±\pm 0.2443.89 ±\pm 0.5543.69 ±\pm 0.24
Appendix
Table 5: Sample FID (mean ±\pm std over seeds) by routing function, DCGAN backbone, C=10C=10 charts, and iteration counts matched within each dataset. Lower is better; best in bold, second best underlined.
Figure 7: 10-chart Variational Atlas Autoencoders (ours) and 1-chart capacity-matched VAE reconstructions of randomly selected held-out inputs from MNIST, Fashion-MNIST and CelebA.
Figure 8: Interpolation trajectories between two charts through a portal on the double-torus geometry. For any two points on the torus, we can create interpolation trajectories in our charts’ latent spaces that remain close to the torus surface (the data manifold). The red line illustrates straight-line interpolation through a global latent space, which leaves the surface and thus passes through implausible points. Optimizing portal placement to select shorter paths is left as future work.
Figure 9: Sample images drawn from 10-, 10-, and 20-chart VAAs trained on MNIST, CelebA, and Fashion-MNIST, respectively. Each grid represents samples generated from a distinct chart. Samples from a capacity-matched VAE are shown on the right.
More in this topic

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 209 papers

Explore similar work

CardsList
Feb 26, 2026

Learning Tangent Bundles and Characteristic Classes with Autoencoder Atlases

We introduce a theoretical framework that connects multi-chart autoencoders in manifold learning with the classical theory of vector bundles and characteristic classes. Rather than viewing autoencoders as producing a single global Euclidean embedding, we treat a collection of locally trained encoder-decoder pairs as a learned atlas on a manifold. We show that any reconstruction-consistent autoencoder atlas canonically defines transition maps satisfying the cocycle condition, and that linearising these transition maps yields a vector bundle coinciding with the tangent bundle when the latent dimension matches the intrinsic dimension of the manifold. This construction provides direct access to differential-topological invariants of the data. In particular, we show that the first Stiefel-Whitney class can be computed from the signs of the Jacobians of learned transition maps, yielding an algorithmic criterion for detecting orientability. We also show that non-trivial characteristic classes provide obstructions to single-chart representations, and that the minimum number of autoencoder charts is determined by the good cover structure of the manifold. Finally, we apply our methodology to low-dimensional orientable and non-orientable manifolds, as well as to a non-orientable high-dimensional image dataset.

Eduardo Paluzo-Hidalgo, Yuichi Ike

Department of Applied Mathematics I, University of Sevilla, Sevilla, Spain · Graduate School of Mathematical Sciences, The University of Tokyo, Tokyo, Japan

Manifold LearningAutoencoders
AskPDF
Jun 5, 2026

Constructing VAE Latent Spaces with Prescribed Topology

Variational autoencoders (VAEs) learn low-dimensional latent representations of high-dimensional data. When the data lies on a manifold with non-Euclidean topology, the standard Gaussian prior introduces a topological mismatch that degrades reconstruction quality and prevents faithful representation. We present a constructive mathematical framework that resolves this mismatch for all manifolds that admit a product covering space. These are manifolds expressible as products of elementary factors (circles, intervals, or lines) or as quotients of such products by a finite symmetry group. The class includes cylinders, tori, Möbius strips, Klein bottles, and real projective spaces. Factorized distributions over the elementary factors yield product topologies with closed-form, decoupled KL divergences, so that each latent factor can be shaped independently while keeping training tractable. We catalogue reparametrizable encoder-prior pairs for periodic, bounded, and unbounded supports, and provide coordinate transformations that allow standard neural networks to output non-Euclidean parameters with smooth gradients. For quotient manifolds, the decoder receives group-invariant features of the covering-space coordinates, so that identified points produce identical outputs. Anchor constraints fix the coordinate system relative to the data or create soft topological holes. Experiments on synthetic manifolds and real-image datasets (rotated and cyclically shifted MNIST) confirm that a topology-matched prior aligns KL regularization with the data manifold. The resulting topology-aware models outperform the Gaussian baseline at all practically relevant regularization strengths. The code is available at https://github.com/JvHulst/VAE-Topology.

Jilles S. van Hulst, Jakub M. Tomczak, W. P. M. H. Heemels +1

Control Systems Technology Section, Department of Mechanical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands. · The research is carried out as part of the ITEA4 20216 ASIMOV project.

Variational AutoencodersManifold LearningTopological Representation LearningTopological MappingLatent Space Geometry
AskPDF
May 15, 2026

Entropic Auto-Encoding via Implicit Free-Energy Minimization

Despite their ubiquity, variational autoencoders (VAEs) inherently suffer from posterior collapse, a failure mode in which latent variables are effectively ignored. This failure arises because explicit prior imposition drives optimization toward loss landscape regions corresponding to uninformative latent representations. Here, we introduce Entropic Autoencoders (EAEs), a framework in which reconstruction loss is the only explicit objective, and entropy generates the latent variables' prior implicitly through a free energy-minimizing ensemble of encoders. This ensemble biases learning toward high-volume regions of near-optimal solutions, while decoder updates direct the search trajectories toward informative latent representations. We demonstrate that EAEs mitigate posterior collapse by learning non-Gaussian, multimodal latent distributions that yield diverse, data-consistent generations and preserve different forms of underlying structure in the data. As a proof-of-concept, we show that an EAE captures a superposition of the known low-dimensional dynamics of a reaction-diffusion process. Then, we show that an EAE identifies implicit categorical distinctions in MNIST latent representations, and displays a hierarchical understanding of facial structure on the CelebA dataset, from an "all-human" face to individual-dependent features.

Hazhir Aliahmadi, Irina Babayan, Greg van Anders

Department of Physics, Engineering Physics & Astronomy, Queen’s University, Kingston, ON, Canada

Variational AutoencodersPosterior CollapseAutoencodersEntropy Regularization
AskPDF