Empirical Variational Autoencoder

Kaede Shiohara

cs.LGOct 5, 2026

Empirical Variational Autoencoder

Authors: Kaede Shiohara

Organizations: The University of Tokyo

Abstract

We present Empirical Variational Autoencoder, a general generative framework for continuous-valued (i.e., non-vector-quantized) sequences. EVA is based on the evidence lower bound of the Variational Autoencoder (VAE) but learns autoregressive latent priors empirically from training data, which can be implemented only by an additional single linear layer on top of VAEs. By replacing the conventional standard-Gaussian constraint with the self-predicted priors, EVA significantly alleviates the latent distribution gap between prior and posterior which is typically observed in conventional VAEs, and leads to high-fidelity ancestral sampling for sequential data generation. Extensive experiments on image and sound synthesis demonstrate that EVA achieves competitive generation quality with autoregressive diffusion baselines despite its much faster inference time.

Figures & tables

Figure 1 : Curated examples generated by EVA-L trained on ImageNet (256 $\times$ 256).

Figure 2 : Framework of EVA. During training, the encoder maps the observed sequence

x_{1:T}

to latent variables

z_{1:T}

, the decoder reconstructs each token

\hat{x}_{t}

from the latent

z_{1:t}

, and learns the autoregressive latent prior

p(z_{t}\mid z_{1:t-1})

by matching it to the posterior

q(z_{t}\mid x_{1:T})

. During inference, the model autoregressively samples the latent sequence from the empirical prior and decodes each token causally to generate a continuous-valued sequence.

Figure 3 : Toy Experiment on Multimodal Sequence Structure. Each panel shows samples in the two-dimensional $(x_{0},x_{1})$ space, with the horizontal axis corresponding to $x_{0}$ and the vertical axis corresponding to $x_{1}$ . The blue points are sampled from the true data distribution while the orange points are generated by each model. AR tends to average over plausible modes and fails to recover the multimodal joint structure. GMM fails once the multimodal structure exceeds its finite mixture capacity. Causal VAE captures the major modes, but still produces noticeable inter-mode samples. In contrast, EVA concentrates samples on the true modes and substantially suppresses inter-mode samples, indicating that the learned autoregressive prior better represents latent dependencies across sequence steps. Best viewed in zoom.

Table 1 : Comparison with Baselines on ImageNet (256

\times

256) and VGGSound. AR-Diffusion (12 blocks) has the same number of Transformer blocks as EVA at inference time while AR-Diffusion (24 blocks) has the same number of Transformer blocks at training time . In the right, we visualize AR-Diffusion with different denoising steps

\{10,15,20,30,40,50\}

. Despite the fewer inference parameters and faster inference time, our method achieves competitive generation fidelity.

Figure 4 : Latent Space Visualization. Each patch represents bi-directional KL divergence between the source latent ( + ) and each latent. Lighter indicates similar to the source patch.

Figure 5 : Image Completion. The darker regions (like ) in the first row indicate the areas to be re-generated.

Panel of Figure 6 — Figure 6 : Latent Size and Model Size. (a) Different latent sizes with EVA-B. The variant with a latent dimension of 128 achieves the best FID of 7.68 with a CFG scale of 7. (b) Different backbone sizes with a latent size of 64. EVA-L achieves an FID of 6.04 with a CFG scale of 7.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7 : Uncurated Examples.

Panel of Figure 8 — Figure 8 : Uncurated Examples.

Explore similar work

Entropic Auto-Encoding via Implicit Free-Energy Minimization
May 15, 2026Hazhir Aliahmadi, Irina Babayan, Greg van AndersConditional Variational Autoencoder Autoencoder Architectures
Department of Physics, Engineering Physics & Astronomy, Queen’s University, Kingston, ON, Canada
ELVAE: Evidential Learning-Based Variational Autoencoder for Uncertainty-Aware Generation
Aug 11, 2026Ge WangEvidential Uncertainty
Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, New York, USA
eXact-Prior Variational Autoencoder (X-VAE): Learning Data-Adaptive Gaussian Mixture Priors for Latent Distributions
Jun 30, 2026Qijun Chen, Shaofan LiConditional Variational Autoencoder Star-Vae
Department of Civil and Environmental Engineering, University of California, Berkeley, CA, 94720, USA

May 15, 2026, cs.LGJ/K move · Enter open · S save

Entropic Auto-Encoding via Implicit Free-Energy Minimization

Hazhir Aliahmadi, Irina Babayan, Greg van Anders

Department of Physics, Engineering Physics & Astronomy, Queen’s University, Kingston, ON, Canada

Conditional Variational Autoencoder Autoencoder Architectures Latent Variable Free Energy Principle Collapse

Open paper AskPDF arXiv

Despite their ubiquity, variational autoencoders (VAEs) inherently suffer from posterior collapse, a failure mode in which latent variables are effectively ignored. This failure arises because explicit prior imposition drives optimization toward loss landscape regions corresponding to uninformative latent representations. Here, we introduce Entropic Autoencoders (EAEs), a framework in which reconstruction loss is the only explicit objective, and entropy generates the latent variables' prior implicitly through a free energy-minimizing ensemble of encoders. This ensemble biases learning toward high-volume regions of near-optimal solutions, while decoder updates direct the search trajectories toward informative latent representations. We demonstrate that EAEs mitigate posterior collapse by learning non-Gaussian, multimodal latent distributions that yield diverse, data-consistent generations and preserve different forms of underlying structure in the data. As a proof-of-concept, we show that an EAE captures a superposition of the known low-dimensional dynamics of a reaction-diffusion process. Then, we show that an EAE identifies implicit categorical distinctions in MNIST latent representations, and displays a hierarchical understanding of facial structure on the CelebA dataset, from an "all-human" face to individual-dependent features.

Abstract

Explore similar work

Entropic Auto-Encoding via Implicit Free-Energy Minimization

ELVAE: Evidential Learning-Based Variational Autoencoder for Uncertainty-Aware Generation

eXact-Prior Variational Autoencoder (X-VAE): Learning Data-Adaptive Gaussian Mixture Priors for Latent Distributions