Variational Autoencoders

Also known as VAE

Overview

Variational autoencoders learn probability distributions over compact latent codes and decode samples from them, typically by balancing reconstruction quality against a KL-divergence regularizer in the evidence lower bound. Common approaches use Gaussian, mixture, flow-based, spherical, or heavy-tailed priors, and add hierarchical, disentangling, temporal, multimodal, geometric, or physics-informed structure. They are used for image, video, audio, molecule, medical, and scientific data, both as generative models and as compact representations for downstream prediction or diffusion. A central tension is that stronger regularization can make generation easier while causing posterior collapse or losing useful detail; reconstruction scores alone also do not guarantee good samples, calibrated uncertainty, or useful latent structure.

Latest papers

221 papers16 papers in the last four weeks, down 43% on the four weeks before. 0.1% of all new papers.

Sep 28, 2025

Latent Representation Learning from 3D Brain MRI for Latent Analysis and Prediction in Multiple Sclerosis

Neurological diseases can cause significant cognitive impairment, yet linking brain structure to cognitive function remains challenging due to the limitations of conventional methods and the black-box nature of deep learning. We propose InfoVAE-Med3D, an adaptation of InfoVAE for three-dimensional brain MRI that learns informative latent representations by relaxing the mutual information regularization penalty between inputs and latent representations. We evaluated the method on a healthy control (HC) dataset (n=6,503n=6{,}503) with age and sex information and a multiple sclerosis (MS) dataset (n=916n=916) with age, sex, and Symbol Digit Modalities Test (SDMT) scores. Two-dimensional visualizations revealed sex-related clustering and smooth transitions in grey and white matter volumes. Removing the mutual information regularization penalty generally improved MRI reconstruction and brain age and SDMT prediction using support vector regression, outperforming other variational autoencoder variants and achieving competitive performance with deep learning predictors, particularly on the MS dataset. InfoVAE-Med3D provides informative latent representations for investigating brain-related characteristics and supports latent-space analysis and imaging biomarker discovery for cognitive impairment, particularly in MS.
Sep 6, 2025

Robust Amortized Simulation-Based Inference via Learned Error Models

Recent advances in neural density estimation have enabled amortized Bayesian inference for complex stochastic simulators. However, these methods rely on simulators accurately reflecting the true data-generating process and can degrade significantly under model misspecification. We consider the setting where multiple unlabeled observations are available and introduce robust variational neural posterior estimation (RVNP), an amortized Bayesian inference method that uses an importance-weighted autoencoder to jointly learn a misspecification-robust posterior proxy and an explicit error model that captures the simulation-to-reality gap. Our results show that RVNP can recover robust posterior inference in a data-driven and interpretable manner, outperforming previous methods on multiple metrics across misspecified benchmarks.
Aug 16, 2025

DE-VAE: Revealing Uncertainty in Parametric and Inverse Projections with Variational Autoencoders using Differential Entropy

Recently, autoencoders (AEs) have gained interest for creating parametric and invertible projections of multidimensional data. Parametric projections make it possible to embed new, unseen samples without recalculating the entire projection, while invertible projections allow the synthesis of new data instances. However, existing methods perform poorly when dealing with out-of-distribution samples in either the data or embedding space. Thus, we propose DE-VAE, an uncertainty-aware variational AE using differential entropy (DE) to improve the learned parametric and invertible projections. Given a fixed projection, we train DE-VAE to learn a mapping into 2D space and an inverse mapping back to the original space. We conduct quantitative and qualitative evaluations on four well-known datasets, using UMAP and t-SNE as baseline projection methods. Our findings show that DE-VAE can create parametric and inverse projections with comparable accuracy to other current AE-based approaches while enabling the analysis of embedding uncertainty.
Jul 31, 2025

L-GTA: Latent Generative Modeling for Time Series Augmentation

Data augmentation is becoming increasingly important across various areas of time series analysis, including forecasting, classification, and anomaly detection. We introduce the Latent Generative Temporal Augmentation (L-GTA) model, a generative approach based on a Variational Autoencoder with a Bi-LSTM backbone and temporal self-attention. The model learns a latent representation for each timestep and applies controlled perturbations such as jittering, magnitude warping, or drift. We define an equivariance objective to further encourage consistency between latent space and data space transformations. As a result, the augmented samples show predictable and interpretable transformation signatures. We evaluate L-GTA on several real-world datasets against SOTA generative methods, including TimeGAN, TimeVAE, and Diffusion-TS, as well as direct transformation approaches. Across experiments on downstream forecasting, distribution fidelity, and controllability of transformation intensity, L-GTA consistently outperforms competing approaches. In downstream forecasting, it reduces prediction error by up to 26% compared to the strongest generative method and 27% relative to using the original data without augmentation.
Jun 26, 2025

Spherical Cauchy Variational Autoencoders: Heavy Angular Tails and Exact KL Evaluation

Heavy-tailed posteriors are routine in Euclidean variational autoencoders, where the Student family relaxes the Gaussian without new machinery. The sphere has had no comparable option. Von Mises-Fisher distribution needs modified Bessel functions and a rejection sampler, and Power Spherical buys its closed forms by forcing the density to vanish at the antipode. We develop the spherical Cauchy distribution as a hyperspherical posterior that needs neither compromise. Stereographic projection carries it to a multivariate Student law, and a Möbius transformation turns a uniform spherical draw into an exact posterior sample from inner products, norms, and scalar arithmetic. The same transformation settles the regularizer. Evaluating the density along the sampling map reduces the Kullback-Leibler (KL) divergence to the uniform prior to a scalar expectation whose expansion terminates in every even ambient dimension, leaving one logarithm and a polynomial with finitely many correction terms. Odd dimensions admit certified truncation of value and gradient, the KL is increasing and convex in concentration, and the same function gives the pairwise KL. At matched modal curvature it has broader angular tails and a smaller KL penalty than both alternatives, so equal local precision costs less regularization. In dimension 128 the fused evaluator runs 1.5 times faster per latent-layer step than Power Spherical and 4.2 times faster than robust von Mises-Fisher on CPU, with factors of 1.6 and 5.4 on CUDA. Across five paired seeds it attains the lowest MNIST reconstruction loss at every tested dimension and lowers held-out viewpoint-gap negative log-likelihood on smallNORB by 3.6 percent.
Jun 24, 2025

Cross-Layer Discrete Concept Discovery for Interpreting Language Models

Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure. Cross-layer sparse autoencoders (SAEs) address layer mixing but operate in continuous space, where concepts split across many neurons without clear boundaries. We introduce Cross-Layer Vector Quantized-Variational Autoencoder (CLVQ-VAE), a novel framework which maps representations from a lower layer to a higher layer through a discrete vector-quantization bottleneck, collapsing duplicated residual-stream features into compact, interpretable concept vectors. Our approach combines top-k temperature-based sampling with exponential moving average (EMA) codebook updates, providing controlled exploration of the discrete latent space while maintaining codebook diversity. Across both encoder- and decoder-based models on ERASER-Movie, Jigsaw, and AGNews, CLVQ-VAE outperforms clustering, single-layer vector quantized-variational autoencoder (VQ-VAE), and sparse autoencoder (SAE) baselines across three evaluation axes: removing identified concepts drops downstream probe accuracy by up to 93%, LLM judges rank our concepts first in 66.7% of comparisons, and human annotators recover model predictions from our visualizations with 78% accuracy versus 54% for clustering.
Jun 23, 2025

Latent Space Analysis for Interpretable Uncertainty in Melanoma Classification

Melanoma is a highly aggressive skin cancer, making early and accurate diagnosis critical. While deep learning excels in skin lesion classification, standard ``black-box" models struggle to explain diagnostic uncertainty, limiting clinical trust. This work introduces a hybrid framework combining a class-aware adversarial Variational Autoencoder and an XGBoost classifier, transcending simple binary classification by leveraging a generative latent space for interpretable decision support. Guided by adversarial training, the model learns the visual characteristics of skin lesions and projects them into a continuous latent space, ensuring that similar images are grouped closely together. Trained on this latent space, the XGBoost classifier achieves a robust AUC of 0.868, competing closely with state-of-the-art models. For borderline cases, the framework enables clinicians to leverage the latent topology through Content-Based Image Retrieval. This provides a dual benefit: it allows the clinician to visually compare an ambiguous lesion against biopsy-confirmed precedents and acts as an early warning sign since a borderline classification can indicate that a lesion shares features of both nevi and melanomas, potentially requiring close monitoring. Our approach translates algorithmic hesitation into transparent, evidence-based visual support, bridging the gap between predictive performance and clinical trust.
Jun 20, 2025

Variational Learning of Disentangled Representations

Disentangled representations separate factors that are shared across conditions from those that are condition-specific. Such separation is needed for generalization to new domains, treatments, patients, or species. A dominant line of work pursues this goal through variational formulations. While these approaches achieve partial disentanglement, they often exhibit three common limitations: they either do not remove all condition-specific information from the condition-specific representation, allow the condition-specific representation to become uninformative, or impose independence assumptions that do not reflect the underlying generative process. In this work, we introduce DisCoVR, a variational framework that addresses these limitations. Its objective is aligned with the probabilistic structure of the data-generating process, and includes an adversarial term that prevents condition-specific information from being encoded in the condition-specific representation.DisCoVR reconstructs the data from both shared and condition-specific representations, ensuring that each remains informative, and uses a structured prior that further reinforces the informativeness of both representations. We show that across synthetic, image, and single-cell RNA-sequencing datasets, DisCoVR achieves stronger disentanglement compared to previous approaches.
Jun 1, 2025

Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis

Despite the empirical success of Diffusion Models (DMs) and Variational Autoencoders (VAEs), their generalization performance remains theoretically underexplored, especially lacking a full consideration of the shared encoder-generator structure. Leveraging recent information-theoretic tools, we propose a unified theoretical framework that provides guarantees for the generalization of both the encoder and generator by treating them as randomized mappings. This framework further enables (1) a refined analysis for VAEs, accounting for the generator's generalization, which was previously overlooked; (2) illustrating an explicit trade-off in generalization terms for DMs that depends on the diffusion time TT; and (3) providing computable bounds for DMs based solely on the training data, allowing the selection of the optimal TT and the integration of such bounds into the optimization process to improve model performance. Empirical results on both synthetic and real datasets illustrate the validity of the proposed theory.
Aug 2, 2024

Autoencoders in Function Space

Autoencoders have found widespread application in both their original deterministic form and in their variational formulation (VAEs). In scientific applications and in image processing it is often of interest to consider data that are viewed as functions; while discretisation (of differential equations arising in the sciences) or pixellation (of images) renders problems finite dimensional in practice, conceiving first of algorithms that operate on functions, and only then discretising or pixellating, leads to better algorithms that smoothly operate between resolutions. In this paper function-space versions of the autoencoder (FAE) and variational autoencoder (FVAE) are introduced, analysed, and deployed. Well-definedness of the objective governing VAEs is a subtle issue, particularly in function space, limiting applicability. For the FVAE objective to be well defined requires compatibility of the data distribution with the chosen generative model; this can be achieved, for example, when the data arise from a stochastic differential equation, but is generally restrictive. The FAE objective, on the other hand, is well defined in many situations where FVAE fails to be. Pairing the FVAE and FAE objectives with neural operator architectures that can be evaluated on any mesh enables new applications of autoencoders to inpainting, superresolution, and generative modelling of scientific data.
Apr 20, 2022

Unsupervised Domain Adaptation for Cardiac Segmentation: Towards Structure Mutual Information Maximization

Unsupervised domain adaptation approaches have recently succeeded in various medical image segmentation tasks. The reported works often tackle the domain shift problem by aligning the domain-invariant features and minimizing the domain-specific discrepancies. That strategy works well when the difference between a specific domain and between different domains is slight. However, the generalization ability of these models on diverse imaging modalities remains a significant challenge. This paper introduces UDA-VAE++, an unsupervised domain adaptation framework for cardiac segmentation. We show that the evidence lower bound of VAE-based domain adaptation penalizes the mutual information between the image and the latent space, including the structure information that segmentation relies on. Maximizing the structure mutual information removes this penalty on the structure information, while the remaining information is still compressed. To estimate this term, we develop a novel Structure Mutual Information Estimation (SMIE) block with a global estimator, a local estimator, and a prior information matching estimator to maximize the mutual information between the reconstruction and segmentation tasks. Specifically, we design a novel sequential reparameterization scheme that enables information flow and variance correction from the low-resolution latent space to the high-resolution latent space. Comprehensive experiments on benchmark cardiac segmentation datasets demonstrate that our model outperforms previous state-of-the-art qualitatively and quantitatively. The code is available at https://github.com/LOUEY233/Toward-Mutual-Information