We propose denoising diffusion variational inference (DDVI), a black-box variational inference algorithm for latent variable models which relies on diffusion models as flexible approximate posteriors. Specifically, our method introduces an expressive class of diffusion-based variational posteriors that perform iterative refinement in latent space; we train these posteriors with a novel regularized evidence lower bound (ELBO) on the marginal likelihood inspired by the wake-sleep algorithm. Our method is easy to implement (it fits a regularized extension of the ELBO), is compatible with black-box variational inference, and generally outperforms alternative classes of approximate posteriors based on normalizing flows or adversarial networks. We find that DDVI improves inference and learning in deep latent variable models across common benchmarks as well as on a motivating task in biology---inferring latent ancestry from human genomes---where it outperforms strong baselines on the 1000 Genomes dataset.
Figures & tables
Figure 1: Denoising diffusion variational inference in a VAE. Between the encoder and decoder, we have a diffusion model to map a simple distribution into a complex distribution over latents.
Figure 2: Unsupervised visualization on MNIST with three priors (pinwheel, Swiss roll, square). Each color is one class.
Figure 3: Semi-supervised visualization on MNIST (1,000 labels) with the three priors. Each color is one class.
Figure 4: Visualization of genotype clusters. Each color indicates one population.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Diffusion Samplers
GFlowNet-based
DDVI (ours)
( Berner, Richter, and Ullrich 2022 ; Zhang and Chen 2021 ; Vargas, Grathwohl, and Doucet 2023 ; Richter, Berner, and Liu 2023 ; Akhound-Sadegh et al. 2024 )
( Zhang et al. 2023 ; Sendera et al. 2024 ; Malkin et al. 2022 )
Semi-implicit variational inference (SIVI) expands the representational capacity of variational families, but optimizing the Evidence Lower Bound (ELBO) remains challenging because the marginal score of the semi-implicit distribution is generally intractable. Unbiased Implicit Variational Inference (UIVI) and related methods address this difficulty through reparameterized ELBO gradients, but existing approaches estimate the required score indirectly through reverse-conditional sampling or approximation, which can be computationally costly and unstable. We propose Denoising Implicit Variational Inference (DIVI), which learns the marginal score directly via denoising score matching. The learned score is used as a plug-in estimator in the pathwise ELBO gradient, replacing inner reverse-conditional sampling with a score-network evaluation. We analyze DIVI as inexact stochastic gradient ascent and show, under stated assumptions, that the averaged stationarity measure is controlled by the usual optimization term and the average score-matching error. Empirically, we evaluate DIVI on both synthetic distributions and a variety of real-data Bayesian inference tasks. The results show that DIVI improves over the current UIVI baselines while reducing the cost of ELBO-gradient estimation.
Zitong Wang, Zekun Wu, Longlin Yu +1
School of Mathematical Sciences, Peking University
Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.
Jinlin Lai, Antonio Linero, Yuling Yao
College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin
Differential privacy (DP) provides robust privacy guarantees for statistical inference, but this can lead to unreliable results and biases in downstream applications. While several noise-aware approaches have been proposed which integrate DP perturbation into the inference, they are limited to specific types of simple probabilistic models. In this work, we propose a novel method for noise-aware approximate Bayesian inference based on stochastic gradient variational inference which can also be applied to high-dimensional and non-conjugate models. We also propose a more accurate evaluation method for noise-aware posteriors. Empirically, our inference method has similar performance to existing methods in the domain where they are applicable. Outside this domain, we obtain accurate coverages on high-dimensional Bayesian linear regression and well-calibrated predictive probabilities on Bayesian logistic regression with the UCI Adult dataset.