Deep latent variable models (DLVMs) underpin many modern analysis pipelines, for example in single-cell genomics, yet their prior distributions have received markedly little attention. The VampPrior approximates the optimal prior (the aggregate posterior) and often attains high model evidence. Here, we identify two key limitations of the VampPrior: it ties prior and posterior uncertainty to a single amortized covariance and clusters poorly due to its fixed uniform mixture weights. We unleash the VampPrior by decoupling its prior and posterior covariances, resolving the conflict of generating high prior and low posterior entropy with the same shared parameters and substantially improving model evidence. We then replace the uniform mixture with a Dirichlet process. When combined, our proposals provide higher model evidence than the VampPrior and robust non-parametric clustering. Augmenting single cell Variational Inference (scVI) with our prior improves single-cell integration and clustering across six atlases while significantly raising model evidence. Our approach changes only the prior and thus can be applied to any DLVM with a continuous latent space.
Figures & tables
Figure 1: Effect of varying the number of components K on FashionMNIST with u initialized from randomly selected training data. Lines and bands are means ±1 standard error over 20 seeds. Dashed connectors mark configurations at which the seed-paired one-sided Wilcoxon signed-rank test rejects at p<0.05 ; arrows indicate the direction of improvement.
Figure 2: Modeling a 2D anisotropic GMM (three well-separated components) with a 1D latent and K=40 available components. Top: data colored by cluster assignment. Middle and bottom: the prior’s component densities weighted by the mixture proportions π and by realized test-set usage, respectively. Only our DP VampPrior-Unleashed ( 2(c) ) recovers the three true clusters.
Figure 3: Evidence–NMI Pareto frontiers. Each point is one hyperparameter configuration averaged over 20 seeds; large points are the non-dominated configurations and the line connecting them interpolates the Pareto frontier. The dashed box spans the standard normal VAE (origin, × ) to the best per-datum log marginal and NMI attained by any prior. Configurations whose mean log marginal or NMI falls outside the grey box, i.e., worse than the standard normal VAE on either axis, are excluded before the frontier is computed. Stars mark each prior’s P∗ , the frontier point closest to the normalized ideal (top-right corner of grey box after normalization to unit box). Both our VampPrior-Unleashed and DP-VampPrior-Unleashed offer sizable frontier advancements.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Effect of varying the number of components K on FashionMNIST. Lines and bands are means ±1 standard error over 20 seeds. Dashed connectors mark configurations at which the seed-paired one-sided Wilcoxon signed-rank test rejects at p<0.05 ; arrows indicate the direction of improvement.
Figure 6: Effect of varying the number of components K on MNIST. Lines and bands are means ±1 standard error over 20 seeds. Dashed connectors mark configurations at which the seed-paired one-sided Wilcoxon signed-rank test rejects at p<0.05 ; arrows indicate the direction of improvement.
Figure 7: Effect of varying the concentration parameter α on FashionMNIST with truncation set to K=100 . Lines and bands are means ±1 standard error over 20 seeds. Dashed connectors mark configurations at which the seed-paired one-sided Wilcoxon signed-rank test rejects at p<0.05 ; arrows indicate the direction of improvement.
Figure 8: Effect of varying the concentration parameter α on MNIST with truncation set to K=100 . Lines and bands are means ±1 standard error over 20 seeds. Dashed connectors mark configurations at which the seed-paired one-sided Wilcoxon signed-rank test rejects at p<0.05 ; arrows indicate the direction of improvement.
Dataset
# Cells
# Donors
Cramér’s V
Cohen’s k
Association
DKD
39,176
11
0.145
10
Moderate
Immune Cell Atlas
329,762
12
0.227
11
High
GTEx v9
209,126
16
0.480
15
High
Mouse Pancreas Atlas
301,796
56
0.381
12
High
HypoMap
384,925
24
0.055
12
Low
Tabula Sapiens
483,152
29
0.444
28
High
Appendix
Table 4: Summary of single-cell RNA-seq datasets used in this study, including the number of cells, number of donors, Cramér’s V statistic for the association between cell type and donor, k=min(r,c)−1 for the r×c donor-by-cell-type contingency table, and a qualitative assessment of association strength. Following Cohen (2013) , association is Low, Moderate, or High when V exceeds 0.1/k , 0.3/k , or 0.5/k , respectively.
Latent Dirichlet Allocation (LDA) is a foundational model for discovering latent thematic structure in discrete data, but its Dirichlet prior cannot represent the rich correlations and hierarchical relationships often present among topics. We introduce the framework of Latent Dirichlet-Tree Allocation (LDTA), a generalization of LDA that replaces the Dirichlet prior with an arbitrary Dirichlet-Tree (DT) distribution. LDTA preserves LDA's generative structure but enables expressive, tree-structured priors over topic proportions. To perform inference, we develop universal mean-field variational inference and Expectation Propagation, providing tractable updates for all DT. We reveal the vectorized nature of the two inference methods through theoretical development, and perform fully vectorized, GPU-accelerated implementations. The resulting framework substantially expands the modeling capacity of LDA while maintaining scalability and computational efficiency.
Zheng Wang, Nizar Bouguila
Concordia Institute for Information Systems Engineering Concordia University Montreal, QC H3G 1M8, Canada
Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.
Jinlin Lai, Antonio Linero, Yuling Yao
College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin
Density-based clustering is a well-established framework that defines clusters through topological or geometric features of the data-generating density, yielding flexible procedures that have proven effective on large and complex datasets. We address the problem of obtaining scalable Bayesian-style uncertainty quantification by combining density-based clustering with martingale posterior distributions. Density uncertainty is first quantified through a fully parallelized predictive resampling scheme driven by model score functions, naturally accommodating modern differentiable density estimators such as normalizing flows. Posterior samples of the clustering are then obtained by applying density-based clustering to each random density sampled from the martingale posterior, enabling inference on any clustering-related quantity. Casting the estimand as a density functional also enables a rigorous frequentist convergence analysis. We illustrate the method through applications to image and single-cell RNA sequencing data, where it recovers meaningful clustering structures, including the assignment of previously unseen data and biologically interpretable cell grouping, while running in minutes on a single GPU.
Nicola Bariletto, Stephen G. Walker
Department of Statistics and Data Sciences, UT Austin