BayesNDE: Bayesian Generative Modeling for Neural Density Estimation

Chenglin Li; Qiao Liu

stat.MLSep 30, 2026

BayesNDE: Bayesian Generative Modeling for Neural Density Estimation

Authors: Chenglin Li, Qiao Liu

Organizations: Department of Biostatistics, Yale University New Haven, Connecticut, USA

Abstract

Density estimation is a fundamental problem in statistics and machine learning. In this work, we introduce BayesNDE, a neural density estimator based on Bayesian generative modeling. BayesNDE learns a Bayesian generative model and evaluates its density without requiring invertible networks or Jacobian-determinant computation. For each observation, it infers a sample-specific latent posterior to construct an adaptive proposal that focuses computation on regions contributing most to its density. Bridge sampling then combines samples from this proposal with separate posterior samples to estimate the density. Experiments on nonlinear and multimodal synthetic datasets show improved estimation of density values and better recovery of the density structure compared to the state-of-the-art neural density estimators. Applications to real-world datasets further demonstrate improved anomaly detection. Together, these results highlight BayesNDE as a flexible and effective neural density estimator, demonstrating how posterior inference can turn generative models into tools for density estimation. The code and tutorials are available at https://github.com/liuq-lab/BayesNDE.

Figures & tables

Figure 1 : Overview of BayesNDE.

Figure 2 : True and estimated densities for the two-dimensional involute distribution and independent Gaussian mixture.

Method	Unconditional		Conditional
	BANK	ParkTele	Pendigits10	Vehicle	EEGEye
MADE	14.38 $\pm$ 0.27	34.34 $\pm$ 0.67	-54.56 $\pm$ 0.96	-51.84 $\pm$ 1.41	-50.25 $\pm$ 0.18
MAF	-53.15 $\pm$ 0.20	36.99 $\pm$ 0.66	-52.29 $\pm$ 0.78	-52.30 $\pm$ 2.25	-49.47 $\pm$ 0.21
Real NVP	27.21 $\pm$ 0.38	36.45 $\pm$ 0.68	-54.59 $\pm$ 1.16	-63.03 $\pm$ 3.61	-49.87 $\pm$ 0.22
Resflow	23.68 $\pm$ 0.18	36.25 $\pm$ 0.63	-47.77 $\pm$ 0.64	-45.97 $\pm$ 1.13	-48.33 $\pm$ 0.29
CFMs	10.23 $\pm$ 0.07	16.32 $\pm$ 0.19	-54.40 $\pm$ 0.36	-49.65 $\pm$ 0.87	-51.45 $\pm$ 0.14

Table 1 : Average test log-likelihood (nats) on real-world datasets. Uncertainties are two standard errors of the mean test log-likelihood across test points. The best result for each dataset is shown in bold.

	OC-SVM	I-Forest	Real NVP	MAF	CFMs	Resflow	Roundtrip	BayesNDE
Shuttle	0.953	0.958	0.735	0.861	0.932	0.477	0.958	0.965
Cardio	0.533	0.556	0.178	0.200	0.511	0.244	0.311	0.556
Pendigits	0.154	0.128	0.000	0.000	0.000	0.000	0.000	0.180

Table 2 : Precision@k on the ODDS anomaly-detection benchmarks, where k equals the number of outliers in the test split. Results are means over the same three random seeds for every method; OC-SVM is deterministic. Higher values are better.

Figure 3 : Conditional image generation on MNIST. The first column shows real MNIST examples; the remaining columns show samples generated by BayesNDE and MAF. For each digit class, representative generated images are shown in decreasing order of conditional density $p_{\bm{\theta}}({\bm{x}}\mid y)$ .

Panel of Figure 4 — Figure 4 : Empirical evaluation across benchmarks. (a) Density estimation accuracy across varying dimensions; (b) Classification performance on MNIST.

Method	Spearman $\uparrow$
BayesNDE (2D)	0.9214
w.o. EGM init.	0.0129
w.o. var. reg.	0.8929
BayesNDE (15D)	0.6311
w.o. BS	0.5932
w.o. EGM reg. (MMD)	0.5848

Table 3 : Ablation results on synthetic data.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dim( $z$ )	Dim( $x$ )	Outliers (%)	Test outliers ( $k$ )	Train	Validation	Test
					# examples
Shuttle	3	9	7.15	365	39770	4418	4909
Cardio	5	21	9.61	15	1484	164	183
Pendigits	5	16	2.27	13	5565	618	687

Appendix

Table 4 : Basic descriptions of the three datasets used for outlier detection in our study.

Dataset	Domain	Dim( ${\bm{z}}$ )	Dim( ${\bm{x}}$ )	# of examples
Dataset	Domain	Dim( ${\bm{z}}$ )	Dim( ${\bm{x}}$ )	Train	Validation	Test
Vehicle	Transportation	6	18	684	77	85
ParkTele	Medicine	8	14	4,760	528	587
Pendigits10	Handwriting	8	16	8,902	990	1,100
EEGEye	Neuroscience	5	14	12,130	1,348	1,498
BANK	Finance	8	17	36,621	4,069	4,521

Appendix

Table 5 : Basic descriptions of the 6 datasets for density estimation used in our study. BANK and ParkTele are used for unconditional density estimation; Pendigits10, Vehicle and EEGEye for conditional density estimation, where

\dim({\bm{x}})

counts only the continuous features and the class label is the conditioning variable.

Conditional decoder $G_{{\bm{\theta}}}$	Image discriminator $D_{x}$
Inputs: $z\in\mathbb{R}^{10}$ and $y\in\{0,1\}^{10}$	Inputs: $x\in[0,1]^{784}$ and $y\in\{0,1\}^{10}$
$\operatorname{Concat}(z,y)\in\mathbb{R}^{20}$	Reshape $x$ to $28\times 28\times 1$ ; tile $y$ to $28\times 28\times 10$
FC $6272$ , LReLU	$\operatorname{Concat}(x,\operatorname{tile}(y))$ : $28\times 28\times 11$
Reshape to $7\times 7\times 128$	Conv ( $k=5$ , $s=2$ , $c=32$ ), LReLU: $14\times 14\times 32$
Tile $y$ to $7\times 7\times 10$ and concatenate: $7\times 7\times 138$	Conv ( $k=5$ , $s=2$ , $c=64$ ), LReLU: $7\times 7\times 64$
Transposed conv ( $k=3$ , $s=2$ , $c=64$ ), BN, LReLU: $14\times 14\times 64$	Conv ( $k=3$ , $s=2$ , $c=128$ ), LReLU: $4\times 4\times 128$

Appendix

Table 6 : Network architecture used by BayesNDE for conditional MNIST image generation and density estimation. Images are scaled to

[0,1]^{784}

, the class label

y\in\{0,1\}^{10}

is one-hot encoded, and the latent variable is

z\in\mathbb{R}^{10}

. For convolutional layers,

k

s

, and

c

denote kernel size, stride, and number of output channels, respectively. All convolutions use same padding. BN denotes batch normalization and LReLU has negative slope

0.2

. The decoder has two output heads that parameterize a Gaussian distribution in logit space.

		EGM			Iterative
Dataset	$z$	$\alpha$	$\lambda_{\mathrm{var}}$	$\lambda_{\mathrm{corr}}$	$\lambda^{\mathrm{iter}}_{\mathrm{var}}$	$\lambda^{\mathrm{iter}}_{\mathrm{corr}}$	$\lambda_{\mathrm{PP}}$
BANK	8	0.1	0.01	0.3	0	0.3	1000
ParkTele	8	0	0	0.3	0	0.3	1000
Pendigits10	8	0	0	0.3	0.01	0.3	0
Vehicle	6	0	0	0	0	0	0
EEGEye	5	0.1	0.01	0	0.01	0	0

Appendix

Table 7 : Regularization weights for the real datasets, fixed on validation data before any test point is scored.

Figure 5 : (a) Sampling-budget curve on the

15

-dimensional independent GMM benchmark: rank correlation between

\log\hat{p}

and the ground-truth log-density against the posterior budget

M

. Shaded bands are

\pm 1

standard error over three estimator replicates, and the dotted horizontal line marks the highest accuracy importance sampling attains at any budget, which bridge sampling already exceeds at

M=400

. (b) Running time on BANK: the time to fit each estimator plus the time the fitted estimator takes to score

500

test points, drawn uniformly at random without replacement from the

4521

-row test split. All six methods use all

17

columns and were timed on one node of the same type (Intel Xeon 6442Y, four cores, a single NVIDIA A40). The vertical axis is logarithmic.

$M$	Bridge	IS	$\Delta$
	Spearman( $\log p$ )
$100$	$\mathbf{0.5804}\pm 0.0049$	$0.5270\pm 0.0033$	$+0.0535$
$200$	$\mathbf{0.6034}\pm 0.0090$	$0.5501\pm 0.0146$	$+0.0534$
$400$	$\mathbf{0.6311}\pm 0.0080$	$0.5932\pm 0.0053$	$+0.0379$
$800$	$\mathbf{0.6300}\pm 0.0042$	$0.6014\pm 0.0022$	$+0.0286$
$1600$	$\mathbf{0.6207}\pm 0.0011$	$0.6039\pm 0.0009$	$+0.0168$

Appendix

Table 8 : Sampling-budget sweep at

p=15

, mean

\pm

standard error over three estimator replicates on

500

test points.

\Delta

is bridge minus importance sampling.

Explore similar work

Aug 10, 2026cs.LG

In-Context Density Estimation for Tabular Data

Density estimation underlies many unsupervised tasks on tabular data such as anomaly detection, out-of-distribution detection, and data augmentation. Although all these problems reduce to questions about where probability mass lies, they are typically solved individually by fitting a separate model to each dataset, with its own hyperparameters and tuning budget. We introduce ICED, an in-context, energy-based density estimator that removes this per-dataset cost. ICED is a transformer-based model pretrained once on a synthetic prior built specifically for density estimation under an objective that fits log-density where it is informative and preserves its ordering elsewhere. In the inference, it reads a dataset as context and returns an unnormalized log-density for any query point in a single forward pass, with no fitting, sampling, or hyperparameter selection. A single frozen ICED model then drives four tasks usually handled by four specialized pipelines: density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative augmentation. Across all four, it is competitive with the strongest task-specific method, while being the only approach that needs no retraining, no tuning, and no labels to move between them. The code is available at https://github.com/gmum/iced.

Patryk Marszałek, Jacek Tabor, Marek Śmieja

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland

Density Ratio Estimation Out-Of-Distribution Detection Dense Prediction Unsupervised Tabular Data In-Context

Ask PDF

May 13, 2026stat.ML

Adaptive Kernel Density Estimation with Pre-training

Density estimation in high-dimensional settings is an important and challenging statistical problem.Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure.

Ruitong Zhang, Ke Deng

Department of Statistics and Data Science Tsinghua University Beijing, China

Density Ratio Estimation Kernel Method Exponential Family Pretraining

Ask PDF

May 9, 2026cs.AI

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Density estimation is a central primitive in probabilistic modeling, yet continuous, discrete, and mixed-variable domains are often treated by separate objectives, limiting the ability to exploit a common statistical structure across data types. Continuous score-based methods rely on log-density gradients, while discrete extensions typically use concrete score whose unbounded targets become unstable near low-probability states. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM replaces ordinary density-ratio regression with a bounded energy-difference transform and derives from it a sample-only training objective with the constant target 1. The learned scalar potential recovers log p without partition-function estimation or explicit unbounded ratio regression. Across continuous, discrete, and mixed-variable benchmarks, CTEM substantially improves density estimation over competitive baselines and yields higher-quality samples under standard sampling procedures.

Zhijun Zeng, Yixuan Jiang, Pipi Hu +1

Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China

Density Ratio Estimation Energy Resources Probabilistic Model Unified Framework

Ask PDF