Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes

August Arnstad; Geir Olve Storvik; Filippo Pagani; Leiv Rønneberg

stat.MLOct 23, 2026

Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes

Authors: August Arnstad, Geir Olve Storvik, Filippo Pagani, Leiv Rønneberg

Organizations: Department of Mathematics, University of Oslo, NO · Department of Statistics, University of Warwick, UK

Abstract

There is growing evidence that the performance of a neural network depends largely on a small subset of its weights, which are typically identified by training a dense network and pruning it afterwards. Within a Bayesian framework, pruning can be directly incorporated through a spike-and-slab prior, providing a proper treatment of the uncertainties involved, both with respect to pruning and to parameter uncertainty. In Bayesian neural networks (BNNs), sampling by MCMC is notoriously hard. We demonstrate that modern Monte Carlo methods based on piecewise deterministic Markov processes (PDMPs) can be used to sample posteriors of BNNs asymptotically exact whenever numerically constructed rate bounds hold. Spike-and-slab priors can be incorporated through sticky versions of the PDMPs, setting weights exactly to zero during sampling. We examine the predictive performance and convergence of the samplers, and show that sparse BNNs sampled this way can be more robust to input noise than dense ones.

Figures & tables

Figure 1: Sticky PDMPs sample exact zeros. (a) Example of a sticky ZigZag parameter path. (b) Posterior inclusion probabilities of weights in a BNN sampled with sticky PDMPs. (c) Inclusion probabilities of the MNIST input pixel weights, low where pixels carry no information.

Figure 2: Posterior predictive fits on the sharp dataset. The dashed black line is the true function $f$ , solid lines are posterior predictive means, and thin lines lie two predictive standard deviations away from the mean.

Sampler	Sparsity	RMSE	NLL	CRPS
Boomerang	–	0.115	-0.740	0.066
ZigZag	–	0.116	-0.739	0.067
NUTS	–	0.115	-0.742	0.066
Sticky Boomerang	85%	0.109	-0.801	0.062
Sticky ZigZag	87%	0.109	-0.799	0.062
LBBNN	95%	0.111	-0.778	0.063

Table 1: Predictive performance on the sharp dataset, on evenly spaced test points in

[-3,3]

, for a single run of each method. Sparsity is the percentage of weights that are exactly zero, averaged over draws. Lower is better for RMSE, NLL and CRPS, and the best value is in bold.

Network	Sampler	$w$	Sparsity	RMSE	NLL	CRPS	$C_{0.9}$
$d=752$	Boomerang	–	–	2.88 $\pm$ .19	2.48 $\pm$ .08	1.58 $\pm$ .10	90%
	ZigZag	–	–	2.88 $\pm$ .22	2.49 $\pm$ .08	1.58 $\pm$ .12	91%
	NUTS	–	–	2.86 $\pm$ .22	2.48 $\pm$ .08	1.57 $\pm$ .12	91%
	Sticky Boomerang	0.3	64%	2.98 $\pm$ .18	2.51 $\pm$ .06	1.62 $\pm$ .10	89%
	Sticky ZigZag	0.3	67%	2.97 $\pm$ .23	2.51 $\pm$ .08	1.63 $\pm$ .12	92%
	Sticky Boomerang	0.1	85%	3.07 $\pm$ .19	2.54 $\pm$ .06	1.67 $\pm$ .10	90%

Table 2: Predictive performance on the Boston Housing test splits, given as mean

\pm

standard error (SEM) over 5 independent train/test splits.

d

is the number of parameters of the small and medium networks. Sparsity is the percentage of weights sampled exactly zero, averaged over draws.

C_{0.9}

is the share of test responses inside the central 90% predictive interval, with a SEM of 2 to 3 percentage points (omitted for space).

Figure 3: Chain means of

\log\sigma

(top) and test RMSE (bottom) on the first Boston split. Filled dots show each chain’s mean with one posterior standard deviation, and hollow dots the MAP it started from.

Panel of Figure 4 — Figure 4: Analyses following Izmailov et al. (2021a) (a) and Izmailov et al. (2021b) (b), adapted to our samplers for the MNIST FFN. (a) First-layer weights of four hidden units. (b) Robustness to Gaussian noise of standard deviation $\sigma$ on the $[0,1]$ pixel scale.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	One-dimensional	UCI small	UCI medium	MNIST FFN	MNIST LeNet-5
Hidden layers	$[100]$	$[50]$	$[50,50,50]$	$[256,256]$	LeNet-5
Activation	$\tanh$	$\tanh$	$\tanh$	ReLU	$\tanh$
Likelihood	Gaussian, $\sigma$ fixed	Gaussian, $\sigma$ learned	Gaussian, $\sigma$ learned	Categorical	Categorical
$\sigma_{w}$ , $\sigma_{b}$	3, 3	1, 1	1, 1	2, 2	2, 2
Noise prior scale $s$	–	per dataset ⋆	per dataset ⋆	–	–
Inclusion weight $w$	0.1	0.3, 0.1	0.3, 0.1	0.05	0.05

Appendix

Table 3: Settings per experiment. ⋆ Per dataset, in the order Boston, Energy, Concrete and Yacht,

s=0.3

0.03

0.2

0.01

and

c=0.75

0.5

10

0.5

Panel of Figure 5 — Figure 5: ZigZag and Boomerang on the banana target. (a) Posterior draws overlaid on the target’s density contours, and (b) the marginals against the exact analytic density.

Figure 6: Boomerang (blue) on the banana target for reference precision scalings $c$ (columns) and refreshment rates $\lambda_{\mathrm{ref}}$ (rows), with the ZigZag (green) as a fixed benchmark in every panel.

Figure 7: Sparse linear regression, pooled over five data sets, with Gibbs as a reference sampler. Left, posterior means of the first nine coefficients with

\pm 1

posterior standard deviation, and the true coefficients

\beta_{i}

. Right, deviation of the sticky PDMPs’ posterior inclusion probabilities from those of Gibbs, with a band of two Monte Carlo standard deviations.

Panel of Figure 8 — Figure 8: Posterior predictive fits on the Hernandez (top left), multiscale (top right) and gap (bottom) datasets. In each panel, the dashed black line is the true function, the solid coloured lines are the posterior predictive means, and the thin lines of the same colour lie two predictive standard deviations above and below the mean.

	Hernandez				Multiscale				Gap
Sampler	Sparsity	RMSE	NLL	CRPS	Sparsity	RMSE	NLL	CRPS	Sparsity	RMSE	NLL	CRPS
Boomerang	–	4.129	2.782	2.293	–	0.119	-0.710	0.067	–	0.199	-0.277	0.118
ZigZag	–	3.989	2.762	2.134	–	0.119	-0.707	0.066	–	0.224	-0.167	0.128
NUTS	–	4.121	2.781	2.309	–	0.119	-0.713	0.067	–	0.214	-0.275	0.116
Sticky Boomerang	87%	4.498	2.832	2.425	85%	0.127	-0.626	0.071	85%	0.210	-0.140	0.117
Sticky ZigZag	87%	4.369	2.811	2.383	84%	0.123	-0.665	0.069	86%	0.196	-0.141	0.111

Appendix

Table 4: Predictive performance on the one-dimensional datasets.

Network	Sampler	$w$	Sparsity	RMSE	NLL	CRPS	$C_{0.9}$
$d=502$	Boomerang	–	–	0.35 $\pm$ .02	0.37 $\pm$ .06	0.19 $\pm$ .01	90%
	ZigZag	–	–	0.34 $\pm$ .01	0.35 $\pm$ .03	0.19 $\pm$ .01	91%
	NUTS	–	–	0.33 $\pm$ .02	0.30 $\pm$ .04	0.18 $\pm$ .01	91%
	Sticky Boomerang	0.3	42%	0.34 $\pm$ .02	0.35 $\pm$ .07	0.19 $\pm$ .01	90%
	Sticky ZigZag	0.3	50%	0.34 $\pm$ .01	0.35 $\pm$ .03	0.19 $\pm$ .01	91%
	Sticky Boomerang	0.1	51%	0.35 $\pm$ .02	0.39 $\pm$ .07	0.19 $\pm$ .01	88%

Appendix

Table 5: Predictive performance on the UCI energy dataset, mean

\pm

SEM over 5 independent train/test splits. Sparsity denotes the percentage of weights sampled exactly zero, averaged over draws.

C_{0.9}

is the share of test responses inside the central 90% predictive interval, with an SEM of 1 to 3 percentage points.

Network	Sampler	$w$	Sparsity	RMSE	NLL	CRPS	$C_{0.9}$
$d=502$	Boomerang	–	–	4.97 $\pm$ .20	3.01 $\pm$ .04	2.70 $\pm$ .09	87%
	ZigZag	–	–	5.12 $\pm$ .15	3.03 $\pm$ .03	2.79 $\pm$ .07	88%
	NUTS	–	–	5.15 $\pm$ .19	3.03 $\pm$ .03	2.80 $\pm$ .08	89%
	Sticky Boomerang	0.3	54%	5.15 $\pm$ .22	3.05 $\pm$ .04	2.81 $\pm$ .09	88%
	Sticky ZigZag	0.3	61%	5.38 $\pm$ .20	3.09 $\pm$ .04	2.96 $\pm$ .10	88%
	Sticky Boomerang	0.1	72%	5.32 $\pm$ .24	3.08 $\pm$ .04	2.91 $\pm$ .10	87%

Appendix

Table 6: Predictive performance on the UCI concrete dataset, mean

\pm

SEM over 5 independent train/test splits. Sparsity denotes the percentage of weights sampled exactly zero, averaged over draws.

C_{0.9}

is the share of test responses inside the central 90% predictive interval, with an SEM of 0 to 2 percentage points.

Network	Sampler	$w$	Sparsity	RMSE	NLL	CRPS	$C_{0.9}$
$d=402$	Boomerang	–	–	0.32 $\pm$ .07	0.45 $\pm$ .26	0.17 $\pm$ .02	92%
	ZigZag	–	–	0.33 $\pm$ .07	0.68 $\pm$ .40	0.15 $\pm$ .02	87%
	NUTS	–	–	0.28 $\pm$ .08	0.15 $\pm$ .62	0.11 $\pm$ .02	88%
	Sticky Boomerang	0.3	54%	0.34 $\pm$ .07	0.84 $\pm$ .46	0.16 $\pm$ .02	85%
	Sticky ZigZag	0.3	56%	0.33 $\pm$ .06	0.63 $\pm$ .35	0.15 $\pm$ .02	86%
	Sticky Boomerang	0.1	67%	0.32 $\pm$ .06	0.56 $\pm$ .36	0.15 $\pm$ .02	85%

Appendix

Table 7: Predictive performance on the UCI yacht dataset, mean

\pm

SEM over 5 independent train/test splits. Sparsity denotes the percentage of weights sampled exactly zero, averaged over draws.

C_{0.9}

is the share of test responses inside the central 90% predictive interval, with an SEM of 1 to 5 percentage points.

Network	Sampler	$w$	$\log\sigma$	test RMSE	$\log\sigma$	test RMSE	$\log\sigma$	test RMSE
			$\hat{R}$		bulk ESS		bulk ESS per $10^{7}$ grad. evals
$d=752$	ZigZag	–	1.00	1.00	2,362	3,799	591	950
	Boomerang	–	1.01	1.00	2,622	9,629	656	2,407
	NUTS	–	1.00	1.00	4,854	3,961	12,965	10,580
	Sticky ZigZag	0.3	1.00	1.00	2,763	3,529	691	882
	Sticky Boomerang	0.3	1.01	1.00	1,676	4,095	419	1,024

Appendix

Table 8: Convergence diagnostics on the first Boston split, four chains per sampler started from different MAPs, with

10^{7}

gradient evaluations per PDMP chain and one NUTS chain per MAP. Rank-normalised

\hat{R}

and bulk ESS of

\log\sigma

and the test RMSE, pooled over the four chains, which hold

16{,}000

draws for the PDMPs and 4000 for NUTS. The last two columns are rates, the bulk ESS per

10^{7}

gradient evaluations, the budget of one PDMP chain. For NUTS, warmup is included when counting gradients.

Panel of Figure 9 — Figure 9: Traces after burn-in on the first Boston split, with $G=10^{6}$ (coloured) and $G=10^{7}$ (grey) gradient evaluations per chain. The NLL is per draw, not of the predictive mixture.

Figure 10: Effect of the prior inclusion weight $w$ on Boston for the small ( $d=752$ , top) and medium ( $d=5852$ , bottom) networks. (a) Share of weights that are exactly zero, averaged over the posterior draws, with the prior sparsity $1-w$ dotted. (b) Test RMSE and (c) test NLL, with dense NUTS as a horizontal reference.

Figure 11: Single segment against the grid bound. (a) Gradient evaluations per event under a constant rate

\lambda

as a function of the window length, from the expressions above (lines) and a simulation of both schemes (dots), with the window the adaptation settles at dotted. (b) Events per gradient evaluation with the grid (hatched) and the single segment (solid), on the small network for a single split of the Boston data.

	FFN					LeNet-5
Model	Sparsity	Acc	ECE	NLL	Brier	Sparsity	Acc	ECE	NLL	Brier
SGD	–	0.9836	0.0117	0.0854	0.0277	–	0.9912	0.0053	0.0342	0.0148
$\beta_{\mathrm{ref}}$	–	0.9812	0.0070	0.0620	0.0284	–	0.9908	0.0048	0.0297	0.0146
ZigZag	–	0.9756	0.0065	0.0806	0.0370	–	0.9923	0.0088	0.0307	0.0142
$\beta_{\mathrm{ref}}^{\star}$	91%	0.9821	0.0073	0.0585	0.0273	89%	0.9909	0.0055	0.0310	0.0144
Sticky ZigZag	98%	0.9797	0.0164	0.0759	0.0339	92%	0.9904	0.0107	0.0369	0.0163

Appendix

Table 9: Predictive performance on the 8000 MNIST test images not used for pruning, for the FFN and LeNet-5.

Panel of Figure 12 — Figure 12: The analyses of Figure 4 for the LeNet-5. (a) The six first-layer filters. (b) Robustness to Gaussian noise of standard deviation $\sigma$ on the $[0,1]$ pixel scale.

	Boston			Energy			Concrete			Yacht
$\Delta$	RMSE	NLL	CRPS	RMSE	NLL	CRPS	RMSE	NLL	CRPS	RMSE	NLL	CRPS
Boomerang	-.08	-.08	-.11	-.00	-.03	-.00	-.01	-.03	-.06	.01	.04	.04
ZigZag	-.11	-.08	-.07	-.00	-.08	-.00	.09	-.07	.09	-.01	-.31	-.01
Sticky Boomerang	-.02	-.01	.01	-.00	-.00	-.00	.08	.03	.06	-.00	.19	-.00
Sticky ZigZag	.04	.00	.03	-.01	-.07	-.00	.42	-.00	.27	-.01	-.69	-.01

Appendix

Table 10: Difference in predictive performance between full-batch and minibatch runs of the small network, averaged over the 5 splits of each UCI dataset. Values are full batch minus minibatch on the original scale, so negative values indicate a performance loss from minibatching. The minibatch runs use batches of 32 observations and

10^{5}

events, the full-batch runs

10^{6}

gradient evaluations. Both use the grid bound of an earlier version of our implementation.

Bayesian Neural Networks

Bayesian neural networks place probability distributions over neural-network parameters or functions to quantify predictive uncertainty and support robust decisions.

See all 90 papers

Dirichlet Scale Mixture Priors for Bayesian Neural Networks
Feb 23, 2026August Arnstad, Leiv Rønneberg, Geir StorvikBayesian Neural Networks Sparse Neural Networks
Department of Statistics & Data Science, University of Oslo.
Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales
Oct 7, 2026Marius P Linhard, Maurizio FilipponeBayesian Neural Networks Conditional Distribution Estimation
Université Grenoble Alpes, France · KAUST, Saudi Arabia
Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning
May 20, 2026Emanuel Sommer, David RügamerBayesian Neural Networks Efficient Neural Network Inference
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany.

Feb 23, 2026J/K move · Enter open · S save

Dirichlet Scale Mixture Priors for Bayesian Neural Networks

August Arnstad, Leiv Rønneberg, Geir Storvik

Department of Statistics & Data Science, University of Oslo.

Bayesian Neural Networks Sparse Neural Networks Bayesian Mixture Models

Open paper AskPDF arXiv

Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.

Abstract

Bayesian Neural Networks

Explore similar work

Dirichlet Scale Mixture Priors for Bayesian Neural Networks

Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning