Dirichlet Scale Mixture Priors for Bayesian Neural Networks

August Arnstad; Leiv Rønneberg; Geir Storvik

stat.MLFeb 23, 2026

Dirichlet Scale Mixture Priors for Bayesian Neural Networks

Authors: August Arnstad, Leiv Rønneberg, Geir Storvik

Organizations: Department of Statistics & Data Science, University of Oslo.

Abstract

Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.

Figures & tables

Figure 1: Monte Carlo estimates of the dispersion ratio

CV^{2}(\tilde{\lambda}_{j}^{\,2})

for different scale priors as a function of their tail-controlling parameter

(\sigma,k,a,b,\nu)

. The curves are evaluated at three fixed values of the regularization parameter: (i) the median of the prior on

c

, (ii) the

0.9

quantile of the prior on

c^{2}

, and (iii) a very large value of

c

, corresponding to an essentially unregularized regime. The horizontal line indicates the threshold

1/(p\alpha)

separating negative and positive covariance regimes. Smaller values of

c

attenuate dispersion and increasingly favor negative covariance among the variance components

\tilde{\lambda}_{j}^{\,2}\xi_{jk}

Figure 2:

p(\kappa\mid\sigma,\tau)

in the Dirichlet horseshoe (

\nu=1

), Dirichlet Student’s t ( Theorem 4.1 ), and the classical horseshoe ( Equation 12 ).

\kappa=1

indicates full shrinkage, and

\kappa=0

indicates no shrinkage at all. It is clear that the Dirichlet methods shrinks more aggressively, as the shrinkage factor

\kappa

has more mass close to

1

, than for the horseshoe.

Figure 3: Boxplot of posterior samples from

(w_{1},w_{5},w_{6})

for the linear regression model, for different correlations. The dotted blue line represents the underlying, true coefficient.

Figure 4: Estimated density of posterior samples for

w_{1},w_{5},w_{6}

(left) and histogram of

\kappa_{1},\kappa_{5},\kappa_{6}

(right) for the linear regression model, for

\rho=0.9

. The dashed black line represents the true coefficient, and the dotted purple line the GLS estimate.

Figure 5: Boxplots of seed-level median CRPS for each model and training sample size on the independent Friedman (left) and correlated Friedman (right) datasets. For each training size

N\in\{100,200,500\}

, five independent datasets are used. Each box summarizes the five median CRPS values, where each value is computed from posterior predictive ensembles evaluated on a large generated test set.

	Uncorrelated			Correlated
Model	N=100	N=200	N=500	N=100	N=200	N=500
Gauss	2.601 (0.035)	1.443 (0.042)	1.150 (0.018)	2.547 (0.083)	1.459 (0.060)	1.143 (0.017)
RHS	2.079 (0.033)	1.260 (0.035)	1.113 (0.018)	1.583 (0.107)	1.233 (0.063)	1.057 (0.011)
DHS	2.359 (0.036)	1.243 (0.037)	1.106 (0.018)	1.846 (0.199)	1.232 (0.056)	1.057 (0.011)
DST	1.887 (0.042)	1.252 (0.036)	1.107 (0.017)	1.515 (0.099)	1.215 (0.060)	1.049 (0.009)

Table 1: Boxplots of posterior mean RMSE for each model and training sample size on the correlated Friedman data. For each

N\in\{100,200,500\}

, five independent datasets are used. For each dataset, predictions are formed by averaging over posterior draws and RMSE is computed on a large generated test set.

Panel of Figure 6 — Figure 6: Trace plots of the effective number of non-zero parameters $m_{\mathrm{eff}}=\mathrm{tr}\!\big((P+S)^{-1}S\big)$ for different models on the Friedman dataset with independent and correlated input features. Each curve corresponds to a single fitted model and shows $m_{\mathrm{eff}}$ across 4000 posterior draws.

Panel of Figure 7 — Figure 7: Posterior mean RMSE as a function of sparsity level in Bayesian neural networks on Friedman datasets, aggregated across random seeds. The left panel shows the independent case and the right panel shows the correlated case. The upper panels correspond to the prune-per-sample scheme, the lower panels show posterior pruning.

Panel of Figure 8 — Figure 8: a) Pointwise CRPS distributions for each model on the Abalone dataset at three training sizes. Each box summarizes CRPS across test points computed from posterior predictive ensembles. b) Posterior mean RMSE as a function of sparsity level for the Abalone dataset. The left column shows sample-wise pruning, where masks are recomputed for each posterior draw, while the right column shows posterior pruning using a single global mask per model.

	$0.1N$		$0.2N$		$N$
Model	RMSE	PNLL	RMSE	PNLL	RMSE	PNLL
Gauss	2.918	2.534	2.662	2.433	1.990	2.103
RHS	2.578	2.434	2.243	2.232	1.949	2.087
DHS	2.475	2.365	2.240	2.229	1.965	2.096
DST	2.702	2.497	2.236	2.229	1.956	2.087

Table 2: Posterior mean RMSE and test-set negative log-likelihood on the Abalone dataset for three training sizes (10%, 20%, and full data).

Figure 9: The left figure shows the posterior mean network, in which the thickness of edges are proportional to the mean absolute value of weights. The right table displays feature importances from a SHAP analysis for the Abalone dataset.

Model	Acc	NLL	ECE
Gauss	0.9386	0.1288	0.0267
RHS	0.9649	0.1005	0.0320
DHS	0.9649	0.0943	0.0305
DST	0.9649	0.1004	0.0291

Table 3: Posterior mean performance of networks on the Breastcancer dataset.

Figure 10: Left: Robustness of the softmax probabilities in the classification on the breastcancer dataset. The x-axis displays the disturbance $\varepsilon$ , and the y-axis displays the fraction $\delta/\varepsilon$ . Right: Fractions of safe (S), partially safe (P), and unsafe (U) outcomes from the $M=4000$ samples, defined in terms of the probability of adversarial label change $p_{2}$ , on the breast cancer dataset.

Panel of Figure 11 — Figure 11: Eigenvalue spectra of $(I+G)^{-1}G$ across different priors on the Friedman dataset under two input settings.

Figure 12: The effective number of non-zero parameters for the different models, as calculated from

m_{\mathrm{eff}}=\mathrm{tr}\left((P+S)^{-1}S\right)

Figure 13: Eigenvalue curve

Figure 14: Boxplot of aggregated median CRPS across models and training sample sizes for the Friedman data.

	Uncorrelated			Correlated
Model	N=100	N=200	N=500	N=100	N=200	N=500
DHS	2.359	1.243	1.106	1.846	1.232	1.057
DST	1.875	1.252	1.107	1.515	1.215	1.049
BHS	1.912	1.247	1.107	1.513	1.214	1.049
BST	1.897	1.248	1.107	1.510	1.215	1.048

Table 4: Comparison of aggregated posterior RMSE for different models and training sample sizes.

Panel of Figure 15 — Figure 15: Effective number of non-zero parameters $m_{\mathrm{eff}}=\mathrm{tr}\!\big((P+S)^{-1}S\big)$ for different models on the Friedman dataset with independent and correlated input features.

Figure 16: Posterior mean RMSE as a function of sparsity level in the BNNs on the independent Friedman datasets. The upper panels show results for the prune per sample scheme, whereas the lower panel shows posterior pruning.

Figure 17: Posterior mean RMSE as a function of sparsity level in the BNNs on the correlated Friedman datasets. The upper panels show results for the prune per sample scheme, whereas the lower panel shows posterior pruning.

Figure 18: Correlation coefficient matrix for the Friedman dataset

Figure 19: Plot of

\hat{R}

for the network output variable across models and datasets for the Friedman dataset with

N\in\{100,200,500\}

Model	$\max\hat{R}$	Med $\hat{R}$	$N_{\mathrm{div}}/M$	Med $\mathrm{ESS}_{\mathrm{tail}}/M$	Med $\mathrm{ESS}_{\mathrm{bulk}}/M$	$N$
Gauss	1.001	1.000	0.004	0.901	0.895	100
Gauss	1.002	1.000	0.000	0.923	0.866	200
Gauss	1.009	1.001	0.000	0.853	0.610	500
RHS	1.002	1.000	0.007	0.886	0.717	100
RHS	1.002	1.000	0.000	0.921	0.849	200
RHS	1.002	1.000	0.000	0.845	0.644	500

Table 5: Sampler diagnostics

Panel of Figure 20 — Figure 20: Trace plots for the four priors.

Explore similar work

On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors
Mar 23, 2026Julius Kobialka, Emanuel Sommer, Chris Kolb +3Bayesian Neural Networks
LMU Munich, MCML · HTWG Konstanz
Minimaxity and Admissibility of Bayesian Neural Networks
Apr 6, 2026Daniel Andrew Coulson, Martin T. WellsBayesian Neural Networks Minimax Estimation
Department of Statistics and Data Science, Cornell University
From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference
Feb 26, 2026Gracielle Antunes de Araújo, Flávio B. GonçalvesBayesian Neural Networks Shallow Neural Networks
Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.

Mar 23, 2026J/K move · Enter open · S save

On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

Julius Kobialka, Emanuel Sommer, Chris Kolb, Juntae Kwon+2

LMU Munich, MCML · HTWG Konstanz

Bayesian Neural Networks

Open paper AskPDF arXiv

Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.

Abstract

Explore similar work

On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

Minimaxity and Admissibility of Bayesian Neural Networks

From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference