Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.
Figures & tables
Figure 1: Monte Carlo estimates of the dispersion ratio CV2(λ~j2) for different scale priors as a function of their tail-controlling parameter (σ,k,a,b,ν) . The curves are evaluated at three fixed values of the regularization parameter: (i) the median of the prior on c , (ii) the 0.9 quantile of the prior on c2 , and (iii) a very large value of c , corresponding to an essentially unregularized regime. The horizontal line indicates the threshold 1/(pα) separating negative and positive covariance regimes. Smaller values of c attenuate dispersion and increasingly favor negative covariance among the variance components λ~j2ξjk .
Figure 2: p(κ∣σ,τ) in the Dirichlet horseshoe ( ν=1 ), Dirichlet Student’s t ( Theorem 4.1 ), and the classical horseshoe ( Equation 12 ). κ=1 indicates full shrinkage, and κ=0 indicates no shrinkage at all. It is clear that the Dirichlet methods shrinks more aggressively, as the shrinkage factor κ has more mass close to 1 , than for the horseshoe.
Figure 3: Boxplot of posterior samples from (w1,w5,w6) for the linear regression model, for different correlations. The dotted blue line represents the underlying, true coefficient.
Figure 4: Estimated density of posterior samples for w1,w5,w6 (left) and histogram of κ1,κ5,κ6 (right) for the linear regression model, for ρ=0.9 . The dashed black line represents the true coefficient, and the dotted purple line the GLS estimate.
Figure 5: Boxplots of seed-level median CRPS for each model and training sample size on the independent Friedman (left) and correlated Friedman (right) datasets. For each training size N∈{100,200,500} , five independent datasets are used. Each box summarizes the five median CRPS values, where each value is computed from posterior predictive ensembles evaluated on a large generated test set.
Uncorrelated
Correlated
Model
N=100
N=200
N=500
N=100
N=200
N=500
Gauss
2.601 (0.035)
1.443 (0.042)
1.150 (0.018)
2.547 (0.083)
1.459 (0.060)
1.143 (0.017)
RHS
2.079 (0.033)
1.260 (0.035)
1.113 (0.018)
1.583 (0.107)
1.233 (0.063)
1.057 (0.011)
DHS
2.359 (0.036)
1.243 (0.037)
1.106 (0.018)
1.846 (0.199)
1.232 (0.056)
1.057 (0.011)
DST
1.887 (0.042)
1.252 (0.036)
1.107 (0.017)
1.515 (0.099)
1.215 (0.060)
1.049 (0.009)
Table 1: Boxplots of posterior mean RMSE for each model and training sample size on the correlated Friedman data. For each N∈{100,200,500} , five independent datasets are used. For each dataset, predictions are formed by averaging over posterior draws and RMSE is computed on a large generated test set.
Figure 6: Trace plots of the effective number of non-zero parameters meff=tr((P+S)−1S) for different models on the Friedman dataset with independent and correlated input features. Each curve corresponds to a single fitted model and shows meff across 4000 posterior draws.
Figure 7: Posterior mean RMSE as a function of sparsity level in Bayesian neural networks on Friedman datasets, aggregated across random seeds. The left panel shows the independent case and the right panel shows the correlated case. The upper panels correspond to the prune-per-sample scheme, the lower panels show posterior pruning.
Figure 8: a) Pointwise CRPS distributions for each model on the Abalone dataset at three training sizes. Each box summarizes CRPS across test points computed from posterior predictive ensembles. b) Posterior mean RMSE as a function of sparsity level for the Abalone dataset. The left column shows sample-wise pruning, where masks are recomputed for each posterior draw, while the right column shows posterior pruning using a single global mask per model.
0.1N
0.2N
N
Model
RMSE
PNLL
RMSE
PNLL
RMSE
PNLL
Gauss
2.918
2.534
2.662
2.433
1.990
2.103
RHS
2.578
2.434
2.243
2.232
1.949
2.087
DHS
2.475
2.365
2.240
2.229
1.965
2.096
DST
2.702
2.497
2.236
2.229
1.956
2.087
Table 2: Posterior mean RMSE and test-set negative log-likelihood on the Abalone dataset for three training sizes (10%, 20%, and full data).
Figure 9: The left figure shows the posterior mean network, in which the thickness of edges are proportional to the mean absolute value of weights. The right table displays feature importances from a SHAP analysis for the Abalone dataset.
Model
Acc
NLL
ECE
Gauss
0.9386
0.1288
0.0267
RHS
0.9649
0.1005
0.0320
DHS
0.9649
0.0943
0.0305
DST
0.9649
0.1004
0.0291
Table 3: Posterior mean performance of networks on the Breastcancer dataset.
Figure 10: Left: Robustness of the softmax probabilities in the classification on the breastcancer dataset. The x-axis displays the disturbance ε , and the y-axis displays the fraction δ/ε . Right: Fractions of safe (S), partially safe (P), and unsafe (U) outcomes from the M=4000 samples, defined in terms of the probability of adversarial label change p2 , on the breast cancer dataset.
Figure 11: Eigenvalue spectra of (I+G)−1G across different priors on the Friedman dataset under two input settings.
Figure 12: The effective number of non-zero parameters for the different models, as calculated from meff=tr((P+S)−1S)
Figure 13: Eigenvalue curve
Figure 14: Boxplot of aggregated median CRPS across models and training sample sizes for the Friedman data.
Uncorrelated
Correlated
Model
N=100
N=200
N=500
N=100
N=200
N=500
DHS
2.359
1.243
1.106
1.846
1.232
1.057
DST
1.875
1.252
1.107
1.515
1.215
1.049
BHS
1.912
1.247
1.107
1.513
1.214
1.049
BST
1.897
1.248
1.107
1.510
1.215
1.048
Table 4: Comparison of aggregated posterior RMSE for different models and training sample sizes.
Figure 15: Effective number of non-zero parameters meff=tr((P+S)−1S) for different models on the Friedman dataset with independent and correlated input features.
Figure 16: Posterior mean RMSE as a function of sparsity level in the BNNs on the independent Friedman datasets. The upper panels show results for the prune per sample scheme, whereas the lower panel shows posterior pruning.
Figure 17: Posterior mean RMSE as a function of sparsity level in the BNNs on the correlated Friedman datasets. The upper panels show results for the prune per sample scheme, whereas the lower panel shows posterior pruning.
Figure 18: Correlation coefficient matrix for the Friedman dataset
Figure 19: Plot of R^ for the network output variable across models and datasets for the Friedman dataset with N∈{100,200,500}
Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.
Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.
Daniel Andrew Coulson, Martin T. Wells
Department of Statistics and Data Science, Cornell University
In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.
Gracielle Antunes de Araújo, Flávio B. Gonçalves
Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.