Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.
Figures & tables
Figure 1: Sampled marginal bivariate posterior densities of two weights (axes, lighter means higher density) for an underparametrized (left) and an overparametrized model (right) on the same task using 10 million posterior samples.
Figure 2: Posterior marginals of 1 - M - 1 fully-connected network (FCN) with zero-mean Gaussian priors and different variances τ2 and data y=x+ϵ , ϵ∼N(0,0.1) .
Figure 3: Univariate marginal distributions of selected first hidden layer weights for sampled BNNs and their change with varying network size (number of parameters in brackets) on the concrete regression task. The 20 chain-wise distributions are overlaid with transparency, and the aggregated joint marginal is shown as a solid black line. Despite similar downstream performance, the marginals differ markedly between the two network parameterizations.
Figure 4: Bivariate marginal posterior densities of a 4-hidden layer BNN fitted on the ionosphere dataset. The grid visualizes the empirical densities of 8 million posterior samples obtained from 8,000 independent chains. Columns display representative densities of different layer weights.
Figure 5: Samples from a 1-2-1 ReLU network on a linear dataset (left) and its separation into image (center) and null space of J (right).
Figure 6: Bivariate marginal posterior densities of a 4-hidden layer BNN fitted on the airfoil dataset. The grid visualizes the empirical densities of 10M posterior samples obtained from 10k independent chains. The rows and columns display representative densities of randomly chosen layer weights.
Figure 7: Empirical marginal posterior view for ReLU networks with increasing widths. As the marginal coverage of the posterior increases, the functional variance recovered by the sampler increases as well.
Figure 8: Empirical covariance and correlation of weights w across posterior samples on the airfoil dataset. The sampler is able to recover an intricate correlation structure in the parameter space.
Figure 9: Minimal singular values S calculated for every hidden neuron cluster across all posterior samples, as well as median minimum singular value per neuron cluster per posterior sample w . The median (first row) or the median of medians (second row) is indicated in red.
Figure 10: Cumulative LPPD increase over the number of chains (standard deviation across 5 random chain orderings) for different architectures and datasets.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Size
Features
Source
Airfoil
1503
5
Dua and Graff (2017)
Bikesharing
17379
13
Fanaee-T (2013)
Concrete
1030
8
Yeh (1998)
Energy
768
8
Tsanas and Xifara (2012)
Ionosphere
351
34
Sigillito et al. (1989)
F(ashion)-MNIST
60000
28x28
Xiao et al. (2017)
Appendix
Table 1: Benchmark datasets overview.
Figure 11: Empirical marginal distribution of the bias of a four-hidden-layer MLP with a normal prior on the bias as well as a uniform prior on the bias .
Figure 12: Cumulative LPPD over the number of chains (standard deviation across 5 random chain orderings) on the airfoil dataset.
Figure 13: Cumulative LPPD over the number of chains (standard deviation across 5 random chain orderings) for the same CNNv1 setting on Fashion-MNIST as in Figure 14 (left) and the same fully-connected neural network (FCN) as the one in Figure 12 but fitted on the larger bikesharing dataset. As a reference, the DE with 1k members only achieves an LPPD of -0.3627 for CNNv1 and 0.4935 for the FCN. Also, the predictive accuracy of the BDE (1k) exceeds that of the DE (1k) in both settings (left: accuracy 0.8715>0.8706 , right: RMSE 0.2258<0.2508 ).
Figure 14: (Bivariate) marginal posterior densities of a small 4-hidden layer convolutional BNN fitted on the Fashion-MNIST dataset ( CNNv1 of Sommer et al., 2025 ). The grid visualizes the empirical densities of 1M posterior samples obtained from 1k independent chains. The rows and columns (Conv, three hidden, and output weights) display representative densities of randomly chosen weights of the network. We employ scale-adapted SGHMC ( Springenberg et al., 2016 ) for sampling, use an ensemble of 10 with 10k warmup steps, 10k sampling steps, thinning of 10, step size 0.001, momentum decay 0.05, batch size 256, and standard normal isotropic priors.
Figure 15: (Bivariate) marginal posterior densities of a 4-hidden layer fully-connected BNN fitted on the bikesharing dataset (regression task, and same architecture as in Figure 6 ). The grid visualizes the empirical densities of 1M posterior samples obtained from 1k independent chains. The rows and columns (input, three hidden, and output weights) display representative densities of randomly chosen weights of the network.
Figure 16: (Bivariate) marginal posterior densities of a 4-hidden layer fully-connected BNN fitted on the ionosphere dataset (binary classification task, and same architecture as in Figure 6 ). The grid visualizes the empirical densities of 8M posterior samples obtained from 8k independent chains. The rows and columns (input, three hidden, and output weights) display representative densities of randomly chosen weights of the network.
Figure 17: (Bivariate) marginal posterior densities of the underparametrized one-hidden layer fully-connected BNN fitted on the airfoil dataset (following the small f1 architecture of Wiese et al. (2023) ). The grid visualizes the empirical densities of 8M posterior samples obtained from 8k independent chains. The rows and columns (input and output weights) display representative densities of randomly chosen weights of the network.
Dataset
Laplace
MFVI
- DE (10)
- MCMC
LPPD
Airfoil
−1.056±0.003
−0.975±0.004
−0.293±0.096
−0.016±0.293
Bikesharing
−1.046±0.001
−0.990±0.005
−0.223±0.181
−0.060±0.096
Concrete
−1.131±0.036
−0.998±0.007
−0.510±0.189
−0.042±0.056
Energy
−1.046±0.004
−0.945±0.002
−1.561±0.101
−1.947±0.047
RMSE
Airfoil
−0.237±0.013
−0.276±0.009
−0.269±0.016
−0.184±0.016
Bikesharing
−0.252±0.006
−0.318±0.018
−0.253±0.015
−0.262±0.018
Appendix
Table 2: Mean RMSE ( ↓ ) and LPPD ( ↑ ) results ( ± standard deviation across 3 train-test splits) for a 3 hidden-layer fully-connected neural network on regression tasks. Numbers in brackets indicate the number of ensemble members/chains.
Laplace
MFVI
DE (10)
BDE (10)
Accuracy ( ↑ )
−0.7851±0.0067
−0.7133±0.0102
−0.8222±0.0013
−0.8273±0.0020
LPPD ( ↑ )
−1.6731±0.0571
−0.8410±0.0193
−0.5341±0.0050
−0.5149±0.0030
Wallclock Time in Minutes ( ↓ )
.12.92±0.41
.15.78±0.27
. 11.59±0.34
. 14.41±0.02
Appendix
Table 3: Image classification task on CIFAR-10 using a small ResNet with 76106 parameters. We employ scale-adapted SGHMC for sampling, use an ensemble of 10 with 5k warmup steps, 25k sampling steps, thinning of 250, step size 0.001, momentum decay 0.05, batch size 256, and standard normal isotropic priors. Mean accuracy ( ↑ ) and LPPD ( ↑ ) results ( ± standard deviation) for CIFAR10 classification are reported. Numbers in brackets indicate ensemble members/chains.
Model
Ensemble Members
Dataset
LPPD ( ↑ )
RMSE/Accuracy
DE
125
airfoil
0.0285±0.0266
0.2870±0.0240
BDE
10
airfoil
0.7660±0.0075
0.1415±0.0079
DE
8000
ionosphere
−0.2923
0.9154
BDE
1
ionosphere
−0.1544
0.9436
Appendix
Table 4: Computational cost and predictive performance comparison between DE and BDE on the airfoil and ionosphere datasets.
Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.
August Arnstad, Leiv Rønneberg, Geir Storvik
Department of Statistics & Data Science, University of Oslo.
The practical adoption of sampling-based inference (SAI) in Bayesian neural networks (BNNs) remains limited, partly due to persistent misconceptions about the feasibility and efficiency of sampling. This position paper argues that SAI has achieved computational parity with optimization-based methods and is at the verge of superseding such methods for effective and efficient inference in BNNs. This development should be in the interest of the whole community, promoting BNNs as a principled paradigm with its long-standing yet unfulfilled promise of providing principled uncertainty quantification for neural networks. SAI can even do more -- yielding superior prediction performance through model averaging, serving as the foundation for a plethora of possible downstream tasks, and providing crucial insights into the landscape of BNNs. In order to make such a change happen and unfold the potential of sampling, overcoming current misconceptions is a necessary first step. The next step is to realign research efforts toward addressing remaining challenges in SAI. In particular, the community must focus on two core problems: sufficient exploration of the posterior landscape and high-fidelity distillation of posterior samples for efficient downstream inference. By addressing conceptual and practical obstacles, we can unlock the full potential of SAI and establish it as a central tool in Bayesian deep learning.
Emanuel Sommer, David Rügamer
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany.
Bayesian neural networks need not be fully stochastic to be universal conditional density approximators, but it remains open which parameters should be stochastic. We learn this split by applying deep weight factorization to the prior scales, which are the standard deviations of the parameter priors, while fitting the functional prior to a Gaussian process with a maximum mean discrepancy objective. A parameter whose prior scale falls below a cutoff becomes deterministic and is optimized during inference, so the regularizer sparsifies stochasticity rather than capacity. We give a certificate for universal conditional density approximation that is checkable in linear time, together with a minimal repair when it fails. We further show that the common hybrid scheme of sampling some parameters and optimizing the others is stochastic approximation for a type-II maximum a posteriori objective, and that coupled step sizes can leave a tracking error that does not vanish as the step size shrinks. On a bimodal target, the learned split stays close to an unconstrained reference across all budgets and is insensitive to the cutoff, while random masks that distribute the same prior scales across layers are worse by up to two orders of magnitude. On UCI benchmarks, our method performs on par with a fully stochastic network while keeping about half of its parameters deterministic.
Marius P Linhard, Maurizio Filippone
Université Grenoble Alpes, France · KAUST, Saudi Arabia