On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

Julius Kobialka; Emanuel Sommer; Chris Kolb; Juntae Kwon; Daniel Dold; David Rügamer

cs.LGMar 23, 2026

On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

Authors: Julius Kobialka, Emanuel Sommer, Chris Kolb, Juntae Kwon, Daniel Dold, David Rügamer

Organizations: LMU Munich, MCML · HTWG Konstanz

Abstract

Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.

Figures & tables

Panel of Figure 1 — Figure 1: Sampled marginal bivariate posterior densities of two weights (axes, lighter means higher density) for an underparametrized (left) and an overparametrized model (right) on the same task using 10 million posterior samples.

Figure 2: Posterior marginals of 1 -

M

- 1 fully-connected network (FCN) with zero-mean Gaussian priors and different variances

\tau^{2}

and data

y=x+\epsilon

\epsilon\sim N(0,0.1)

Panel of Figure 3 — Figure 3: Univariate marginal distributions of selected first hidden layer weights for sampled BNNs and their change with varying network size (number of parameters in brackets) on the concrete regression task. The 20 chain-wise distributions are overlaid with transparency, and the aggregated joint marginal is shown as a solid black line. Despite similar downstream performance, the marginals differ markedly between the two network parameterizations.

Panel of Figure 4 — Figure 4: Bivariate marginal posterior densities of a 4-hidden layer BNN fitted on the ionosphere dataset. The grid visualizes the empirical densities of 8 million posterior samples obtained from 8,000 independent chains. Columns display representative densities of different layer weights.

Figure 5: Samples from a 1-2-1 ReLU network on a linear dataset (left) and its separation into image (center) and null space of $\mathbf{J}$ (right).

Panel of Figure 6 — Figure 6: Bivariate marginal posterior densities of a 4-hidden layer BNN fitted on the airfoil dataset. The grid visualizes the empirical densities of 10M posterior samples obtained from 10k independent chains. The rows and columns display representative densities of randomly chosen layer weights.

Figure 7: Empirical marginal posterior view for ReLU networks with increasing widths. As the marginal coverage of the posterior increases, the functional variance recovered by the sampler increases as well.

Panel of Figure 8 — Figure 8: Empirical covariance and correlation of weights $\mathbf{w}$ across posterior samples on the airfoil dataset. The sampler is able to recover an intricate correlation structure in the parameter space.

Figure 9: Minimal singular values

S

calculated for every hidden neuron cluster across all posterior samples, as well as median minimum singular value per neuron cluster per posterior sample

\mathbf{w}

. The median (first row) or the median of medians (second row) is indicated in red.

Panel of Figure 10 — Figure 10: Cumulative LPPD increase over the number of chains (standard deviation across 5 random chain orderings) for different architectures and datasets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Size	Features	Source
Airfoil	1503	5	Dua and Graff (2017)
Bikesharing	17379	13	Fanaee-T (2013)
Concrete	1030	8	Yeh (1998)
Energy	768	8	Tsanas and Xifara (2012)
Ionosphere	351	34	Sigillito et al. (1989)
F(ashion)-MNIST	60000	28x28	Xiao et al. (2017)

Appendix

Table 1: Benchmark datasets overview.

Figure 11: Empirical marginal distribution of the bias of a four-hidden-layer MLP with a normal prior on the bias as well as a uniform prior on the bias .

Figure 12: Cumulative LPPD over the number of chains (standard deviation across 5 random chain orderings) on the airfoil dataset.

Panel of Figure 13 — Figure 13: Cumulative LPPD over the number of chains (standard deviation across 5 random chain orderings) for the same CNNv1 setting on Fashion-MNIST as in Figure 14 (left) and the same fully-connected neural network (FCN) as the one in Figure 12 but fitted on the larger bikesharing dataset. As a reference, the DE with 1k members only achieves an LPPD of -0.3627 for CNNv1 and 0.4935 for the FCN. Also, the predictive accuracy of the BDE (1k) exceeds that of the DE (1k) in both settings (left: accuracy $0.8715>0.8706$ , right: RMSE $0.2258<0.2508$ ).

Figure 14: (Bivariate) marginal posterior densities of a small 4-hidden layer convolutional BNN fitted on the Fashion-MNIST dataset ( CNNv1 of Sommer et al., 2025 ). The grid visualizes the empirical densities of 1M posterior samples obtained from 1k independent chains. The rows and columns (Conv, three hidden, and output weights) display representative densities of randomly chosen weights of the network. We employ scale-adapted SGHMC ( Springenberg et al., 2016 ) for sampling, use an ensemble of 10 with 10k warmup steps, 10k sampling steps, thinning of 10, step size 0.001, momentum decay 0.05, batch size 256, and standard normal isotropic priors.

Panel of Figure 15 — Figure 15: (Bivariate) marginal posterior densities of a 4-hidden layer fully-connected BNN fitted on the bikesharing dataset (regression task, and same architecture as in Figure 6 ). The grid visualizes the empirical densities of 1M posterior samples obtained from 1k independent chains. The rows and columns (input, three hidden, and output weights) display representative densities of randomly chosen weights of the network.

Panel of Figure 16 — Figure 16: (Bivariate) marginal posterior densities of a 4-hidden layer fully-connected BNN fitted on the ionosphere dataset (binary classification task, and same architecture as in Figure 6 ). The grid visualizes the empirical densities of 8M posterior samples obtained from 8k independent chains. The rows and columns (input, three hidden, and output weights) display representative densities of randomly chosen weights of the network.

Panel of Figure 17 — Figure 17: (Bivariate) marginal posterior densities of the underparametrized one-hidden layer fully-connected BNN fitted on the airfoil dataset (following the small $f_{1}$ architecture of Wiese et al. (2023) ). The grid visualizes the empirical densities of 8M posterior samples obtained from 8k independent chains. The rows and columns (input and output weights) display representative densities of randomly chosen weights of the network.

	Dataset	Laplace	MFVI	- DE (10)	- MCMC
LPPD	Airfoil	$-1.056\pm 0.003$	$-0.975\pm 0.004$	$-0.293\pm 0.096$	$\mathbf{\phantom{-}0.016\pm 0.293}$
	Bikesharing	$-1.046\pm 0.001$	$-0.990\pm 0.005$	$-0.223\pm 0.181$	$\mathbf{-0.060\pm 0.096}$
	Concrete	$-1.131\pm 0.036$	$-0.998\pm 0.007$	$-0.510\pm 0.189$	$\mathbf{\phantom{-}0.042\pm 0.056}$
	Energy	$-1.046\pm 0.004$	$-0.945\pm 0.002$	$\phantom{-}1.561\pm 0.101$	$\mathbf{\phantom{-}1.947\pm 0.047}$
RMSE	Airfoil	$\phantom{-}0.237\pm 0.013$	$\phantom{-}0.276\pm 0.009$	$\phantom{-}0.269\pm 0.016$	$\mathbf{\phantom{-}0.184\pm 0.016}$
	Bikesharing	$\phantom{-}0.252\pm 0.006$	$\phantom{-}0.318\pm 0.018$	$\mathbf{\phantom{-}0.253\pm 0.015}$	$\phantom{-}0.262\pm 0.018$

Appendix

Table 2: Mean RMSE (

\downarrow

) and LPPD (

\uparrow

) results (

\pm

standard deviation across 3 train-test splits) for a 3 hidden-layer fully-connected neural network on regression tasks. Numbers in brackets indicate the number of ensemble members/chains.

	Laplace	MFVI	DE (10)	BDE (10)
Accuracy ( $\uparrow$ )	$\phantom{-}0.7851\pm 0.0067$	$\phantom{-}0.7133\pm 0.0102$	$\phantom{-}0.8222\pm 0.0013$	$\mathbf{\phantom{-}0.8273\pm 0.0020}$
LPPD ( $\uparrow$ )	$-1.6731\pm 0.0571$	$-0.8410\pm 0.0193$	$-0.5341\pm 0.0050$	$\mathbf{-0.5149\pm 0.0030}$
Wallclock Time in Minutes ( $\downarrow$ )	$\phantom{.}12.92\pm 0.41$	$\phantom{.}15.78\pm 0.27$	. $\mathbf{11.59\pm 0.34}$	. $14.41\pm 0.02$

Appendix

Table 3: Image classification task on CIFAR-10 using a small ResNet with 76106 parameters. We employ scale-adapted SGHMC for sampling, use an ensemble of 10 with 5k warmup steps, 25k sampling steps, thinning of 250, step size 0.001, momentum decay 0.05, batch size 256, and standard normal isotropic priors. Mean accuracy (

\uparrow

) and LPPD (

\uparrow

) results (

\pm

standard deviation) for CIFAR10 classification are reported. Numbers in brackets indicate ensemble members/chains.

Model	Ensemble Members	Dataset	LPPD ( $\uparrow$ )	RMSE/Accuracy
DE	125	airfoil	$0.0285\pm 0.0266$	$0.2870\pm 0.0240$
BDE	10	airfoil	$\mathbf{0.7660\pm 0.0075}$	$\mathbf{0.1415\pm 0.0079}$
DE	8000	ionosphere	$-0.2923$	$0.9154$
BDE	1	ionosphere	$\mathbf{-0.1544}$	$\mathbf{0.9436}$

Appendix

Table 4: Computational cost and predictive performance comparison between DE and BDE on the airfoil and ionosphere datasets.

Explore similar work

Dirichlet Scale Mixture Priors for Bayesian Neural Networks
Feb 23, 2026August Arnstad, Leiv Rønneberg, Geir StorvikBayesian Neural Networks Sparse Neural Networks
Department of Statistics & Data Science, University of Oslo.
Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning
May 20, 2026Emanuel Sommer, David RügamerBayesian Neural Networks Efficient Neural Network Inference
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany.
Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales
Oct 7, 2026Marius P Linhard, Maurizio FilipponeBayesian Neural Networks Conditional Distribution Estimation
Université Grenoble Alpes, France · KAUST, Saudi Arabia

Feb 23, 2026J/K move · Enter open · S save

Dirichlet Scale Mixture Priors for Bayesian Neural Networks

August Arnstad, Leiv Rønneberg, Geir Storvik

Department of Statistics & Data Science, University of Oslo.

Bayesian Neural Networks Sparse Neural Networks Bayesian Mixture Models

Open paper AskPDF arXiv

Neural networks are the cornerstone of modern machine learning, yet can be difficult to interpret, give overconfident predictions and are vulnerable to adversarial attacks. Bayesian neural networks (BNNs) provide some alleviation of these limitations, but have problems of their own. The key step of specifying prior distributions in BNNs is no trivial task, yet is often skipped out of convenience. In this work, we propose a new class of prior distributions for BNNs, the Dirichlet scale mixture (DSM) prior, that addresses current limitations in Bayesian neural networks through structured, sparsity-inducing shrinkage. Theoretically, we derive general dependence structures and shrinkage results for DSM priors and show how they manifest under the geometry induced by neural networks. In experiments on simulated and real world data we find that the DSM priors encourages sparse networks through implicit feature selection, show robustness under adversarial attacks and deliver competitive predictive performance with substantially fewer effective parameters. In particular, their advantages appear most pronounced in correlated, moderately small data regimes, and are more amenable to weight pruning. Moreover, by adopting heavy-tailed shrinkage mechanisms, our approach aligns with recent findings that such priors can mitigate the cold posterior effect, offering a principled alternative to the commonly used Gaussian priors.

Abstract

Explore similar work

Dirichlet Scale Mixture Priors for Bayesian Neural Networks

Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales