Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

Marius P Linhard; Maurizio Filippone

stat.MLOct 7, 2026

Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

Authors: Marius P Linhard, Maurizio Filippone

Organizations: Université Grenoble Alpes, France · KAUST, Saudi Arabia

Abstract

Bayesian neural networks need not be fully stochastic to be universal conditional density approximators, but it remains open which parameters should be stochastic. We learn this split by applying deep weight factorization to the prior scales, which are the standard deviations of the parameter priors, while fitting the functional prior to a Gaussian process with a maximum mean discrepancy objective. A parameter whose prior scale falls below a cutoff becomes deterministic and is optimized during inference, so the regularizer sparsifies stochasticity rather than capacity. We give a certificate for universal conditional density approximation that is checkable in linear time, together with a minimal repair when it fails. We further show that the common hybrid scheme of sampling some parameters and optimizing the others is stochastic approximation for a type-II maximum a posteriori objective, and that coupled step sizes can leave a tracking error that does not vanish as the step size shrinks. On a bimodal target, the learned split stays close to an unconstrained reference across all budgets and is insensitive to the cutoff, while random masks that distribute the same prior scales across layers are worse by up to two orders of magnitude. On UCI benchmarks, our method performs on par with a fully stochastic network while keeping about half of its parameters deterministic.

Figures & tables

Figure 1: Share of deterministic parameters per layer for DWF-PSNN before repair, with mean and standard deviation over ten splits

Figure 2: Predictive distribution at the bimodal input, with the true conditional dashed and certified configurations in blue. (a) Unconstrained reference, (b) output layer randomized, (c) no nonlinearity, (d) every weight randomized with fixed scales, (e) one randomized bias.

variant	condition violated	$W_{1}$
sampling floor	—	.020
point predictor	—	.681
noise input, $d_{L}$	reference, none	$.053\pm.004$
layer 1, $d_{L}$	none	$.074\pm.002$
layer 1, $d_{L}$ , weight	vanishes at $x=0$	$.284\pm.002$
output layer, $d_{L}$	bias coverage	$.568\pm.002$

Table 1: Mean Wasserstein-1 distance to the true conditional over nine held out inputs, with the standard error over five data seeds. The sampling floor is the distance between two independent samples from the true conditional at this sample size, and the point predictor is what a model reaches by matching the conditional median.

Figure 3: Learned and random splits on the bimodal target over five data seeds. Random masks permute the learned scales within each layer (circles) or across all layers (triangles) and are colored by whether they satisfy the certificate.

Figure 4: Cutoff applied after the optimization on the bimodal target, over five data seeds. Left, share of deterministic parameters. Right, distance to the true conditional after refitting with the thresholded scales.

$\gamma$	$0.03$	$2.5$	$10^{13}$
rmse ( $\downarrow$ )	.316 $\pm$ .033	.296 $\pm$ .029	.291 $\pm$ .027
nll ( $\downarrow$ )	.274 $\pm$ .095	.205 $\pm$ .074	.193 $\pm$ .066
ece ( $\downarrow$ )	.085 $\pm$ .059	.115 $\pm$ .083	.119 $\pm$ .074
$\delta$	.04 $\pm$ .01	23.97 $\pm$ .48	76.94 $\pm$ 4.41

Table 2: Results on CONCRETE for regimes (ii) and (iii) (mean

\pm

std), with the drift

\delta:=\|\boldsymbol{\theta}_{d}^{(T)}-\boldsymbol{\hat{\mu}}_{d}\|

of the deterministic parameters. The sampling hyperparameters differ from those of the benchmark ( Appendix E ).

Figure 5: Top (from left to right): the deterministic fraction against iterations and the smallest surviving and the largest pruned prior scales; the mean ( $\hat{f})$ and the uncentered second-moment $(\hat{g})$ terms on a logarithmic scale; the covariogram of the optimized prior against the target gp . Bottom (from left to right): DWF-PSNN prior and gp sample paths ; the gp posterior; the DWF-PSNN posterior

Classification ( $N=4$ )
Method	acc ( $\uparrow$ )	nll ( $\downarrow$ )	ece ( $\downarrow$ )
DWF-PSNN	2.38	2.63	2.88
ALS-BNN	2.63	2.63	3.13
FLS-BNN	2.88	2.88	2.50
RAND-PSNN	2.38	1.88	2.63
SNI-PSNN	4.75	5.00	3.88

Table 3: Mean ranks across the classification and regression datasets, computed from the mean performance on each dataset with tied ranks averaged. Lower ranks indicate better performance. The Nemenyi critical difference (CD) at

\alpha=0.05

3.05

for both tasks (

N=4

k=5

). The only pairwise difference that exceeds it is between RAND-PSNN and SNI-PSNN in classification nll .

Dataset	$d_{0}$	learned	repair	random
BANKNOTE	4	1.00	0.0	1.00
HTRU2	8	1.00	0.0	1.00
CONCRETE	8	0.90	0.1	0.97
WINE	10	0.00	5.5	0.09
BIKESHARING	12	0.00	7.3	0.01
BOSTON	13	0.00	7.8	0.00

Table 4: Certificate on the UCI datasets. Share of splits in which the learned mask satisfies the certificate before repair, mean number of parameters changed by the repair, and share of 2,000 random masks with the learned budget that satisfy the certificate.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$(d_{0},d_{L})$	Sample size	Parameters	Citation
CONCRETE	(8, 1)	1030	11101	( Yeh, 1998 )
BOSTON	(13, 1)	506	11601	( Harrison and Rubinfeld, 1978 )
BIKESHARING	(12, 1)	731	11501	( Fanaee-T, 2013 )
WINE	(10, 1)	4898	11301	( Cortez et al., 2009 )
ONLINESHOPPERS	(17, 2)	12330	12102	( Sakar and Kastro, 2018 )
CREDIT	(23, 2)	30000	12702	( Yeh, 2009 )

Appendix

Table 5: Dataset characteristics.

	acc in % $(\uparrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	100.00 $\pm$ 0.00	0.009 $\pm$ 0.002	0.009 $\pm$ 0.002	55.32 $\pm$ 2.56
ALS-BNN	100.00 $\pm$ 0.00	0.004 $\pm$ 0.001	0.004 $\pm$ 0.000	0.00
FLS-BNN	100.00 $\pm$ 0.00	0.004 $\pm$ 0.001	0.004 $\pm$ 0.001	95.37
RAND-PSNN	100.00 $\pm$ 0.00	0.005 $\pm$ 0.002	0.005 $\pm$ 0.001	55.32 $\pm$ 2.56
SNI-PSNN	99.20 $\pm$ 0.63	0.032 $\pm$ 0.010	0.025 $\pm$ 0.007	90.00

Appendix

Table 6: Results on BANKNOTE (

\mathrm{mean}\pm\mathrm{std}

	acc in % $(\uparrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	97.83 $\pm$ 0.48	0.069 $\pm$ 0.016	0.007 $\pm$ 0.003	57.31 $\pm$ 2.79
ALS-BNN	97.59 $\pm$ 0.50	0.078 $\pm$ 0.016	0.008 $\pm$ 0.003	0.00
FLS-BNN	97.56 $\pm$ 0.56	0.079 $\pm$ 0.015	0.008 $\pm$ 0.003	91.97
RAND-PSNN	97.82 $\pm$ 0.49	0.070 $\pm$ 0.015	0.007 $\pm$ 0.003	57.31 $\pm$ 2.79
SNI-PSNN	97.41 $\pm$ 0.58	0.086 $\pm$ 0.016	0.009 $\pm$ 0.002	90.00

Appendix

Table 7: Results on HTRU2 (

\mathrm{mean}\pm\mathrm{std}

	acc in % $(\uparrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	89.43 $\pm$ 0.64	0.243 $\pm$ 0.020	0.016 $\pm$ 0.005	61.01 $\pm$ 3.43
ALS-BNN	89.25 $\pm$ 0.74	0.261 $\pm$ 0.019	0.024 $\pm$ 0.009	0.00
FLS-BNN	89.25 $\pm$ 0.85	0.256 $\pm$ 0.020	0.018 $\pm$ 0.006	85.13
RAND-PSNN	89.48 $\pm$ 0.69	0.243 $\pm$ 0.020	0.015 $\pm$ 0.005	60.91 $\pm$ 3.44
SNI-PSNN	88.70 $\pm$ 0.90	0.292 $\pm$ 0.017	0.024 $\pm$ 0.012	90.00

Appendix

Table 8: Results on ONLINESHOPPERS (

\mathrm{mean}\pm\mathrm{std}

	acc in % $(\uparrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	78.02 $\pm$ 0.61	0.508 $\pm$ 0.010	0.022 $\pm$ 0.010	60.90 $\pm$ 2.61
ALS-BNN	78.03 $\pm$ 0.61	0.502 $\pm$ 0.009	0.019 $\pm$ 0.006	0.00
FLS-BNN	78.03 $\pm$ 0.61	0.506 $\pm$ 0.009	0.015 $\pm$ 0.005	81.11
RAND-PSNN	78.02 $\pm$ 0.61	0.500 $\pm$ 0.009	0.023 $\pm$ 0.006	60.68 $\pm$ 2.61
SNI-PSNN	78.02 $\pm$ 0.61	0.527 $\pm$ 0.007	0.014 $\pm$ 0.013	90.00

Appendix

Table 9: Results on CREDIT (

\mathrm{mean}\pm\mathrm{std}

	rmse $(\downarrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	0.298 $\pm$ 0.027	0.214 $\pm$ 0.073	0.107 $\pm$ 0.066	48.69 $\pm$ 1.27
ALS-BNN	0.300 $\pm$ 0.032	0.283 $\pm$ 0.068	0.134 $\pm$ 0.062	0.00
FLS-BNN	0.335 $\pm$ 0.030	0.286 $\pm$ 0.085	0.085 $\pm$ 0.062	91.89
RAND-PSNN	0.331 $\pm$ 0.034	0.322 $\pm$ 0.113	0.075 $\pm$ 0.073	48.69 $\pm$ 1.27
SNI-PSNN	0.332 $\pm$ 0.039	0.326 $\pm$ 0.133	0.097 $\pm$ 0.088	90.00

Appendix

Table 10: Results on CONCRETE (

\mathrm{mean}\pm\mathrm{std}

	rmse $(\downarrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	0.771 $\pm$ 0.023	2.210 $\pm$ 0.145	0.127 $\pm$ 0.015	47.12 $\pm$ 2.93
ALS-BNN	0.723 $\pm$ 0.031	1.049 $\pm$ 0.065	0.017 $\pm$ 0.015	0.00
FLS-BNN	0.856 $\pm$ 0.024	3.395 $\pm$ 0.202	0.207 $\pm$ 0.017	90.27
RAND-PSNN	0.841 $\pm$ 0.035	3.314 $\pm$ 0.294	0.149 $\pm$ 0.050	47.07 $\pm$ 2.93
SNI-PSNN	0.844 $\pm$ 0.036	3.338 $\pm$ 0.304	0.135 $\pm$ 0.029	90.00

Appendix

Table 11: Results on WINE (

\mathrm{mean}\pm\mathrm{std}

	rmse $(\downarrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	0.105 $\pm$ 0.034	-0.159 $\pm$ 0.034	0.585 $\pm$ 0.196	47.81 $\pm$ 2.95
ALS-BNN	0.155 $\pm$ 0.037	0.049 $\pm$ 0.038	0.432 $\pm$ 0.084	0.00
FLS-BNN	0.101 $\pm$ 0.040	-0.164 $\pm$ 0.041	0.614 $\pm$ 0.213	88.70
RAND-PSNN	0.101 $\pm$ 0.037	-0.174 $\pm$ 0.050	0.600 $\pm$ 0.191	47.75 $\pm$ 2.95
SNI-PSNN	0.104 $\pm$ 0.033	-0.173 $\pm$ 0.033	0.674 $\pm$ 0.180	90.00

Appendix

Table 12: Results on BIKESHARING (

\mathrm{mean}\pm\mathrm{std}

	rmse $(\downarrow)$	nll $(\downarrow)$	ece $(\downarrow)$	Deterministic parameters in %
DWF-PSNN	0.386 $\pm$ 0.113	0.482 $\pm$ 0.331	0.209 $\pm$ 0.145	48.03 $\pm$ 2.53
ALS-BNN	0.451 $\pm$ 0.107	0.470 $\pm$ 0.170	0.226 $\pm$ 0.166	0.00
FLS-BNN	0.392 $\pm$ 0.116	0.513 $\pm$ 0.337	0.213 $\pm$ 0.172	87.93
RAND-PSNN	0.398 $\pm$ 0.102	0.608 $\pm$ 0.385	0.196 $\pm$ 0.193	47.97 $\pm$ 2.53
SNI-PSNN	0.408 $\pm$ 0.106	0.652 $\pm$ 0.425	0.310 $\pm$ 0.305	90.00

Appendix

Table 13: Results on BOSTON (

\mathrm{mean}\pm\mathrm{std}

Dataset	Layer 1	Layer 2	Output layer	first-layer biases	output bias
	Deterministic parameters			Stochastic	Deterministic
ONLINESHOPPERS	$0.87\pm 0.01$	$0.56\pm 0.04$	$0.86\pm 0.03$	9–17	6/10
CREDIT	$0.86\pm 0.01$	$0.54\pm 0.03$	$0.89\pm 0.02$	7–20	10/10
BANKNOTE	$0.87\pm 0.02$	$0.53\pm 0.03$	$0.84\pm 0.02$	8–18	0/10
HTRU2	$0.87\pm 0.01$	$0.54\pm 0.03$	$0.89\pm 0.02$	10–18	7/10
CONCRETE	$0.80\pm 0.01$	$0.46\pm 0.01$	$0.78\pm 0.04$	11–22	4/10

Appendix

Table 14: Learned split of DWF-PSNN. The first columns give the share of deterministic parameters per layer, with mean and standard deviation over the ten splits. The next column gives the range over splits of the number of stochastic first-layer biases out of 100. The last column gives the number of splits in which the output bias is deterministic.

			$W_{1}$
variant	condition violated	$n_{\mathrm{stoch}}$	fixed	learned
sampling floor	—	—	0.020	0.020
point predictor	—	0	0.681	0.681
noise input, $d_{L}$	reference, none	0	$0.053\pm 0.004$	$0.053\pm 0.004$
first layer, $d_{L}$	none	1	$0.074\pm 0.002$	$0.075\pm 0.002$
first layer, $d_{L}$ , weight	vanishes at $x=0$	1	$0.284\pm 0.002$	$0.320\pm 0.010$

Appendix

Table 15: The violated conditions are the ones in Proposition 4.7 . Values are the mean Wasserstein-1 distance to the true conditional over nine held out inputs, with the standard error over five data seeds. The sampling floor is the distance between two independent samples from the true conditional at this sample size, so no model can go below it, and the point predictor is what a model reaches by matching the conditional median without representing any spread. The last column repeats the study with learnable prior scales.

Figure 6: Distance to the true conditional against the number of randomized parameters in the output layer, with the certified configuration and the sampling floor as horizontal lines. Mean over five data seeds with the standard error.

Figure 7: All variants of Table 15 at three inputs, with fixed prior scales. In the panel titles

d

denotes the output dimension

d_{L}

, rank denotes the condition

|T|\geq d_{0}

on the fully deterministic units and no noise corresponds to no randomness in Table 15 . Grey dashed is the true conditional, blue marks the configurations the certificate admits. The vertical range is fixed by the target density, so mass outside it appears flat.

Figure 8: Histogram of

\log_{10}|u_{\theta}|

on the bimodal target over five data seeds, with the cutoff

c=0.01

on the prior scale, which corresponds to

|u|\approx 0.2

, as a dashed line. For larger

\lambda

the pruned products

u_{\theta}

collapse towards zero.

Hyperparameter	Value
Batch size	64
Number of iterations	10000
Regularization ( $\lambda$ )	(9, 12, 15) / D
Number of function samples	100
Learning rate	0.01
Kernel	Polynomial of degree 2

Appendix

Table 16: Hyperparameters used for prior optimization

Hyperparameter	Value
Batch size	64
Number of samples per chain	30
Discarded samples before sampling	10
Thinning interval	200
Number of chains	4
Momentum decay	0.02

Appendix

Table 17: Hyperparameters used for all models but SNI-PSNN

Hyperparameter	Value
Batch size	64
Prior precision	1.0
Stochastic fraction	0.1
Noise variance	0.1
Epochs	1000
Learning rate	0.01

Appendix

Table 18: Hyperparameters used for SNI-PSNN

Hyperparameter	Value
Batch size	64
Number of samples per chain	30
Discarded samples before sampling	10
Thinning interval	200
Number of chains	4
Momentum decay	0.02

Appendix

Table 19: Hyperparameters used for all models but SNI-PSNN

Hyperparameter	Value
Batch size	64
Prior precision	1.0
Stochastic fraction	0.1
Noise variance	0.1
Epochs	1000
Learning rate	0.01

Appendix

Table 20: Hyperparameters used for SNI-PSNN

Hyperparameter	Value
Batch size	128
Number of iterations	15000
Regularization ( $\lambda$ )	(1.2, 2.4, 4.8) / D
Number of function samples	100
Learning rate	0.03
Kernel	Polynomial of degree 2

Appendix

Table 21: Hyperparameters used for prior optimization

Hyperparameter	Value
Batch size	64
Number of samples per chain	60
Discarded samples before sampling	20
Thinning interval	100
Number of chains	4
Momentum decay	0.01

Appendix

Table 22: Sampling hyperparameters

Hyperparameter	Value
Batch size	64
Number of samples per chain	100
Discarded samples before sampling	10
Number of burn-in steps	10000
Thinning interval	20
Number of chains	4

Appendix

Table 23: Hyperparameters used for sampling

Explore similar work

Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning
May 20, 2026Emanuel Sommer, David RügamerBayesian Neural Networks Efficient Neural Network Inference
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany.
Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime
May 28, 2026Paolo Baglioni, Christian Keup, Vincenzo Zimbardo +4Bayesian Neural Networks Neural Network Generalization
1INFN, Sezione di Milano Bicocca, Piazza della Scienza 3, 20126, Milano, Italy · 2INFN, Gruppo Collegato di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy · 3Dipartimento di Scienze Matematiche, Fisiche e Informatiche, Universit`a degli Studi di Parma, Parco Area delle Scienze, 7/A 43124 Parma, Italy +2
When Has a Bayesian Neural Network Sampled Enough? Adaptive Inference Time with Statistical Guarantees
Oct 8, 2026Fabian Denoodt, Sibylle HessAdaptive Sampling Bayesian Neural Networks
Eindhoven University of Technology

May 20, 2026, cs.LGJ/K move · Enter open · S save

Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

Emanuel Sommer, David Rügamer

Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany.

Bayesian Neural Networks Efficient Neural Network Inference Bayesian Inference Posterior Sampling

Open paper AskPDF arXiv

The practical adoption of sampling-based inference (SAI) in Bayesian neural networks (BNNs) remains limited, partly due to persistent misconceptions about the feasibility and efficiency of sampling. This position paper argues that SAI has achieved computational parity with optimization-based methods and is at the verge of superseding such methods for effective and efficient inference in BNNs. This development should be in the interest of the whole community, promoting BNNs as a principled paradigm with its long-standing yet unfulfilled promise of providing principled uncertainty quantification for neural networks. SAI can even do more -- yielding superior prediction performance through model averaging, serving as the foundation for a plethora of possible downstream tasks, and providing crucial insights into the landscape of BNNs. In order to make such a change happen and unfold the potential of sampling, overcoming current misconceptions is a necessary first step. The next step is to realign research efforts toward addressing remaining challenges in SAI. In particular, the community must focus on two core problems: sufficient exploration of the posterior landscape and high-fidelity distillation of posterior samples for efficient downstream inference. By addressing conceptual and practical obstacles, we can unlock the full potential of SAI and establish it as a central tool in Bayesian deep learning.

Abstract

Explore similar work

Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime

When Has a Bayesian Neural Network Sampled Enough? Adaptive Inference Time with Statistical Guarantees