stat.MLJan 30, 2026

Singular Bayesian Neural Networks

Authors: Mame Diarra Toure, David A. Stephens

Organizations: Department of Mathematics and Statistics, McGill University, Montreal, Canada.

Abstract

Bayesian neural networks promise calibrated uncertainty but require O(mn)O(mn) parameters for standard mean-field Gaussian posteriors. We argue this cost is often unnecessary, particularly when weight matrices exhibit fast singular value decay. By parameterizing weights as W=AB⊤W = AB^{\top} with A∈Rm×rA \in \mathbb{R}^{m \times r}, B∈Rn×rB \in \mathbb{R}^{n \times r}, we induce a posterior that is singular with respect to the Lebesgue measure, concentrating on the rank-rr manifold. This singularity captures structured weight correlations through shared latent factors, geometrically distinct from mean-field's independence assumption. We derive PAC-Bayes generalization bounds whose complexity term scales as r(m+n)\sqrt{r(m+n)} instead of mn\sqrt{m n}, and prove loss bounds that decompose the error into optimization and rank-induced bias using the Eckart-Young-Mirsky theorem. We further adapt recent Gaussian complexity bounds for low-rank deterministic networks to Bayesian predictive means. Empirically, across MLPs, LSTMs, and Transformers on standard benchmarks, our method achieves competitive predictive performance while using up to 33×33\times fewer parameters than 5-member Deep Ensembles. It substantially improves OOD detection and often improves calibration relative to mean-field and perturbation baselines, while Deep Ensembles can still be stronger on in-distribution likelihood-based metrics.

Figures & tables

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 19, 2026

PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models.
Oct 15, 2026

Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes

Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space Θ⊆RDΘ\subseteq \mathbb{R}^D, causing relative entropy to scale with network dimension DD. For modern overparameterized neural networks where D≫nD \gg n, this parameter-counting dependency yields vacuous generalization bounds (>1,000%> 1{,}000\%). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions M1(Z)\mathcal{M}_1(\mathcal{Z}) via a Dirichlet Process (DP), DP(α,F0)\mathcal{DP}(α, F_0), inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count DD, width, and depth. On intrinsic d∗d^*-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate O(n−1/(d∗+2)ln⁡n)\mathcal{O}(n^{-1/(d^*+2)} \sqrt{\ln n}). Across vision architectures scaling up to 4.2 million parameters (D=7,850D = 7{,}850 to D=4,203,522D = 4{,}203{,}522) on the full MNIST benchmark, while classical weight-space bounds scale past 940%940\% (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{8.8%8.8\%--9.4%9.4\%}.
Oct 7, 2026

Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

Bayesian neural networks need not be fully stochastic to be universal conditional density approximators, but it remains open which parameters should be stochastic. We learn this split by applying deep weight factorization to the prior scales, which are the standard deviations of the parameter priors, while fitting the functional prior to a Gaussian process with a maximum mean discrepancy objective. A parameter whose prior scale falls below a cutoff becomes deterministic and is optimized during inference, so the regularizer sparsifies stochasticity rather than capacity. We give a certificate for universal conditional density approximation that is checkable in linear time, together with a minimal repair when it fails. We further show that the common hybrid scheme of sampling some parameters and optimizing the others is stochastic approximation for a type-II maximum a posteriori objective, and that coupled step sizes can leave a tracking error that does not vanish as the step size shrinks. On a bimodal target, the learned split stays close to an unconstrained reference across all budgets and is insensitive to the cutoff, while random masks that distribute the same prior scales across layers are worse by up to two orders of magnitude. On UCI benchmarks, our method performs on par with a fully stochastic network while keeping about half of its parameters deterministic.