Singular Bayesian Neural Networks
Organizations: Department of Mathematics and Statistics, McGill University, Montreal, Canada.
Abstract
Bayesian neural networks promise calibrated uncertainty but require parameters for standard mean-field Gaussian posteriors. We argue this cost is often unnecessary, particularly when weight matrices exhibit fast singular value decay. By parameterizing weights as with , , we induce a posterior that is singular with respect to the Lebesgue measure, concentrating on the rank- manifold. This singularity captures structured weight correlations through shared latent factors, geometrically distinct from mean-field's independence assumption. We derive PAC-Bayes generalization bounds whose complexity term scales as instead of , and prove loss bounds that decompose the error into optimization and rank-induced bias using the Eckart-Young-Mirsky theorem. We further adapt recent Gaussian complexity bounds for low-rank deterministic networks to Bayesian predictive means. Empirically, across MLPs, LSTMs, and Transformers on standard benchmarks, our method achieves competitive predictive performance while using up to fewer parameters than 5-member Deep Ensembles. It substantially improves OOD detection and often improves calibration relative to mean-field and perturbation baselines, while Deep Ensembles can still be stronger on in-distribution likelihood-based metrics.
Figures & tables
| Model | AUROC | AUPR-Err | AUC-OOD | AUPR-OOD | AUPR-In | NLL | Params |
|---|---|---|---|---|---|---|---|
| Deterministic | .922 ±.003 | .145 ±.035 | .500 ±.000 | .544 ±.000 | .456 ±.000 | .284 ±.017 | 22.4k |
| Deep Ens. | .929 ±.002 | .237 ±.035 | .738 ±.061 | .754 ±.048 | .721 ±.053 | .300 ±.021 | 112k |
| Full-Rank | .895 ±.004 | .412 ±.029 | .770 ±.051 | .759 ±.055 | .807 ±.035 | .401 ±.033 | 44.8k |
| Low-Rank (ours) | .895 ±.001 | .540 ±.027 | .802 ±.018 | .788 ±.034 | .824 ±.013 | .433 ±.020 | 13.6k |
| LR-SVD (ours) | .898 ±.001 | .486 ±.155 | .713 ±.069 | .719 ±.046 | .738 ±.074 | .452 ±.035 | 13.6k |
| Rank-1 | .901 ±.010 | .322 ±.019 | .705 ±.073 | .742 ±.062 | .663 ±.080 | .317 ±.024 | 23.3k |
| Model | MAE | ECE | PICP | AUROC-OOD | AUPR-OOD | Params |
|---|---|---|---|---|---|---|
| Deterministic | 10.79 ±0.16 | – | – | 0.500 | 0.500 | 33K |
| Full-Rank BBB | 10.55 ±0.11 | 0.111 ±0.007 | 0.788 ±0.012 | 0.492 ±0.039 | 0.743 ±0.019 | 132K |
| Low-Rank (ours) | 10.63 ±0.20 | 0.114 ±0.005 | 0.790 ±0.006 | 0.710 ±0.021 | 0.861 ±0.022 | 47K |
| LR-SVD (ours) | 10.70 ±0.03 | 0.139 ±0.022 | 0.773 ±0.022 | 0.687 ±0.068 | 0.812 ±0.026 | 47K |
| Rank-1 Mult. | 10.80 ±0.15 | 0.307 ±0.015 | 0.449 ±0.025 | 0.580 ±0.046 | 0.751 ±0.028 | 66K |
| Deep Ensemble (5) | 10.45 ±0.03 | 0.317 ±0.012 | 0.310 ±0.021 | 0.730 ±0.029 | 0.883 ±0.012 | 330K |
| Model | Acc | NLL | AUROC-OOD | MI Ratio | AUPR-In | AUPR-Succ | Params |
|---|---|---|---|---|---|---|---|
| Deterministic | .812 ±.006 | .490 ±.003 | .500 ±.000 | .000 ±.000 | .102 ±.000 | .812 ±.006 | 9.9M |
| Deep Ens. | .825 ±.006 | .434 ±.023 | .657 ±.009 | 1.55 ±0.05 | .267 ±.016 | .933 ±.003 | 49.6M |
| Full-Rank | .752 ±.011 | .552 ±.010 | .622 ±.019 | 1.31 ±0.07 | .222 ±.028 | .888 ±.006 | 19.8M |
| Low-Rank (ours) | .806 ±.003 | .527 ±.006 | .640 ±.028 | 1.35 ±0.10 | .302 ±.027 | .917 ±.005 | 1.5M |
| LR-SVD (ours) | .800 ±.005 | .516 ±.002 | .616 ±.011 | 1.22 ±0.10 | .272 ±.012 | .923 ±.003 | 1.5M |
| Rank-1 | .795 ±.013 | .507 ±.005 | .613 ±.015 | 1.11 ±0.09 | .271 ±.027 | .931 ±.004 | 10.0M |
| Model | Params | Peak Mem. | Epoch Time |
|---|---|---|---|
| Low-Rank BBB | 1.47M | 357.5MB | 5.88s |
| Full-Rank BBB | 19.84M | 721.1MB | 6.45s |
| Deep Ensemble | 49.61M | 670.1MB | 18.99s |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Main Paper | Statement | Appendix Location |
| Related Work and background | ||
| – | Comparison with related low-rank approaches | Section A |
| Section 3.1 : Geometry and Singularity | ||
| Definition 3.1 | Induced posterior | Defined in main text |
| Lemma 3.2 | Constrained support | Section B.4 |
| Lemma 3.3 | Measure zero of | Section B.5 |
| Method | Backbone | Parameter count | Posterior | Weight |
|---|---|---|---|---|
| training | (mean + var) | support | correlations | |
| Mean-field VI ( Blundell et al., 2015 ) | Learned Bayes. | Full | Independent | |
| Low-rank cov. ( Swiatkowski et al., 2020 ; Tomczak et al., 2020 ) | Learned Bayes. | Full | Via covariance | |
| Rank-1 mult. ( Dusenberry et al., 2020a ) | Learned det. + pert. | Full | Rank-1 structure | |
| Bella ( Doan et al., 2025 ) | Frozen + adapt | Affine subspace | Via | |
| LoRA + Bayes ( Yang et al., 2024 ) | Frozen + adapt | Affine subspace | Via |
| Loss Function | (Lipschitz) | (Smooth) | Notes |
|---|---|---|---|
| MAE | 1 | N/A | Not differentiable |
| MSE | 1 | Requires bounded outputs | |
| Cross-Entropy | Global for softmax | ||
| Binary Cross-Entropy | 1 | Global for sigmoid | |
| Hinge Loss | 1 | N/A | Not differentiable everywhere |
| Model | Parameters | Initialization |
|---|---|---|
| Deterministic | 22.4K | Glorot uniform |
| Deep Ensemble (5×) | 112.7K | Glorot uniform, different seeds |
| Full-Rank BBB | 44.8K | , |
| Low-Rank ( ) | 13.6K | He/Glorot with damping |
| LR-SVD ( ) | 13.6K | SVD of pretrained deterministic |
| Rank-1 Mult. | 23.3K | Glorot + small perturbations |
| Model | Layer Type | Rank | Initialization |
|---|---|---|---|
| Deterministic | Dense | – | Glorot (input), Orthogonal (recurrent) |
| Deep Ensemble ( ) | Dense | – | Glorot/Orthogonal, different seeds |
| Full-Rank BBB | DenseVariational | – | , |
| Low-Rank ( ) | LowRankDenseVar. | [14,20] | Adaptive with damping (Sec. F ) |
| LR-SVD ) | LowRankDenseVar. | [14,20] | SVD of pretrained deterministic |
| Rank-1 Mult. | Rank1DenseVar. | 1 | Glorot base + small perturbations |
| Model | Params | Init (abbr.) |
|---|---|---|
| Det. | 9.92M | Glorot (Dense), Unif (Emb) |
| DE (5×) | 49.61M | Det. init, diff seeds |
| FR-BBB | 19.84M | , |
| LR-BBB ( ) | 1.47M | G/H adapt., damp=0.85; rand; |
| LR-SVD ( ) | 1.47M | SVD init from Det. |
| R1-BBB | 10.01M | Glorot; |
| Method | Training Time (s) |
|---|---|
| Deterministic Dense | 52.5 |
| Deep Ensemble (5 models) | 221.1 |
| Full-Rank BBB | 502.4 |
| Rank-1 Multiplicative | 503.2 |
| Low-Rank Gaussian SVD init | 524.0 |
| Low-Rank Gaussian ( ) | 532.2 |
| Model | Calib. AUC |
|---|---|
| Full-Rank BBB | 0.110 |
| Low-Rank ( 14,20) | 0.104 |
| Low-Rank (SVD) | 0.126 |
| Rank-1 Mult. | 0.302 |
| Deep Ensemble (5) | 0.307 |
| Model | 100% Retention | 90% Retention | 80% Retention |
|---|---|---|---|
| Full-Rank BBB | 10.55 (0.0%) | 9.46 (11.2%) | 8.82 (17.2%) |
| Low-Rank BBB | 10.63 (0.0%) | 9.43 (10.6%) | 8.71 (17.4%) |
| Low-Rank SVD | 10.70 (0.0%) | 9.50 (11.8%) | 9.06 (15.8%) |
| Rank-1 BBB | 10.80 (0.0%) | 9.45 (12.5%) | 9.42 (12.8%) |
| Deep Ensemble | 10.45 (0.0%) | 9.45 (9.1%) | 9.21 (11.4%) |
| Model | Params | Peak Mem. | Epoch Time |
|---|---|---|---|
| Baseline Transformer | 9.92M | 505.4MB | 4.18s |
| Low-Rank BBB | 1.47M | 357.5MB | 5.88s |
| Full-Rank BBB | 19.84M | 721.1MB | 6.45s |
| Deep Ensemble (5) | 49.61M | 670.1MB | 18.99s |
| Params | Peak Mem. | Epoch Time | Single Pass | MC20 / MC100 | ||
|---|---|---|---|---|---|---|
| 4 | 8 | 0.50M | 337.8MB | 5.06s | 0.214s | 6.18s / 30.65s |
| 8 | 12 | 0.86M | 344.7MB | 5.41s | 0.221s | 6.26s / 31.39s |
| 10 | 16 | 1.10M | 359.8MB | 5.88s | 0.228s | 6.51s / 31.75s |
| 12 | 24 | 1.46M | 365.0MB | 5.70s | 0.225s | 6.36s / 32.18s |
| 16 | 32 | 1.94M | 380.2MB | 6.18s | 0.229s | 6.49s / 32.95s |
| 24 | 48 | 2.90M | 421.9MB | 6.29s | 0.232s | 6.55s / 32.65s |
| Model | Avg. Time (min) |
|---|---|
| Baseline Transformer | 7.70 |
| Low-Rank SVD Init | 7.95 |
| Low-Rank Random Init | 8.22 |
| Rank-1 BBB | 12.52 |
| Full-Rank BBB | 23.08 |
| Deep Ensemble | 64.72 |
| AUROC-OOD | AUPR-OOD | AUPR-In | ||||
|---|---|---|---|---|---|---|
| Model | MI | STD | MI | STD | MI | STD |
| Deterministic | .500 | .500 | .897 | .897 | .103 | .103 |
| Deep Ensemble | .658 | .663 | .930 | .932 | .267 | .282 |
| Full-Rank BBB | .622 | .623 | .921 | .921 | .222 | .223 |
| Low-Rank BBB | .641 | .642 | .919 | .920 | .302 | .303 |
| LR-SVD BBB | .616 | .619 | .903 | .904 | .273 | .274 |
| Model | Acc | AUROC-OOD-MI | ECE | NLL |
|---|---|---|---|---|
| Low-Rank Bayesian Ensemble | 0.814 | 0.731 | 0.054 | 0.415 |
| Low-Rank BBB (single run) | 0.795 | 0.647 | 0.149 | 0.520 |
| Low-Rank BBB (4-seed mean) | 0.805 ±0.006 | 0.638 ±0.027 | 0.166 ±0.005 | 0.523 ±0.002 |
| Deep Ensemble (4-seed mean) | 0.825 ±0.007 | 0.658 ±0.010 | 0.066 ±0.011 | 0.434 ±0.023 |
| SWAG (4-seed mean) | 0.808 ±0.010 | 0.613 ±0.023 | 0.111 ±0.008 | 0.556 ±0.022 |
| Model | Acc | NLL | ECE | AUROC-OOD MI | MI Ratio | Params | Time (s) |
|---|---|---|---|---|---|---|---|
| SWAG | 0.808 ±0.010 | 0.556 ±0.022 | 0.111 ±0.008 | 0.613 ±0.023 | 1.240 ±0.072 | 208.37M | 1593.4 |
| Model | AUROC | NLL | ECE | AUROC-OOD MI | MI Ratio | Params | Time (s) |
|---|---|---|---|---|---|---|---|
| SWAG | 0.922 ±0.001 | 0.359 ±0.002 | 0.190 ±0.001 | 0.634 ±0.047 | 1.601 ±0.214 | 470.42K | 104.9 |
| Model | MAE | RMSE | NLL | ECE | PICP | MPIW | AUROC-OOD | AUPR-OOD | Params |
|---|---|---|---|---|---|---|---|---|---|
| SWAG | 10.542 ±0.057 | 17.313 ±0.044 | 4.323 ±0.003 | 0.216 ±0.001 | 0.973 ±0.001 | 87.01 ±0.44 | 0.585 ±0.054 | 0.783 ±0.026 | 1.18M |
| Model | Hidden Rank(s) | Output Rank | Total Params | Compression |
|---|---|---|---|---|
| Full-Rank BBB | – | – | 4,788,010 | 1.0 |
| Low-Rank Gaussian | 25 | 10 | 245,810 | 19.5 |
| Low-Rank Laplace | 25 | 10 | 245,810 | 19.5 |
| Rank-1 Multiplicative | – | – | 2,406,398 | 2.0 |
| Model | Accuracy | NLL | Brier | ECE | MI |
|---|---|---|---|---|---|
| Full-Rank BBB | |||||
| Low-Rank Gaussian | |||||
| Low-Rank Laplace | |||||
| Rank-1 Multiplicative |
| Model | EM, B=15 | EW, B=15 | EM, B=10 | EW, B=30 |
|---|---|---|---|---|
| Full-Rank BBB | ||||
| Low-Rank Gaussian | ||||
| Low-Rank Laplace | ||||
| Rank-1 Multiplicative |
| Model | Accuracy | NLL | Brier | ECE | MI |
|---|---|---|---|---|---|
| Full-Rank BBB | |||||
| Low-Rank Gaussian | |||||
| Low-Rank Laplace | |||||
| Rank-1 Multiplicative |
| Model | EM, B=15 | EW, B=15 | EM, B=10 | EW, B=30 |
|---|---|---|---|---|
| Low-Rank Gaussian | ||||
| Low-Rank Laplace | ||||
| Full-Rank BBB | ||||
| Rank-1 Multiplicative |
| Model | Single Pass RMSE | MC-200 RMSE |
|---|---|---|
| Full-Rank BNN | 0.49623 | 0.49528 |
| Low-Rank BNN | 0.50732 | 0.50296 |
| Model | In-Domain IQR | Out-of-Domain IQR | Ratio | Parameters |
|---|---|---|---|---|
| Full-Rank BNN | 20,802 | |||
| Low-Rank BNN | 7,202 |
Bayesian Neural Networks
Bayesian neural networks place probability distributions over neural-network parameters or functions to quantify predictive uncertainty and support robust decisions.
See all 101 papers