Minimaxity and Admissibility of Bayesian Neural Networks

Daniel Andrew Coulson; Martin T. Wells

math.STApr 6, 2026

Minimaxity and Admissibility of Bayesian Neural Networks

Authors: Daniel Andrew Coulson, Martin T. Wells

Organizations: Department of Statistics and Data Science, Cornell University

Abstract

Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.

Figures & tables

Figure 1: Estimated risk for several decision rules in dimension $p=5$ as a function of $||\boldsymbol{\theta}||$ . The plotted rules are the MLE, the fixed-scale BNN rule, the Beta-prime minimax shrinkage rule, and the dropout-BNN rule. For the BNN-based rules, the network depth is $d=3$ , the hidden layer widths are $n_{1}=n_{2}=20$ , and the layer scales are $\sigma_{1}=\sigma_{2}=\sigma_{3}=1$ ; for the dropout-BNN rule, the keep probabilities are $q_{1}=q_{2}=0.8$ with inverted dropout.

Figure 2: Estimated risk for several decision rules in dimension $p=50$ as a function of $||\boldsymbol{\theta}||$ . The plotted rules are the MLE, the fixed-scale BNN rule, the Beta-prime minimax shrinkage rule, and the dropout-BNN rule. For the BNN-based rules, the network depth is $d=3$ , the hidden layer widths are $n_{1}=n_{2}=20$ , and the layer scales are $\sigma_{1}=\sigma_{2}=\sigma_{3}=1$ ; for the dropout-BNN rule, the keep probabilities are $q_{1}=q_{2}=0.8$ with inverted dropout.

Figure 3: Estimated risk for several decision rules in dimension $p=100$ as a function of $||\boldsymbol{\theta}||$ . The plotted rules are the MLE, the fixed-scale BNN rule, the Beta-prime minimax shrinkage rule, and the dropout-BNN rule. For the BNN-based rules, the network depth is $d=3$ , the hidden layer widths are $n_{1}=n_{2}=20$ , and the layer scales are $\sigma_{1}=\sigma_{2}=\sigma_{3}=1$ ; for the dropout-BNN rule, the keep probabilities are $q_{1}=q_{2}=0.8$ with inverted dropout.

Figure 4: Estimated risk for several decision rules in dimension $p=5$ as a function of $\|\boldsymbol{\theta}\|$ under several sparsity regimes. The plotted rules are the MLE, the Beta-prime minimax shrinkage rule, and the horseshoe posterior mean. The true sparsity levels considered are $1$ , $2$ , and $5$ .

Figure 5: Estimated risk for several decision rules in dimension $p=50$ as a function of $\|\boldsymbol{\theta}\|$ under several sparsity regimes. The plotted rules are the MLE, the Beta-prime minimax shrinkage rule, and the horseshoe posterior mean. The true sparsity levels considered are $1$ , $2$ , $5$ , $10$ , $25$ , and $50$ .

Figure 6: Estimated risk for several decision rules in dimension $p=100$ as a function of $\|\boldsymbol{\theta}\|$ under several sparsity regimes. The plotted rules are the MLE, the Beta-prime minimax shrinkage rule, and the horseshoe posterior mean. The true sparsity levels considered are $1$ , $2$ , $5$ , $10$ , $20$ , $50$ , and $100$ .

Bayesian Neural Networks

Bayesian neural networks place probability distributions over neural-network parameters or functions to quantify predictive uncertainty and support robust decisions.

See all 86 papers

Neural Bayesian Networks for Modular and Scalable Decision-Making Under Uncertainty
Oct 12, 2026Giovanni Briglia, Marco Lippi, Stefano Mariani +1Bayesian Networks Efficient Neural Network Inference
University of Modena and Reggio Emilia, Reggio Emilia, Italy · University of Florence, Florence, Italy
From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference
Feb 26, 2026Gracielle Antunes de Araújo, Flávio B. GonçalvesBayesian Neural Networks Shallow Neural Networks
Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.
When Has a Bayesian Neural Network Sampled Enough? Adaptive Inference Time with Statistical Guarantees
Oct 8, 2026Fabian Denoodt, Sibylle HessAdaptive Sampling Bayesian Neural Networks
Eindhoven University of Technology

Oct 12, 2026J/K move · Enter open · S save

Neural Bayesian Networks for Modular and Scalable Decision-Making Under Uncertainty

Giovanni Briglia, Marco Lippi, Stefano Mariani, Franco Zambonelli

University of Modena and Reggio Emilia, Reggio Emilia, Italy · University of Florence, Florence, Italy

Bayesian Networks Efficient Neural Network Inference

Open paper AskPDF arXiv

Bayesian Networks (BNs) are established tools for letting agents reason under uncertainty in sequential decision-making. They split a joint distribution into local conditional mechanisms arranged on a directed acyclic graph, which makes them modular, interpretable, and ready to perform probabilistic inference. Their classical formulation, however, has limitations in both expressiveness and scale: each conditional is usually stored as a table or a linear-Gaussian form, and inference evaluates the graph one node and one query at a time. In this paper we introduce Neural Bayesian Networks (NBNs), in which each conditional is a differentiable, tensor-valued mechanism, while the graph and its factorization are unchanged. NBNs retain the modularity and local mechanisms of BNs, can represent any learnable conditional distribution, and run natively on GPUs. We also release NBN, a PyTorch-based library for learning and inference in NBNs, together with a public benchmark. Our evaluation shows gains in both expressiveness and scale: NBNs fit distributions that classical BN libraries struggle to represent, and on large networks and large batches of queries they run two orders of magnitude faster thanks to GPU parallelization. NBN is available at https://github.com/Giovannibriglia/NeuralBayesianNetworks.

Abstract

Bayesian Neural Networks

Explore similar work

Neural Bayesian Networks for Modular and Scalable Decision-Making Under Uncertainty

From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference

When Has a Bayesian Neural Network Sampled Enough? Adaptive Inference Time with Statistical Guarantees