Function-Space Empirical Bayes Regularisation with Student's t Priors

Pengcheng Hao; Ercan Engin Kuruoglu

cs.LGFeb 25, 2026

Function-Space Empirical Bayes Regularisation with Student's t Priors

Authors: Pengcheng Hao, Ercan Engin Kuruoglu

Organizations: Institute of Data and Information, Tsinghua Shenzhen International Graduate School, Shenzhen, China

Abstract

Bayesian deep learning (BDL) has emerged as a principled approach to produce reliable uncertainty estimates by integrating deep neural networks with Bayesian inference, and the selection of informative prior distributions remains a significant challenge. Various function-space variational inference (FSVI) regularisation methods have been presented, assigning meaningful priors over model predictions. However, these methods typically rely on a Gaussian prior, which fails to capture the heavy-tailed statistical characteristics inherent in neural network outputs. By contrast, this work proposes a novel function-space empirical Bayes regularisation framework -- termed ST-FS-EB -- which employs heavy-tailed Student's $t$ priors in both parameter and function spaces. Also, we approximate the posterior distribution through variational inference (VI), inducing an evidence lower bound (ELBO) objective based on Monte Carlo (MC) dropout. Furthermore, the proposed method is evaluated against various VI-based BDL baselines, and the results demonstrate its robust performance in in-distribution prediction, out-of-distribution (OOD) detection and handling distribution shifts.

Figures & tables

Metric	Dataset	ST-FS-EB(our)	FS-EB	GFSVI	MC Dropout	MFVI	MAP
ACC $\uparrow$	MNIST	99.33 ± 0.052	99.10 ± 0.067	99.17 ± 0.071	99.32 ± 0.035	99.15 ± 0.095	99.09 ± 0.062
	FMNIST	92.38 ± 0.194	90.49 ± 0.218	91.97 ± 0.099	92.29 ± 0.189	90.28 ± 0.281	91.87 ± 0.242
	CIFAR-10	86.52 ± 0.395	74.42 ± 0.426	84.94 ± 0.548	86.54 ± 0.364	74.85 ± 0.416	77.31 ± 23.656
	PathMNIST	87.83 ± 1.143	86.80 ± 1.338	88.32 ± 1.087	87.69 ± 0.771	87.34 ± 1.253	89.15 ± 2.522
	OrganAMNIST	90.16 $\pm$ 0.635	88.56 $\pm$ 0.713	87.99 $\pm$ 0.353	89.81 $\pm$ 0.560	88.55 $\pm$ 0.585	86.15 $\pm$ 1.111
ECE $\downarrow$	MNIST	0.012 ± 0.001	0.010 ± 0.002	0.003 ± 0.001	0.012 ± 0.001	0.008 ± 0.001	0.003 ± 0.001

Table 1: In-distribution prediction performance. Best results are in bold and second best are underlined.

In	OOD	ST-FS-EB(our)	FS-EB	GFSVI	MC Dropout	MFVI	MAP
MNIST	FMNIST	99.86 ± 0.057	100.00 ± 0.000	99.99 ± 0.003	98.59 ± 0.351	98.60 ± 0.231	98.56 ± 0.255
	NotMNIST	99.96 ± 0.011	99.99 ± 0.003	99.99 ± 0.006	96.86 ± 0.286	94.06 ± 0.777	95.43 ± 0.670
	MNIST-c	89.38 ± 0.569	90.90 ± 0.665	90.61 ± 0.909	84.60 ± 0.679	82.69 ± 0.943	82.30 ± 0.743
FMNIST	MNIST	99.70 ± 0.170	99.63 ± 0.116	99.70 ± 0.116	81.22 ± 1.717	77.48 ± 2.726	78.40 ± 2.310
FMNIST	NotMNIST	96.23 ± 0.665	96.97 ± 0.289	95.30 ± 0.535	80.03 ± 1.611	70.57 ± 4.324	69.49 ± 1.552
CIFAR-10	SVHN	84.87 ± 2.269	82.53 ± 1.836	88.96 ± 1.524	85.32 ± 1.174	82.56 ± 2.690	82.70 ± 11.716

Table 2: Out-of-distribution detection performance. Best results are in bold and second best are underlined.

Figure 1: Performance under distribution shift induced by image rotations. The top row ((a)–(c)) reports ACC scores, while the bottom row ((d)–(f)) shows NLL. The horizontal axes denote the rotation angle, ranging from $-30^{\circ}$ to $30^{\circ}$ , and the vertical axis shows the corresponding performance scores.

IN	Metric/OOD	2.1	3.0	5.0	10.0	20.0	Gaussian
MNIST	ACC	99.31 $\pm$ 0.031	99.31 $\pm$ 0.072	99.33 $\pm$ 0.052	99.34 $\pm$ 0.052	99.34 $\pm$ 0.058	99.31 $\pm$ 0.044
	NLL	0.031 $\pm$ 0.001	0.030 $\pm$ 0.002	0.028 $\pm$ 0.001	0.028 $\pm$ 0.002	0.027 $\pm$ 0.001	0.031 $\pm$ 0.001
	FMNIST	99.91 $\pm$ 0.042	99.85 $\pm$ 0.035	99.86 $\pm$ 0.057	99.84 $\pm$ 0.040	99.95 $\pm$ 0.016	99.90 $\pm$ 0.036
	NotMNIST	99.96 $\pm$ 0.011	99.95 $\pm$ 0.010	99.96 $\pm$ 0.011	99.96 $\pm$ 0.011	99.99 $\pm$ 0.004	99.95 $\pm$ 0.013
	MNIST-c	89.53 $\pm$ 0.560	89.26 $\pm$ 0.538	89.38 $\pm$ 0.569	89.25 $\pm$ 0.606	90.22 $\pm$ 0.577	89.51 $\pm$ 0.469
FMNIST	ACC	92.38 $\pm$ 0.194	92.45 $\pm$ 0.211	92.40 $\pm$ 0.197	92.36 $\pm$ 0.198	92.35 $\pm$ 0.117	92.25 $\pm$ 0.238

Table 3: Influence of dof values of Student’s

t

priors. Best results are in bold and second best are underlined.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Range
$v_{\theta}$	$\{2.1,3.0,5.0,10.0,20.0\}$
$\sigma_{\theta}$	$\{10^{\ell}\mid\ell=-6,-5,\dots,1\}$
$\tau_{1}$	$\{10^{\ell}\mid\ell=-6,-5,\dots,2\}$
$\tau_{2}$	$\{10^{\ell}\mid\ell=-6,-5,\dots,2\}$

Appendix

Table 4: Hyperparameter Ranges

Figure 2: Performance under distribution shift induced by image rotations. The top row ((a) (b)) reports ACC scores, while the bottom row ((c) (d)) shows NLL. The horizontal axes denote the rotation angle, ranging from $-30^{\circ}$ to $30^{\circ}$ , and the vertical axis shows the corresponding performance scores.

Metric	Context number	MNIST	FMNIST	CIFAR-10	PathMNIST	OrganAMNIST
ACC	32	99.33 ± 0.052	92.38 ± 0.194	86.52 ± 0.395	87.83 ± 1.143	90.16 ± 0.635
	64	99.31 ± 0.035	92.42 ± 0.268	86.57 ± 0.429	87.02 ± 1.346	90.60 ± 0.457
	128	99.33 ± 0.047	92.35 ± 0.218	86.66 ± 0.245	87.78 ± 1.353	90.65 ± 0.386
NLL	32	0.028 ± 0.001	0.226 ± 0.005	0.398 ± 0.013	0.454 ± 0.069	0.317 ± 0.012
	64	0.029 ± 0.002	0.229 ± 0.007	0.398 ± 0.012	0.448 ± 0.066	0.297 ± 0.017
	128	0.030 ± 0.002	0.230 ± 0.006	0.399 ± 0.007	0.437 ± 0.052	0.291 ± 0.013

Appendix

Table 5: Influence of the number of context points. Best results are in bold. OOD detection evaluation: for MNIST, OOD1 = FashionMNIST and OOD2 = NotMNIST; for FashionMNIST, OOD1 = MNIST and OOD2 = NotMNIST; for CIFAR-10, OOD1 = SVHN and OOD2 = CIFAR-10C0; for PathMNIST, OOD1=BloodMNIST; for OrganAMNIST, OOD1=OrganSMNIST.

Dataset	Setting	ACC	ECE	NLL	OOD1	OOD2
MNIST	TRAIN	99.37 ± 0.048	0.015 ± 0.001	0.032 ± 0.001	99.22 ± 0.156	97.99 ± 0.460
	$\mathbf{x}_{c}$	99.33 ± 0.052	0.012 ± 0.001	0.028 ± 0.001	99.86 ± 0.057	99.96 ± 0.011
FMNIST	TRAIN	92.43 ± 0.237	0.039 ± 0.004	0.226 ± 0.005	85.56 ± 2.097	81.62 ± 2.170
	$\mathbf{x}_{c}$	92.38 ± 0.194	0.038 ± 0.003	0.226 ± 0.005	99.70 ± 0.170	96.23 ± 0.665
CIFAR-10	TRAIN	86.53 ± 0.445	0.027 ± 0.006	0.400 ± 0.013	84.97 ± 2.594	71.95 ± 1.478
	$\mathbf{x}_{c}$	86.52 ± 0.395	0.021 ± 0.007	0.398 ± 0.013	84.87 ± 2.269	71.35 ± 1.225

Appendix

Table 6: Influence of the context distributions. Best results are in bold. OOD detection evaluation: for MNIST, OOD1 = FashionMNIST and OOD2 = NotMNIST; for FashionMNIST, OOD1 = MNIST and OOD2 = NotMNIST; for CIFAR-10, OOD1 = SVHN and OOD2 = CIFAR-10C0; for PathMNIST, OOD1=BloodMNIST; for OrganAMNIST, OOD1=OrganSMNIST.

Metric	Dataset	ST-FS-EB(our)	FS-EB	GFSVI	MC Dropout	MFVI	MAP
Time (s/epoch) $\downarrow$	MNIST	18.52	26.96	40.44	11.24	18.13	17.86
	FMNIST	18.10	26.63	40.76	11.17	17.87	17.62
	CIFAR-10	43.89	58.69	93.01	32.09	44.39	18.45
	PathMNIST	74.34	97.36	136.86	52.39	71.82	45.63
	OrganAMNIST	20.42	25.33	35.62	16.04	22.02	15.24
Memory (MB) $\downarrow$	MNIST	536.14	808.39	1055.66	437.84	633.32	67.09

Appendix

Table 7: Training time and memory.

Explore similar work

Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance
Jul 28, 2026Pei-Hsuan Hsia, Lars H. Heyen, Arvid Weyrauch +4Bayesian Neural Networks Bayesian Inference
Scientific Computing Center, Karlsruhe Institute of Technology · Scientific Computing Center, Karlsruhe Institute of Technology; and Helmholtz AI
Flow-Transformed Implicit Processes for Function-Space Variational Inference
Jun 1, 2026Luis A. Ortega, Andrés R. Masegosa, Thomas D. NielsenBayesian Inference Variational Inference
Aalborg University
Variational Inference for Evidential Deep Learning
May 26, 2026Jiawei Tang, Xinyan Du, Hui Liu +2Evidential Deep Learning Uncertainty Quantification
School of Computer Science and Engineering Southeast University Nanjing 210096, China · School of Computing Information Sciences Saint Francis University2026 Hong Kong SAR, China · CityDepartmentUniversityof Computerof Hong KongScience Hong Kong SAR, China

Jul 28, 2026J/K move · Enter open · S save

Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance

Pei-Hsuan Hsia, Lars H. Heyen, Arvid Weyrauch, Markus Goetz+3

Scientific Computing Center, Karlsruhe Institute of Technology · Scientific Computing Center, Karlsruhe Institute of Technology; and Helmholtz AI

Bayesian Neural Networks Bayesian Inference

Open paper AskPDF arXiv

In Bayesian neural networks (BNNs), variational inference is a widely adopted framework for modeling uncertainty in a distributional way, with the evidence lower bound (ELBO) serving as the standard objective function. Several distributions contribute to the ELBO loss, such as the prior, approximated posterior, and likelihood distribution. Typically, these distributions are all approximated by a Gaussian distribution, since it is easy to compute, allows for reparameterized gradients, and provides a closed-form loss for training. However, several works have highlighted that this assumption may not generally hold, posing the risk of model misspecification. Alternative distributions have been proposed for the prior specifically, while the effect of distribution choice on the likelihood distribution remains unexplored. In this work, our aim is to close this gap by investigating whether alternative assumptions for the likelihood distribution can outperform the commonly used Gaussian. We compare several likelihood distribution assumptions, such as skewed or heavy-tailed, across regression tasks on both artificial and real-world datasets using standard multilayer perceptrons (MLPs). Our findings demonstrate that Student's t yields better predictive performance than a Gaussian likelihood distribution, independent of the data distribution and MLP architecture (depth and width). In some cases, Student's t can also lead to shorter training times, while still being easy to implement.

Abstract

Explore similar work

Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance

Flow-Transformed Implicit Processes for Function-Space Variational Inference

Variational Inference for Evidential Deep Learning