From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference

Gracielle Antunes de Araújo; Flávio B. Gonçalves

stat.MLFeb 26, 2026

From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference

Authors: Gracielle Antunes de Araújo, Flávio B. Gonçalves

Organizations: Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.

Abstract

In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.

Figures & tables

Panel of Figure 1 — Figure 1: Shallow feedforward network (one-hidden-layer MLP): $I$ inputs, $H$ hidden units, and $K$ outputs. The figure illustrates the regression case, where $g(\cdot)\equiv\mathrm{Id}(\cdot)$ .

Panel of Figure 2 — Figure 2: The column $\mathbf{x}^{(p)}\in\mathbb{R}^{I}$ is propagated through the network, producing the output vector $\mathbf{f}(\mathbf{x}^{(p)})\in\mathbb{R}^{K}$ . Repeating this for $p=1,\dots,n$ and stacking the outputs as columns yields $\mathbf{F}=[\mathbf{f}(\mathbf{x}^{(1)})\ \cdots\ \mathbf{f}(\mathbf{x}^{(n)})]\in\mathbb{R}^{K\times n}$ .

Panel of Figure 3 — Figure 3: Limit of a wide one-hidden-layer network to a GP for output $k$ . As $H\to\infty$ , the vector $\bigl(f_{k}^{(1)},\dots,f_{k}^{(n)}\bigr)^{\top}$ converges in distribution to a multivariate normal with covariance $K(\mathbf{X},\mathbf{X})$ , and the collection $\{f_{k}(\mathbf{x})\}_{\mathbf{x}}$ defines a GP $\mathcal{GP}(0,K)$ .

Compared pairs	Correlation
tanh $\times$ sigmoid	0.9999
ReLU $\times$ LeakyReLU ( $\alpha=0.1$ )	0.9983
ReLU $\times$ LeakyReLU ( $\alpha=0.3$ )	0.9919

Table 1: Pearson correlation between the shapes of the kernels (curves as functions of

\rho

with

\sigma_{z}=\sigma_{z^{\prime}}=1

), after centering and normalization.

Scenario	Design of $X$	$I$	$n$	$\boldsymbol{\overline{K_{\mathrm{mix}}(x,x)}}$	Nugget $(\sigma_{\varepsilon})$	$\boldsymbol{\sigma_{\varepsilon}^{2}}$	Time (s)
C1	Uniform	20	10000	2.132745	0.292078	0.085310	63.72
C2	Uniform	80	10000	3.769967	0.388328	0.150799	34.90
C3	Uniform	20	20000	2.131981	0.292026	0.085279	68.46
C4	Uniform	80	20000	3.769044	0.388281	0.150762	74.07
C5	Uniform	20	50000	2.131707	0.292007	0.085268	182.54
C6	Uniform	80	50000	3.768044	0.388229	0.150722	215.66

Table 2: Simulation scenarios using a Vecchia -type construction with a nugget automatically calibrated from

\overline{K_{\mathrm{mix}}(x,x)}

Scenario	Anchors	$\hat{\sigma}_{b}^{2}$	$\hat{\sigma}_{v}^{2}$	$\hat{\sigma}_{u}^{2}$	$\hat{\sigma}_{a}^{2}$	$\hat{\alpha}$	$\hat{w}$	$\hat{\sigma}_{\epsilon}^{2}$
C1	first	1.025371	0.980080	0.981301	0.994972	0.508588	0.495693	$8.56\times 10^{-2}$
C1	$k$ -means	1.026216	0.987962	0.995776	0.987359	0.508575	0.496666	$8.29\times 10^{-2}$
C2	first	1.015887	1.031211	1.030972	0.968986	0.487482	0.512434	$1.64\times 10^{-1}$
C2	$k$ -means	1.028323	1.032194	1.032033	0.968549	0.487544	0.511323	$1.63\times 10^{-1}$
C3	first	1.028055	0.971542	0.972107	0.983965	0.511472	0.491441	$8.64\times 10^{-2}$
C3	$k$ -means	1.023113	0.974400	0.976894	0.996529	0.510373	0.490998	$8.18\times 10^{-2}$

Table 3: MAP estimates under two anchor-selection strategies in Nyström.

Scenario	Anchors	Total time (s)	MAE	MSE	RMSE	MESE	SDESE
C1	first	1.334	0.265983	0.112261	0.335053	0.236112	0.165608
C1	$k$ -means	2.155	0.263822	0.110102	0.331816	0.233065	0.161552
C2	first	1.284	0.573294	0.504148	0.710034	0.892681	0.665183
C2	$k$ -means	2.514	0.534228	0.440660	0.663822	0.829185	0.585415
C3	first	1.881	0.265764	0.110175	0.331926	0.233302	0.152864
C3	$k$ -means	2.137	0.264066	0.109563	0.331004	0.229287	0.153301

Table 4: Total time and test-set metrics under two Nyström anchor-selection strategies.

Quantidade	Média (dp)	[min, max]
MAE	0.273514 (0.004816)	[0.263822, 0.282890]
RMSE	0.342126 (0.005893)	[0.331816, 0.353658]
MESE	0.239397 (0.004587)	[0.231336, 0.246951]
SDESE	0.167666 (0.005495)	[0.161552, 0.179720]
total_s	2.486 (0.616)	[1.829, 4.463]
$\sigma_{\varepsilon}^{2}$	0.08341 (0.00142)	[0.08033, 0.08700]

Table 5: Summary of the 20 replications of scenario C1 (

r=500

k

-means anchors,

n_{\text{train}}=9000

Case	$n$	$I$	$\hat{\sigma}_{b}^{2}$	$\hat{\sigma}_{v}^{2}$	$\hat{\sigma}_{u}^{2}$	$\hat{\sigma}_{a}^{2}$	$\hat{\alpha}$	$\hat{w}$	$\hat{\sigma}_{\varepsilon}^{2}$
S1_super_full_all	21263	79	0.983783	1.031285	1.031180	0.969412	0.487748	0.512268	5.1759e-01
R1_year_full_10k	10000	90	1.028279	1.032023	1.031995	0.968947	0.487489	0.512480	5.4418e-02
R2_year_full_20k	20000	90	1.021539	1.031795	1.031790	0.969213	0.487546	0.512372	5.3020e-02
R3_year_full_50k	50000	90	1.027704	1.032048	1.032012	0.968841	0.487509	0.512499	5.3630e-02

Table 6: MAP estimates of the model parameters on the real-world datasets under

k

-means anchor selection.

Case	Dataset	$n$	$I$	Total time (s)	MAE	RMSE	MESE	SDESE
S1_super_full_all	superconduct	21263	79	1.521	11.996	18.464	347.035	808.186
R1_year_full_10k	yearpredictionmsd	10000	90	2.141	8.151	11.346	129.177	276.568
R2_year_full_20k	yearpredictionmsd	20000	90	1.593	8.483	11.578	134.426	272.604
R3_year_full_50k	yearpredictionmsd	50000	90	2.266	8.551	11.614	135.163	292.555

Table 7: Test-set performance metrics reported on the original target scale and total time on the real datasets.

Case	$\mathrm{sd}(y_{\text{train}})$	MAE z	RMSE z	$\hat{\sigma}_{\varepsilon}^{2}(z)$	$\hat{\sigma}_{\varepsilon}^{2}(\text{orig})$
S1_super_full_all	34.231	0.350	0.539	4.417e-04	5.1759e-01
R1_year_full_10k	11.104	0.734	1.022	4.414e-04	5.4418e-02
R2_year_full_20k	10.957	0.774	1.057	4.416e-04	5.3020e-02
R3_year_full_50k	11.022	0.776	1.054	4.415e-04	5.3630e-02

Table 8: Scaled metrics normalized by the training-target standardization, relative to Table 7 .

$r$	$\hat{\sigma}_{b}^{2}$	$\hat{\sigma}_{v}^{2}$	$\hat{\sigma}_{u}^{2}$	$\hat{\sigma}_{a}^{2}$	$\hat{\alpha}$	$\hat{w}$	$\hat{\sigma}^{2}_{\varepsilon}$
1000	0.999745	1.031698	1.031808	0.968802	0.487477	0.512471	$\,5.280451\times 10^{-2}\,$
1500	1.001928	1.030765	1.031028	0.969650	0.487674	0.512086	$\,5.281479\times 10^{-2}\,$
2000	1.000840	1.031140	1.031370	0.969235	0.487561	0.512327	$\,5.281858\times 10^{-2}\,$
2500	1.000225	1.031369	1.031605	0.968976	0.487469	0.512463	$\,5.281404\times 10^{-2}\,$
3000	0.999901	1.031510	1.031739	0.968839	0.487441	0.512496	$\,5.282333\times 10^{-2}\,$
3500	0.999993	1.031638	1.031865	0.968724	0.487422	0.512520	$\,5.282105\times 10^{-2}\,$

Table 9: Full YearPredictionMSD: MAP estimates of the model parameters under Nyström, using first anchors, for different ranks

r

$r$	Total time (s)	MAE	RMSE	MESE	SDESE
1000	5.284	7.446	10.262	105.416	231.199
1500	10.218	7.179	10.112	102.348	247.560
2000	16.643	7.087	9.947	98.988	248.515
2500	28.760	7.032	10.057	101.158	276.705
3000	41.467	6.986	9.690	93.922	231.058
3500	61.748	6.843	9.648	93.231	236.175

Table 10: Full YearPredictionMSD: test-set performance on the original target scale and total time under Nyström with first anchors and different ranks

r

$r$	$\mathrm{sd}(y_{\text{train}})$	MAE z	RMSE z	$\hat{\sigma}_{\varepsilon}^{2}(z)$	$\hat{\sigma}_{\varepsilon}^{2}(\text{orig})$
1000	10.936	0.681	0.938	$\,4.414865\times 10^{-4}\,$	$\,5.280451\times 10^{-2}\,$
1500	10.936	0.656	0.925	$\,4.415726\times 10^{-4}\,$	$\,5.281479\times 10^{-2}\,$
2000	10.936	0.648	0.910	$\,4.416462\times 10^{-4}\,$	$\,5.281858\times 10^{-2}\,$
2500	10.936	0.643	0.920	$\,4.415373\times 10^{-4}\,$	$\,5.281404\times 10^{-2}\,$
3000	10.936	0.639	0.886	$\,4.416178\times 10^{-4}\,$	$\,5.282333\times 10^{-2}\,$
3500	10.936	0.626	0.882	$\,4.416175\times 10^{-4}\,$	$\,5.282105\times 10^{-2}\,$

Table 11: Full YearPredictionMSD: metrics normalized by the training-target scale from Table 10 .

Roof $T$	$r_{\max}(T)$	Total time (s)	MAE	RMSE	$r_{\text{best}}(T)$	RMSE ${}_{\text{best}}$
30s	2500	28.760	7.032	10.057	2000	9.947
60s	3000	41.467	6.986	9.690	3000	9.690
300s	5000	179.735	7.198	9.677	4000	9.594

Table 12: Choice of

r

on the full YearPredictionMSD dataset under total runtime budgets

T

. We report

r_{\max}(T)

(the largest

r

such that

\mathrm{time}(r)\leq T

) and, additionally, the

r

that yields the lowest RMSE within the same budget.

Explore similar work

Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks
Feb 26, 2026Katerina Papagiannouli, Dario Trevisan, Giuseppe Pio ZittoBayesian Neural Networks Bayesian Inference
Dipartimento di Matematica, Università di Pisa
The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours
Apr 8, 2026Robert Allison, Tomasz Maciazek, Anthony StephensonGaussian Process Regression Nonparametric Regression
School of Mathematics University of Bristol Bristol, BS8 1UG, UK
SIKA-GP: Accelerating Gaussian Process Inference with Sparse Inducing Kernel Approximations for Bayesian Deep Learning
May 26, 2026Wenyuan Zhao, Rui Tuo, Chao TianBayesian Neural Networks Gaussian Processes
Department of Electrical and Computer Engineering, Texas A&M University, College Station, US · Department of Industrial and Systems Engineering, Texas A&M University, College Station, US

Feb 26, 2026J/K move · Enter open · S save

Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks

Katerina Papagiannouli, Dario Trevisan, Giuseppe Pio Zitto

Dipartimento di Matematica, Università di Pisa

Bayesian Neural Networks Bayesian Inference Representation Learning

Open paper AskPDF arXiv

We study wide Bayesian neural networks focusing on the rare but statistically dominant fluctuations that govern posterior concentration, beyond Gaussian-process limits. Large-deviation theory provides explicit variational objectives-rate functions-on predictors, providing an emerging notion of complexity and feature learning directly at the functional level. We show that the posterior output rate function is obtained by a joint optimization over predictors and internal kernels, in contrast with fixed-kernel (NNGP) theory. Numerical experiments demonstrate that the resulting predictions accurately describe finite-width behavior for moderately sized networks, capturing non-Gaussian tails, posterior deformation, and data-dependent kernel selection effects.

Abstract

Explore similar work

Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks

The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours

SIKA-GP: Accelerating Gaussian Process Inference with Sparse Inducing Kernel Approximations for Bayesian Deep Learning