stat.MLFeb 26, 2026

Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks

Authors: Katerina Papagiannouli, Dario Trevisan, Giuseppe Pio Zitto

Organizations: Dipartimento di Matematica, Università di Pisa

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study wide Bayesian neural networks focusing on the rare but statistically dominant fluctuations that govern posterior concentration, beyond Gaussian-process limits. Large-deviation theory provides explicit variational objectives-rate functions-on predictors, providing an emerging notion of complexity and feature learning directly at the functional level. We show that the posterior output rate function is obtained by a joint optimization over predictors and internal kernels, in contrast with fixed-kernel (NNGP) theory. Numerical experiments demonstrate that the resulting predictions accurately describe finite-width behavior for moderately sized networks, capturing non-Gaussian tails, posterior deformation, and data-dependent kernel selection effects.

Figures & tables

Fig. 1: Prior output large-deviation rate functions. Prior rate Iprior(y)I_{\mathrm{prior}}(y) as a function of the output yy for a wide Gaussian neural network with ReLU (left) and tanh⁡\tanh (right) activation.
Fig. 2: Posterior deformation of the output large-deviation rate function. Prior and posterior rate functions Iprior(y)I_{\mathrm{prior}}(y) and Ipost(y)I_{\mathrm{post}}(y) at a fixed test input xtest=3x_{\mathrm{test}}=3 , for a wide Gaussian neural network (left, ReLU activation; right tanh⁡\tanh activation) trained on a Heaviside target
Fig. 3: Posterior MAP prediction curves. Large-deviation MAP prediction y∗(xtest)y^{\ast}(x_{\mathrm{test}}) as a function of the test input xtestx_{\mathrm{test}} , for a wide Gaussian neural network trained on a Heaviside target. Left: ReLU activation. Right: tanh⁡\tanh activation.
Fig. 4: Prior LDP versus NNGP. Left: prior large-deviation rate function compared with the quadratic rate induced by the NNGP kernel. Right: relative operator-norm gap between the kernel selected by the LDP variational problem and the NNGP kernel.
Fig. 5: Posterior LDP versus NNGP. Left: posterior large-deviation rate function compared with the quadratic posterior rate induced by Gaussian-process regression with the NNGP kernel. Right: relative operator-norm gap between the kernel selected by the posterior LDP variational problem and the NNGP kernel.
Fig. 6: Predictive curves: LDP-MAP versus NNGP. Left: large-deviation MAP prediction compared with the NNGP posterior mean as a function of the test input. Right: kernel-gap diagnostic along the same input grid, quantifying deviations from the fixed NNGP kernel.
Fig. 7: Prior tails: Monte Carlo versus LDP. Left: empirical tail decay rates −1nlog⁡p^n(y)-\tfrac{1}{n}\log\hat{p}_{n}(y) for increasing widths compared with the prior large-deviation rate function. Right: empirical output histograms compared with the heuristic density ∝exp⁡(−n Iprior(y))\propto\exp(-n\,I_{\mathrm{prior}}(y)) .
Fig. 8: Posterior samples: MC vs LDP and NNGP. Finite-width posterior samples at n=128n=128 for a ReLU network trained on a Heaviside target. Left: nn -tempered (LDP) scaling, showing concentration around the LDP-MAP prediction. Right: standard NNGP scaling on the same xx -axis, showing much broader Gaussian fluctuations centered at the NNGP posterior mean.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Diagnostics for prior rate computation (ReLU). Rate function, inner and outer gradient norms, minimum kernel diagonal entry, and kernel gap relative to the NNGP kernel.
Fig. 10: Diagnostics for posterior rate computation (ReLU, Heaviside data). Same layout as Figure 9 , illustrating stable optimization under posterior conditioning.
Fig. 11: Diagnostics for MAP prediction computation (ReLU, Heaviside data). Same layout as Figure 9 , illustrating stable optimization also for the prediction.
Fig. 12: MCMC diagnostics for finite-width posterior sampling. Trace plots for multiple (2) MALA chains targeting the nn -tempered posterior at width n=128n=128 .

Explore similar work

CardsList

  1. Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime

    May 28, 2026Paolo Baglioni, Christian Keup, Vincenzo Zimbardo +4Bayesian Neural NetworksNeural Network Generalization

    1INFN, Sezione di Milano Bicocca, Piazza della Scienza 3, 20126, Milano, Italy · 2INFN, Gruppo Collegato di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy · 3Dipartimento di Scienze Matematiche, Fisiche e Informatiche, Universit`a degli Studi di Parma, Parco Area delle Scienze, 7/A 43124 Parma, Italy +2

  2. The Impact of Likelihood Tempering on the Limiting Predictive Moments of Variational Bayesian Linear Neural Networks

    Oct 6, 2026Ian Zhang, Thibault RandrianarisoaBayesian Neural NetworksVariational Inference

    Department of Statistical Sciences, University of Toronto · Dunlap Institute for Astronomy and Astrophysics

  3. From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference

    Feb 26, 2026Gracielle Antunes de Araújo, Flávio B. GonçalvesBayesian Neural NetworksShallow Neural Networks

    Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.