We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On large-variance random networks, median KL-divergence improvements over common baselines range from tens to thousands of times. On a variational Bayes neural network, we show that our method improves upon a state-of-the-art deterministic inference method by an order of magnitude in KL divergence from Monte Carlo ground truth. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Figures & tables
Assumption
References
Σ is small (linearized)
Titensky et al. (2018) ; Nagel and Huber (2022)
Petersen et al. (2024) ; Jungmann et al. (2025)
Bergna et al. (2025) ; Rui Li and Trapp (2025)
Σ is small (unscented)
Astudillo and Neto (2011) ; Abdelaziz et al. (2015)
Σ is diagonal
Abdelaziz et al. (2015) ; Huber (2020)
Goulet et al. (2021) ; Rui Li and Trapp (2025)
Table 1.1 : Comparison of assumptions imposed on Gaussian approximations of neural network layers with input N(μ,Σ) .
Activation function
References
piecewise linear
Cho and Saul (2009) ; Bibi et al. (2018)
Huber (2020) ; Wright et al. (2024)
Akgül et al. (2025)
logistic ( ≈ piecewise exponential)
Astudillo and Neto (2011) ; Abdelaziz et al. (2015)
logistic ( ≈Φ )
Huber (2020)
Heaviside
Wu et al. (2019) ; Wright et al. (2024)
Table 1.2 : Activation functions for which moment propagation has been approximated. Note two distinct approaches to approximating the logistic function.
Figure 4.1 : Empirical small-covariance convergence for a fixed deep sine network. Methodology is described in Appendix I .
Figure 5.1 : Comparison of goodness of approximation (lower KL divergence is better) for all random neural networks, grouped by approximation method, in the small (left) and large (right) input variance scenarios. Four non-finite results are omitted in each scenario.
Figure 5.2 : KL divergence between ground-truth predictive distribution (by Monte Carlo) and approximations for two UCI regression datasets. W24@ k means Wright et al. (2024) with k terms in the series expansion.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure J.1 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the Servo dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.2 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the liver disorders dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.3 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the Auto MPG dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.4 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the real estate valuation dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.5 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the forest fires dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.6 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the infrared thermography temperature dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.7 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the concrete compressive strength dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.8 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the airfoil self-noise dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.9 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the communities and crime dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.10 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the Parkinsons telemonitoring dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure J.11 : KL divergence between ground truth predictive distribution (by Monte Carlo) and approximations for the combined cycle power plant dataset. W24@ k means Wright et al. [2024] with k terms in the series expansion.
Figure K.1 : Output distribution of a stochastic neural network, pseudo-true Normal distribution (“pseudo”), and layer-by-layer moment-matched Normal distribution (“analytic”).
Figure L.1 : Comparison of goodness of approximation (lower KL divergence is better) for all random neural networks, grouped by approximation method, in the small input variance scenario.
Figure L.2 : Comparison of goodness of approximation (lower KL divergence is better) for all random neural networks, grouped by approximation method, in the medium input variance scenario.
Figure L.3 : Comparison of goodness of approximation (lower KL divergence is better) for all random neural networks, grouped by approximation method, in the large input variance scenario.
Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.
Gaussian neural networks (GaNNs) are proposed as a novel regularization mechanism for neural networks. From a Bayesian perspective standard regularization techniques can be viewed as imposing priors over weight-space. Assuming priors over activation-space remains a largely unexplored possibility. GaNNs assume such priors. They do this by treating activities from earlier layers like signals with Gaussian noise and predicting the properties of the noise distribution using an additional unsupervised loss. While training, the unsupervised loss acts as a penalty on unexpected activities, allowing greater weight updates in less surprising directions. The paper demonstrates the superiority of Gaussian neural networks over standard neural networks on a variety of classification and regression tasks. We also investigate the ability of GaNNs to quantify uncertainty.
Peter Kuhn, Victoria Heusinger-Heß
Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut, EMI, Germany
In this paper, we study the estimation of a marginal regression function from independent units with repeated binary, count, or continuous responses using ReLU deep neural networks. In the model, we assume that the dependence is generated by an unobserved random mean function within each unit. We then fit a neural network with a convex generalized regression loss. We show an oracle inequality by separating conditional measurement variation from between-unit variation. In addition, we prove that with n units and m measurements per unit, ReLU networks can attain an integrated mean squared error of order n−1+(nm)−2β/(2β+d), up to logarithmic factors, over β-Hölder classes. We also derive a weighted oracle inequality for unequal cluster sizes and a rate for compositionally smooth functions. For pointwise ensemble inference, we give a projection central limit theorem and prove infinitesimal jackknife consistency under an explicit asymptotic linearity condition. Simulations and real data examples are provided to support our theoretical findings and practical implications.
It is known that randomly initialized neural networks have Gaussian behaviour in the n→∞ limit. Unfortunately, this approximation can be poor for finite n and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is slightly non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).