Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method
Organizations: Department of Mechanical and Aerospace Engineering, University of California, Davis, Davis, CA 95616
Abstract
We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On large-variance random networks, median KL-divergence improvements over common baselines range from tens to thousands of times. On a variational Bayes neural network, we show that our method improves upon a state-of-the-art deterministic inference method by an order of magnitude in KL divergence from Monte Carlo ground truth. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Figures & tables
| Assumption | References |
|---|---|
| is small (linearized) | Titensky et al. (2018) ; Nagel and Huber (2022) |
| Petersen et al. (2024) ; Jungmann et al. (2025) | |
| Bergna et al. (2025) ; Rui Li and Trapp (2025) | |
| is small (unscented) | Astudillo and Neto (2011) ; Abdelaziz et al. (2015) |
| is diagonal | Abdelaziz et al. (2015) ; Huber (2020) |
| Goulet et al. (2021) ; Rui Li and Trapp (2025) |
| Activation function | References |
|---|---|
| piecewise linear | Cho and Saul (2009) ; Bibi et al. (2018) |
| Huber (2020) ; Wright et al. (2024) | |
| Akgül et al. (2025) | |
| logistic ( piecewise exponential) | Astudillo and Neto (2011) ; Abdelaziz et al. (2015) |
| logistic ( ) | Huber (2020) |
| Heaviside | Wu et al. (2019) ; Wright et al. (2024) |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
ReLU Neural Networks
Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.
See all 136 papers