Uncertainty propagation in auto-regressive random neural network models

Janice Adams; Daniele Venturi

stat.MLAug 20, 2026

Uncertainty propagation in auto-regressive random neural network models

Authors: Janice Adams, Daniele Venturi

Organizations: Department of Applied Mathematics, UC Santa Cruz, Santa Cruz, CA 95064

Abstract

We develop analytical and particle-based methods for uncertainty propagation in random neural network models, where both the inputs and network parameters are allowed to be random. Building on the piecewise-linear structure of the Leaky ReLU activation function, we derive a local approximation of the neural network output with respect to perturbations in both its inputs and parameters. This approximation is exact for perturbations that preserve the network activation pattern, and it allows us to compute analytical expressions for the probability density function and characteristic function of the network output, together with closed-form approximations for its mean and covariance. We extend this uncertainty propagation framework to autonomous dynamical systems whose one-step evolution map is represented by a random neural network. Repeated application of this map defines an autoregressive model, for which we derive recursive equations to propagate uncertainty in both the state and network parameters over time. These equations explicitly account for the state-parameter cross-covariance that develops under successive iterations of the network. Numerical experiments on the Lorenz-63 system and the Kuramoto-Sivashinsky equation demonstrate accurate uncertainty propagation through the predictability horizon and the applicability of the proposed framework to high-dimensional dynamical systems.

Figures & tables

Figure 1: A diagram of a fully connected feed-forward neural network, also known as a multi-layer perceptron (MLP). The input and output in this diagram are vectors of dimension

M

, and each of the three hidden layers have dimension

N

Component	Lorenz-63	Kuramoto–Sivashinsky
Dimension of system	3	200
Number of hidden layers	3	4
Hidden width	64 neurons per layer	256 neurons per layer
Total parameters	$\approx$ 9,000	$\approx$ 300,000
Batch size	1024	256
Number of training samples	100,000	500,000

Table 1: Neural network training setup for the Lorenz-63 and Kuramoto–Sivashinsky systems. Both models use the MLP architecture described in Section 2 and are trained using the Adam optimizer with the torch.nn.MSELoss loss function. The data are split into 70% training, 20% validation, and 10% testing sets. The regression data modules are based on those provided in lightning-uq-box [ 23 ] .

Initial state
	Lorenz-63	Kuramoto–Sivashinsky
Standard deviation	$10^{-3}$	$10^{-1}$
MLP parameters
Epochs used	30 (3% of total)	30 (6% of total)
Base standard dev.	$\sim 1.28\times 10^{-6}$	$\sim 2.94\times 10^{-5}$
Scaling factor	100	5

Table 2: Standard deviations of the initial-state and network-parameter perturbations used in the Lorenz-63 and Kuramoto–Sivashinsky uncertainty-propagation experiments. The perturbation magnitudes are chosen to produce appreciable uncertainty growth over relatively short time horizons while ensuring a non-negligible contribution from parameter uncertainty relative to initial-state uncertainty. The initial-state and parameter perturbations are assumed to be statistically independent. Under auto-regressive propagation, however, the state and network parameters become statistically dependent after a single propagation step.

Panel of Figure 2 — Figure 2: Uncertainty propagation over time for the Lorenz-63 system. The blue trajectories in both panels are samples from the Monte Carlo baseline, obtained by propagating the initial samples through the full nonlinear network. Panel (a) shows the mean and standard deviation computed from the Monte Carlo ensemble, while panel (b) shows the corresponding statistics obtained using Resampled Moment Propagation. Despite relying on a local approximation of the network, Resampled Moment Propagation closely reproduces the Monte Carlo mean and covariance over the full forecast horizon. The particle method uses $S=1000$ particles and a resampling interval of 20 time steps. At each resampling step, $L=100$ samples are drawn from the local Gaussian associated with each particle, and the resulting sample pool is uniformly subsampled back to $S=1000$ particles.

Panel of Figure 3 — Figure 3: Joint and marginal distributions for the Lorenz-63 system at $t=4$ , corresponding to 400 time steps. The particle clouds correspond to (a) the Monte Carlo baseline and (b) Resampled Moment Propagation. The Resampled Moment Propagation configuration is the same as in Figure 2 , with $S=1000$ particles, $L=100$ local samples drawn from each particle at each resampling step, and a resampling interval of 20 time steps.

Panel of Figure 4 — Figure 4: Uncertainty propagation over time for the Lorenz-63 system. This is the same as Figure 2 , but with $S=100$ particles. The local samples and resampling steps are the same ( $L=100$ , and resampling every $20$ steps).

Figure 5: Uncertainty propagation over time for the Kuramoto–Sivashinsky system. The blue trajectories in both panels are samples from the Monte Carlo baseline, obtained by propagating the initial samples through the full nonlinear network. Panel (a) shows the mean and standard deviation computed from the Monte Carlo ensemble, while panel (b) shows the corresponding statistics obtained using Resampled Moment Propagation. The particle method uses

S=100

particles and a resampling interval of 10 time steps. At each resampling step,

L=100

samples are drawn from the local Gaussian associated with each particle, and the resulting sample pool is uniformly subsampled back to

S=100

particles.

Figure 6: Mean (top) and one standard deviation (bottom) for the Kuramoto–Sivashinsky system over 10 seconds, corresponding to 500 time steps. Columns (a) and (b) show the statistics obtained from the Monte Carlo baseline and Resampled Moment Propagation, respectively, while column (c) shows their absolute difference. The Resampled Moment Propagation method uses $S=100$ particles and a resampling interval of 10 time steps. At each resampling step, $L=100$ samples are drawn from the local Gaussian associated with each particle, and the resulting sample pool is uniformly subsampled back to $S=100$ particles.

Uncertainty Propagation

Uncertainty propagation studies how measurement, model, and data uncertainty flows through computations and affects downstream predictions and decisions.

See all 44 papers

Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process
Jun 4, 2026Arnaud Descours, Arnaud Guillin, Geoffrey Lacour +3Uncertainty Quantification Neural Network Training Dynamics
LSAF · LMBP · MaIAGE +1
A Distribution-to-Distribution Neural Probabilistic Forecasting Framework for Dynamical Systems
Mar 26, 2026Tianlin Yang, Hailiang Du, Louis AslettMixture Density Networks Probabilistic Time Series Forecasting
Department of Mathematical Sciences, Durham University, Durham DH1 3LE, United Kingdom · School of Mathematics, East China University of Science and Technology, Shanghai 200237, China · Data Science Institute, The London School of Economics and Political Science, London WC2A 2AE, United Kingdom
Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics
Aug 17, 2026Conrad Ainslie, Pedram Hassanzadeh, Michael W. Mahoney +1Neural Network Stability Neural Surrogate Modeling
Department of Applied Mathematics, University of California, Santa Cruz, Santa Cruz, 95064, CA · Department of Geophysical Sciences, University of Chicago, Chicago, IL 60637 · Committee on Computational and Applied Mathematics, University of Chicago, Chicago, IL 60637 +3

Jun 4, 2026J/K move · Enter open · S save

Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process

Arnaud Descours, Arnaud Guillin, Geoffrey Lacour, Manon Michel+2

LSAF · LMBP · MaIAGE+1

Uncertainty Quantification Neural Network Training Dynamics

Open paper AskPDF arXiv

Uncertainty quantification in neural networks prediction is a main issue for usual applications. Our approach seeks at reducing computation costs by directly evaluating uncertainty using PDE's information on the asymptotic variance, rather than the deep ensemble method which may be seen as a Monte Carlo estimation of the prediction, requiring the training of multiple networks. We thus study the law of the limiting process describing the random fluctuations around the mean-field limit of wide two-layer neural networks trained by stochastic gradient descent in a weak-noise regime. Building on a recent trajectorial central limit theorem, in which this limit is characterized as the weak solution of a linear stochastic evolution equation, we identify its law explicitly. More precisely, we show that it is a centered Gaussian process in the dual of a weighted Sobolev space, and we derive a closed covariance representation for the finite-dimensional distributions obtained by testing it against smooth functions. This covariance is expressed through the solution of a backward transport equation with a nonlocal source term, whose coefficients are driven by the mean-field trajectory. As a consequence, by testing against the activation function at a fixed input, we obtain an expression for the limiting variance of the corresponding network-output fluctuations. We illustrate this result numerically on a one-dimensional regression example.

Abstract

Uncertainty Propagation

Explore similar work

Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process

A Distribution-to-Distribution Neural Probabilistic Forecasting Framework for Dynamical Systems

Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics