Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

Arnaud Descours; Geoffrey Lacour

cs.LGOct 7, 2026

Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

Authors: Arnaud Descours, Geoffrey Lacour

Organizations: UCBL · MaIAGE

Abstract

Mean field limits describe the training dynamics of wide neural networks through the evolution of the empirical distribution of their parameters. Although functional central limit theorems characterize the asymptotic fluctuations of this distribution, quantities of practical interest are typically nonlinear observables of the parameter distribution rather than the distribution itself. In this work, we show how these mean field fluctuations propagate to finite dimensional nonlinear observables for shallow neural networks trained by stochastic gradient descent. Working in the weighted Sobolev space in which the limiting fluctuation process is constructed, we apply a functional Delta method under ordinary Fr{é}chet differentiability, without requiring Lions derivatives with respect to the measure variable. We obtain a central limit theorem for the observables and, under a suitable representation of their differentials, an explicit covariance formula inherited from the underlying mean field fluctuation theory. We also study whether prescribed quantities of interest can be recovered from the selected observations. Under a constant rank assumption, we prove that a quantity of interest factors locally through the observation functional if and only if, throughout a neighborhood, the kernel of the differential of the observation is contained in that of the quantity of interest. Thus, a differential condition expressed directly in the ambient Sobolev space yields an exact nonlinear local factorization. These results provide a framework both for quantifying finite-width uncertainty on observable, statistically or physically meaningful quantities and for assessing whether the chosen observations contain the information required to identify them.

Explore similar work

Oct 28, 2022cs.LG

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

To understand the training dynamics of neural networks, prior studies have considered the mean-field limit of two-layer neural networks as the width tends to infinity, establishing theoretical guarantees for its convergence under gradient flow training as well as approximation and generalization capabilities. In this work, we study the infinite-width limit of a type of three-layer neural network where the first-layer weights are randomly sampled and untrained. To rigorously define the limiting model, we extend the mean-field theory by lifting the representation of neurons from Euclidean to functional spaces. This allows us to establish the mean-field training dynamics as a functional gradient flow with a time-varying kernel that remains positive-definite under suitable assumptions, thus proving a linear-rate convergence of its training loss. Furthermore, we define novel function spaces that contain the solutions obtained through the mean-field training dynamics and prove Rademacher complexity bounds for these spaces. Notably, our analysis applies to a range of scaling choices of the model, resulting in two distinct regimes of the mean-field limit that both exhibit feature learning through training.

Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna

Google Research Mountain View, CA 94043 · Courant Institute, New York University New York, NY 10003

Two-Layer Neural Networks Mean-Field Limit Wasserstein Gradient Flows Infinities

Ask PDF

Jun 4, 2026cs.NE

Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process

Uncertainty quantification in neural networks prediction is a main issue for usual applications. Our approach seeks at reducing computation costs by directly evaluating uncertainty using PDE's information on the asymptotic variance, rather than the deep ensemble method which may be seen as a Monte Carlo estimation of the prediction, requiring the training of multiple networks. We thus study the law of the limiting process describing the random fluctuations around the mean-field limit of wide two-layer neural networks trained by stochastic gradient descent in a weak-noise regime. Building on a recent trajectorial central limit theorem, in which this limit is characterized as the weak solution of a linear stochastic evolution equation, we identify its law explicitly. More precisely, we show that it is a centered Gaussian process in the dual of a weighted Sobolev space, and we derive a closed covariance representation for the finite-dimensional distributions obtained by testing it against smooth functions. This covariance is expressed through the solution of a backward transport equation with a nonlocal source term, whose coefficients are driven by the mean-field trajectory. As a consequence, by testing against the activation function at a fixed input, we obtain an expression for the limiting variance of the corresponding network-output fluctuations. We illustrate this result numerically on a one-dimensional regression example.

Arnaud Descours, Arnaud Guillin, Geoffrey Lacour +3

LSAF · LMBP · MaIAGE +1

Two-Layer Neural Networks Fluctuation-Dissipation Theorem Neural Network Mean-Field Limit Covariance Stochastic Gradient Descent

Ask PDF

May 21, 2026stat.ML

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network

f_{\hatρ_t^m}

to its infinite-width counterpart

f_{ρ_t^{MF}}

, which evolves in the mean-field dynamics. While constant-time horizon bounds for

\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\|

may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by

L_t

the mean-field excess MSE loss at time

t

and

m

the number of neurons, under standard regularity assumptions and the condition

\int_0^\infty L_t^{1/2} dt =O(\log d)

, we obtain the uniform in time bound

\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}

whenever

L_t \lesssim t^{-c}

. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than

t^{-2}

, we can attain a loss of

ε

with only

\text{poly}(d/ε)

neurons, training samples, and GD steps.

Margalit Glasgow, Joan Bruna

Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University

Two-Layer Neural Networks Wasserstein Gradient Flows Convergence Chaos Logarithmic Sobolev Inequalities

Ask PDF