Analytical theories of learning in deep networks often assume gradient flow, discarding the finite step size and batch noise of stochastic gradient descent (SGD). Starting from the discrete update, we derive continuous-time equations for the mean and covariance of the parameter distribution under a Gaussian closure for a general stochastic learning, to first order in the learning rate η. Applying these stochastic learning flow (SLF) equations to SGD, we show that the mean descends an effective loss regularized by the covariance and loss curvature, while the covariance, unlike in stochastic differential equation approximations, becomes unstable at curvature 2/η. In SGD ensembles of linear and nonlinear networks, SLF predicts learning curves and mean trajectories more accurately than gradient flow and captures the covariance dynamics, failing where the parameter distribution becomes strongly non-Gaussian. In the full-batch limit, the edge of stability corresponds to a stationary point of the coupled sharpness-covariance dynamics, recovering self-stabilization and central flow, and batch-to-batch curvature fluctuations lower this threshold under SGD.
Figures & tables
Figure 1: Curvature and parameter covariance determine effective loss landscape. (A) A one-dimensional loss landscape with initial parameter value m(0)=−4 . (B) Loss converges to a local or global minimum under gradient descent (dotted lines: simulations; solid lines: theory) with two different learning rates. (C) Parameters converge to a local or global minimum. (D) Curvature of the loss function; dashed lines are critical values hcrit=η2 for each of the learning rates. (E) Covariance of the parameter ensemble under gradient descent, given initial variance c(0)=0.082 . (F) Effective loss L+21ch along the trajectory of the ensemble mean (dotted), compared with L (gray).
Figure 2: Stochastic Learning Flow captures learning curves and parameter changes in a two-layer deep linear network. (A)-(B) Expected loss during training of an ensemble of networks trained with SGD from a single initial condition, as well as corresponding gradient-flow (GF) and Stochastic Learning Flow (SLF) curves in the low-noise (A) and high-noise (B) regimes. (C) The normalized root mean square error (NRMSE) of the SLF vs. SGD and GF vs. SGD loss curves. (D)-(E) Alignment of the ensemble-averaged SGD parameter update direction with the update direction predicted by either GF or SLF in the low-noise (D) and high-noise (E) regimes. (F) The minimum update alignment values during training for various learning rates and batch sizes. Points in (C) and (F) are mean ± std across n=5 identically initialized networks. Open circles denote (η,B) settings for which p(θ,t) is strongly non-Gaussian, limiting the applicability of the SLF theory ( cf. Figure 3 D).
Figure 3: Curvature and covariance dynamics in a deep linear network. (A) An ensemble of identically initialized two-layer DLNs learn two task modes with SGD (dotted); learning dynamics are characterized by SLF (solid) and GF (dashed), with vertical lines denoting times of mode learning. (B) Eigenvalues of the Hessian during training (colored: output-potent; gray: output-null). (C) Parameter covariance from the SGD ensemble projected onto the Hessian eigenmodes (dotted), together with the SLF covariance (colored: mode-projected; gray: total). (D) Relative overshoot in the SLF estimate of the covariance peak in (C) increases with the non-Gaussian third moment of the parameter distribution as batch size decreases (colors as in panel F). (E) The implicit regularization term from the effective loss function during training. (F) The peak values of the implicit regularization during training. Points in (D,F) are means ± std across 5 identically initialized networks. Open circles denote (η,B) settings for which p(θ,t) is strongly non-Gaussian, limiting the applicability of the SLF theory.
Figure 4: Stochastic Learning Flow in a nonlinear network. (A) In a two-layer network trained on binary MNIST, the integrated SLF equations (black) predict the mean parameter trajectory of an ensemble of n=256 SGD-trained replicas to within estimation noise (gray). The prediction is unchanged if SLF uses the replica covariance rather than the integrated covariance (blue), but degrades if the parameter covariance is neglected in the flow equations (red). (B) Ratio of the parameter error at step k=200 assuming C=0 ( cf . red line in (A)) to the error of the full SLF equations ( cf . black line in (A)) (6 initial conditions at each batch size; bars are medians). (C) Ratio of predicted covariance trace to measured covariance trace as a function of the measured covariance trace, which increases with decreasing batch size (for each B , points correspond to optimizer steps and network initializations). (D) Relative covariance error vs . Hill index of the distribution of losses across replicas, quantifying heaviness of the distribution tail (points are 10 timesteps over 0<k≤50 from a single network for each B ; dashed line: index 2, where the 2nd moment of the distribution diverges; colored lines: median sampling precision of TrCmeas in cases where they can be estimated reliably).
Figure S1: The edge of stability is shifted by batch noise. Mean-square EoS threshold λ0 calculated in a two-layer network from ( 37 ), either with (solid line) or without (dashed line) the second term from the equation. Points show critical curvature based on a one-step probe estimator in the same network. Lower panel shows the relative error between the measured and calculated values. Error bars in the top panel and shading in the lower panel show ±2 std of the probe measurement.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · P.N. Lebedev Physical Institute of the Russian Academy of Sciences · Innopolis University