A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

Younghun Lee; Jake Zummo; Wei Biao Wu

stat.MLOct 12, 2026

A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

Authors: Younghun Lee, Jake Zummo, Wei Biao Wu

Organizations: Department of Statistics, University of Chicago

Abstract

Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite $p$ th moment for $p > 1$ , yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.

Figures & tables

Figure 1: (a) Stationary behavior: marginal median (lines) and 5–95% quantiles (bands) of

|\theta_{k}-\theta^{\star}|

under nonlinear heavy-tailed SGDM. (b) Kolmogorov distance between the normalized partial-sum maximum and the Brownian maximum. (c) Kolmogorov distance under optimal quadratic tuning. (d) Averaged limiting variance versus forgetting rate under stochastic curvature, with the theoretical lower bound and explicit tuning. The gray region is drawn by interpolating the numerical lower envelope.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Supplementary diagnostics. (a) Normalized fractional-moment coupling distances for two heavy-tailed nonlinear regimes. (b) First-coordinate Kolmogorov distance to the Gaussian reference at matched normalized steps, with both methods stable. (c) Joint-law half-space distance in the quadratic accelerated setting, including unstable effective SGD. (d) Empirical CDF discrepancy for the scaled last iterate in the linear Student-

t_{3}

model.

$(\alpha,p)$	$L_{p}$	$r$	$w_{p}$	$\widehat{D}_{500,p}/D_{0,p}$
$(1.5,1.3)$	1.4636	0.99857	0.1728	$3.61\times 10^{-6}$
$(1.8,1.6)$	1.3953	0.98846	0.4880	$2.65\times 10^{-6}$

Appendix

Table 1: Verification of the sufficient conditions in Theorem 3.1 for

g=(1+R)\theta+\tfrac{1}{4}\sin\theta+\varepsilon

R=0.15\,T_{\alpha}

(\beta,\gamma)=(0.5,0.01)

, and the observed coupling distance at

k=500

$\theta_{0}$	$n=1{,}000$	$n=3{,}000$	$n=10{,}000$	$n=30{,}000$	$n=100{,}000$
$-3$	0.284	0.094	0.023	0.010	0.006
$1$	0.047	0.031	0.016	0.009	0.009
$3$	0.281	0.099	0.030	0.014	0.005

Appendix

Table 2:

d_{K}(\sup_{s\leq 1}|\mathbb{S}_{n}(s)|,\sup_{s\leq 1}|B(s)|)

for averaged SGDM,

R=0.5\,T_{3}

(\beta,\gamma)=(0.5,0.05)

, 20,000 paths per start,

\widehat{\Sigma}^{\rm av}=0.6599

. Simultaneous 95% DKW bound 0.0126.

$\gamma$	$n$	Var	$\widehat{D}_{0}$	$\widehat{D}_{F}$	$\widehat{D}_{\rm mom}$
0.1	93	1.0345	0.0335	0.0370	0.0370
0.05	240	1.0169	0.0269	0.0289	0.0289
0.025	591	1.0084	0.0224	0.0234	0.0234
0.0125	1403	1.0042	0.0179	0.0184	0.0184

Appendix

Table 3: Student-

t_{3}

last iterate

\theta_{n}/\sqrt{\gamma}

g=\theta+\varepsilon

\varepsilon=T_{3}/\sqrt{3}

\beta=0.5

, 50,000 paths. Var is the exact finite-

n

variance. Simultaneous 95% DKW bound 0.0071.

$(\gamma,\beta)$	$1-q$	$V_{L}$	$\widehat{V}_{L}$	$\Sigma^{\rm av}_{L}$	$\widehat{\Sigma}^{\rm av}_{L}$
$(1,0.00)$	0.0100	1.0050	1.0050	1.0050	1.0077 (0.0023)
$(1,0.50)$	0.0204	1.5113	1.5113	1.0076	1.0085 (0.0023)
$(1,0.81)$	0.0952	3.7033	3.7034	1.0185	1.0169 (0.0023)

Appendix

Table 4: Exact stationary and averaged limiting variances in the

L

direction against Monte Carlo (400,000 chains,

n=20000

; standard error in parentheses).

Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) uses noisy, data-sampled gradients to train models and solve large-scale optimization problems.

See all 214 papers

Stochastic Gradient Descent: A Nonlinear Time Series Perspective
Oct 14, 2026Jiaqi Li, Zhipeng Lou, Stefan Richter +1Richardson Extrapolation Stochastic Gradient Descent
Department of Statistics, Rice University, USA · Department of Mathematics, University of California, San Diego, USA · Institute of Applied Mathematics, Heidelberg University, Germany +1
Scaling Limits of Constant-Stepsize SGD at Flat Minima
Jul 17, 2026Jingyi Zhang, Cheng Mao, Debankur MukherjeeStochastic Optimization Convergence Flat Minima
Georgia Institute of Technology
Stochastic Gradient Descent with Momentum is Algorithmically Stable
May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanGeneralization Bounds Algorithmic Stability
Department of Mathematics, The University of Hong Kong, Hong Kong, China · Department of Mathematics and Mathematical Statistics, Umeå University, Sweden

Oct 14, 2026J/K move · Enter open · S save

Stochastic Gradient Descent: A Nonlinear Time Series Perspective

Jiaqi Li, Zhipeng Lou, Stefan Richter, Wei Biao Wu

Department of Statistics, Rice University, USA · Department of Mathematics, University of California, San Diego, USA · Institute of Applied Mathematics, Heidelberg University, Germany+1

Richardson Extrapolation Stochastic Gradient Descent Stochastic Optimization Convergence

Open paper AskPDF arXiv

This paper investigates the statistical behavior of Stochastic Gradient Descent (SGD) with constant step size under the framework of iterated random functions. Compared to previous studies establishing the convergence of SGD in probability measure, e.g., Wasserstein distance, our approach directly provides the convergence in Euclidean distance by showing the Geometric Moment Contraction (GMC) of SGD. This new convergence can address the non-stationarity of SGD due to fixed initial points and can provide a more refined asymptotic analysis of SGD. Specifically, we prove quenched central limit theorem, invariance principle and

α

-stable convergence for averaged SGD (ASGD) regardless of initialization. Furthermore, we provide a novel perspective to understand the impact of step sizes in SGD by studying its derivative with respect to the step size. The existence of stationary solutions for the derivative processes is shown, enabling the use of interpolation to recover the full-trajectory of SGD. Subsequently, we utilize multiple step sizes and show an enhanced Richardson-Romberg extrapolation with improved bias representation, which brings ASGD estimates closer to the global optimum. Finally, we propose a new online inference method and a bias-reduced variant for the extrapolated ASGD. Empirical confidence intervals are constructed and the coverage probabilities approach the nominal level as

n

grows.

Abstract

Stochastic Gradient Descent

Explore similar work

Stochastic Gradient Descent: A Nonlinear Time Series Perspective

Scaling Limits of Constant-Stepsize SGD at Flat Minima

Stochastic Gradient Descent with Momentum is Algorithmically Stable