Gaussian Limits for SGD Without Stationary Moments

Xiaoli Li; Wei Biao Wu

stat.MLOct 4, 2026

Gaussian Limits for SGD Without Stationary Moments

Authors: Xiaoli Li, Wei Biao Wu

Organizations: University of Chicago

Abstract

Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with $o(a)$ error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Figures & tables

Figure 1: The first-order approximation in the Gaussian-refresh design. (a) Estimated TV divided by

a

for baseline (solid) and corrected (dashed) laws; the dotted line is

3\phi(1)/2

. Bars are 1.96 linearized Monte Carlo standard errors. (b) Median and 10th–90th percentiles of the finite-history correction

\mathcal{R}_{a,n}/a

; the dotted line is

J=1.5

, and shading represents variation across histories. (c) Coverage minus 95%, in percentage points (pp), with 1.96 standard-error bars; Corollary 5.4 predicts baseline error

-100aJz_{.975}\phi(z_{.975})+o(a)

pp. Each configuration uses 24,000 histories.

Figure 2: Temporal covariance matters for last-iterate inference. Coverage errors are in percentage points relative to 95%, with 1.96 Monte Carlo standard-error bars from 6,000 replications. (a) Oracle, lag-zero and feasible HAC calibration; HAC uses

m=8192

. (b) HAC bandwidth sensitivity at

a=.0025

, with fixed axis limits

[-5,1]

pp. The gray dash-dotted curve is the population Bartlett Gaussian-limit benchmark; feasible estimates use observed calibration data.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Representative work	Setting or requirement	Result and relation to this paper
Chen et al. (2022) ; Dieuleveut et al. (2020)	Stationary small-step analysis	Law characterization and moment expansions, respectively; our laws may have all moments infinite.
Khusainov et al. (2025)	Iid, bounded designs; finite third score moment	Finite-history convex-distance bound; our Gaussian-error TV expansion permits dependent unbounded designs.
Guo et al. (1997) ; Durmus et al. (2021a)	Temporal energy or drift/growth control	Product-moment stability; our PSD argument gives pathwise contraction.
Roitershtein (2007) ; Damek & Mentemeier (2026)	Markov-modulated or iid affine recursions	Power-law tails; our examples combine all-moment failure with a Gaussian limit.
Mou et al. (2020)	Fixed-step averaged linear SA	CLT and correction for averages; our correction is for the stationary last iterate.
Merad & Gaïffas (2023) ; Wang et al. (2026)	Invariant-law convergence or quantitative SA with moment control	TV to stationarity or Wasserstein Gaussian bounds; our expansion is from stationarity to Gaussian.

Appendix

Table 1: Different objects and assumptions in closely related theories. The surrounding text examines the temporal conditions for the refresh example.

$a$	Base coverage	Corrected coverage	TV	Corrected TV	MCSE (corr.)
0.04	94.3885	95.0766	123.130	17.859	0.469
0.02	94.6849	95.0293	65.035	6.488	0.199
0.01	94.8373	95.0095	33.976	2.050	0.084
0.005	94.9167	95.0027	17.492	0.588	0.035
0.0025	94.9579	95.0009	8.872	0.185	0.013

Appendix

Table 2: Refresh design,

\rho=.7

, 24,000 histories. Coverage is in percent. TV estimates and the linearized Monte Carlo standard error of corrected TV are in units of

10^{-4}

. The empirical conditional Gaussian mixture defines the estimated finite-history marginal law. Full outputs also include coverage standard errors, seed-wise estimates and CDF diagnostics.

$\rho$	$a$	10th percentile	Median	90th percentile	$\widehat{P}(\|\mathcal{R}_{a,n}/a-1.5\|>.5)$
0	0.04	0.975	1.414	2.389	0.3043
0	0.02	1.078	1.442	2.102	0.1746
0	0.01	1.176	1.466	1.941	0.0897
0	0.005	1.254	1.476	1.811	0.0381
0	0.0025	1.318	1.488	1.719	0.0091
0.7	0.04	0.653	1.095	2.207	0.5507

Appendix

Table 3: Finite-history conditional correction

\mathcal{R}_{a,n}/a

in the refresh design, 24,000 histories per row. Quantiles describe between-history variation;

J=1.5

for every persistence.

Model	Parameter	$a$	Coverage	Corrected coverage	Expansion paths
pareto	0	0.04	93.1154	–	3432
pareto	0	0.01	94.0687	–	1829
pareto	0	0.0025	94.5519	–	941
pareto	0.7	0.04	94.6931	–	226
pareto	0.7	0.01	94.9047	–	87
pareto	0.7	0.0025	94.9703	–	36

Appendix

Table 4: Selected Pareto and log-Weibull configurations, 12,000 histories each. The parameter is

\rho

for Pareto and

c

for log-Weibull. Coverage is in percent. An expansion path contains at least one factor with

|1-aX_{i}^{2}|>1

anywhere in its history. Pareto-

3

does not meet the correction theorem’s fourth design moment, so no corrected result is assigned. Every configuration is retained in the supplied results.

$a$	Oracle	Lag zero	HAC $(2048,13)$	HAC $(8192,9)$	HAC $(8192,19)$
0.04	94.92 (0.28)	77.60 (0.54)	93.60 (0.32)	93.55 (0.32)	94.12 (0.30)
0.01	95.35 (0.27)	76.62 (0.55)	93.92 (0.31)	93.82 (0.31)	94.50 (0.29)
0.0025	95.23 (0.28)	76.17 (0.55)	93.60 (0.32)	93.78 (0.31)	94.38 (0.30)

Appendix

Table 5: Dependent-score ellipse coverage in percent, with Monte Carlo standard errors in parentheses, from 6,000 replications. HAC column headings give calibration size and bandwidth. All runs and all bandwidth choices are in the saved summaries.

Figure 3: Qualitative stationary and process predictions. (a,b) Coverage error relative to 95%, in percentage points, on the same fixed axis range: Pareto AR designs and actual Gaussian/Rademacher/Student-

t_{5}

errors, respectively. (c) Bounded two-time characteristic-function errors relative to the OU prediction, with a zero reference. Bars show 1.96 Monte Carlo standard errors. Log-Weibull results appear in Tables 4 and 7 .

Figure 4: Observed calibration and precision before burn-in,

a=.01

, 48,000 trajectories. (a) Actual 95% coverage with 1.96 Monte Carlo standard-error bars; the dotted line is nominal. Both history-corrected curves are measured from simulated outcomes. (b) Median interval widths and 10th–90th percentile bands across the same trajectories, on a logarithmic scale. Estimated-variance intervals use an earlier block of 65 observations.

$a$	$an$	Naive	Known $\sigma^{2}$	Estimated $\sigma^{2}$	Median width	90th width
0.04	0.08	0.12 (0.02)	95.03 (0.10)	95.06 (0.10)	3.811	16.863
0.04	0.28	2.86 (0.08)	94.86 (0.10)	94.97 (0.10)	1.781	4.308
0.04	1	29.89 (0.21)	95.04 (0.10)	95.02 (0.10)	0.884	1.344
0.04	4	93.05 (0.12)	95.08 (0.10)	95.01 (0.10)	0.594	0.669
0.04	24	94.40 (0.10)	95.01 (0.10)	95.08 (0.10)	0.577	0.645
0.01	0.05	0.00 (0.00)	94.69 (0.10)	94.72 (0.10)	2.186	6.242

Appendix

Table 6: Actual finite-history inference, 48,000 trajectories per step size. Coverage is in percent with Monte Carlo standard errors in parentheses. Width columns describe the estimated-variance history-corrected interval.

Figure 5: Persistence amplifies one extreme innovation. The deterministic path and analytic lower bound describe amplification conditional on an imposed shock; the vertical axis is its base-10 logarithm.

$a$	Histories	Baseline coverage	Corrected coverage	MCSE (baseline, corrected)
0.0025	48000	94.9426	95.2455	(0.0026, 0.0026)
0.000625	24000	94.9772	95.0545	(0.0031, 0.0031)

Appendix

Table 7: Log-Weibull stress checks at

c=1

. Coverage is reported as a percentage, with Monte Carlo standard errors in percentage points.

Explore similar work

Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation
Oct 4, 2026Xiaoli Li, Wei Biao Wu
University of Chicago
Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD
Sep 30, 2026Junghoon SeoStochastic Gradient Descent Gaussian Primitives
PIT IN Corp., South Korea
Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds
Feb 15, 2026Zedong Wang, Yuyang Wang, Ijay Narang +3Stochastic Approximation Finite Time Analysis
H. Milton Stewart School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Brown University, Providence, RI 02912, USA · School of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA +2

Oct 4, 2026, stat.MLJ/K move · Enter open · S save

Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation

Xiaoli Li, Wei Biao Wu

University of Chicago

Open paper AskPDF arXiv

Local Gaussian models of constant-step learning predict output variability and expected losses, but weak convergence alone does not justify these moment predictions. We establish moment-accurate Gaussian mixtures by matching stationary energy with local Ornstein--Uhlenbeck limits, ruling out quadratic tail mass invisible to weak convergence. For step size

a

, the second-order Wasserstein error is

o(\sqrt a)

, uniformly over invariant laws, using each law's actual root weights. The assumptions combine confinement, descent, finitely many hyperbolic equilibria and root continuity with finite-variance innovations. The result yields observable covariances, expected objective gaps and first-order mean shifts, while allowing singular covariances, compatible saddles and weights without a limit. For additive noise given by a fixed invertible transform of independent standardized Student

t_3

coordinates, symmetry gives an order-sharp

\sqrt a

smooth-test bound. Numerical transport calculations demonstrate the value of root-specific covariances; controlled SGD studies assess observable predictions across step sizes, batch sizes and model geometries.

Abstract

Explore similar work

Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation

Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD

Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds