Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation

Xiaoli Li; Wei Biao Wu

stat.MLOct 4, 2026

Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation

Authors: Xiaoli Li, Wei Biao Wu

Organizations: University of Chicago

Abstract

Local Gaussian models of constant-step learning predict output variability and expected losses, but weak convergence alone does not justify these moment predictions. We establish moment-accurate Gaussian mixtures by matching stationary energy with local Ornstein--Uhlenbeck limits, ruling out quadratic tail mass invisible to weak convergence. For step size $a$ , the second-order Wasserstein error is $o(\sqrt a)$ , uniformly over invariant laws, using each law's actual root weights. The assumptions combine confinement, descent, finitely many hyperbolic equilibria and root continuity with finite-variance innovations. The result yields observable covariances, expected objective gaps and first-order mean shifts, while allowing singular covariances, compatible saddles and weights without a limit. For additive noise given by a fixed invertible transform of independent standardized Student $t_3$ coordinates, symmetry gives an order-sharp $\sqrt a$ smooth-test bound. Numerical transport calculations demonstrate the value of root-specific covariances; controlled SGD studies assess observable predictions across step sizes, batch sizes and model geometries.

Figures & tables

Work	Setting and moment control	Relevant conclusion
Chen et al. (2022)	Single-center stability; additive iid finite-variance noise	Stationary scaling limit; Gaussian identification under uniqueness conditions
Dieuleveut et al. (2020)	Strong convexity; higher regularity and moments	Stationary moment and bias expansions
Wei et al. (2025) , Thms. 3.4–3.5	Strong convexity; $L^{p}$ regularity; joint long-time/small-step limit	CLT at $p=2$ ; convex-set bounds at $p\geq 4$
Wang et al. (2026) , Thm. 3.8	Contractive setting; additive noise with finite third moment	Stationary $W_{1}$ rate; derived scaled third-moment bound
This paper	Finite hyperbolic roots; descent; second innovation moments and root continuity	Law-uniform all-root moments and actual-weight $W_{2}$ approximation

Table 1: Gaussian and moment results for SA in their stated model, moment and time settings.

Figure 1: Root-specific moments and transport. (a) Scaled variance deviations from each root’s OU target; dotted lines show the fixed equal-weight pooled reference’s limiting errors. Bars are one standard error across 16 runs. (b) Second-moment trace deviations in the nongradient model; medians and interquartile ranges retain

t_{3}

dispersion. (c) Deterministic same-label scaled transport with alternating weights, using root-specific covariances or the same fixed pooled target.

Figure 2: Neural predictions across step sizes. Output-variance and actual objective-gap deviations share the

[-10,110]\%

range; MARE starts at zero. All four steps, three batches and 16 paths per setting are included. Dashed lines and bands show the mean and one standard error of 64 matched stationary OU paths, whose loss is quadratic.

Model	Output variance $B=1$	Objective gap $B=1$	Output variance $B=16$	Objective gap $B=16$
Digits-256	$1.012\pm 0.005$	$1.030\pm 0.005$	$0.998\pm 0.003$	$0.997\pm 0.003$
Digits-1024	$1.024\pm 0.003$	$1.027\pm 0.003$	$0.996\pm 0.005$	$0.999\pm 0.003$
WDBC-256	$1.003\pm 0.003$	$1.012\pm 0.005$	$0.997\pm 0.003$	$1.003\pm 0.004$

Table 2: Small-step prediction ratios across models and batches at

a\lambda_{\max}(H)=0.000625

. Entries are means

\pm

one standard error across 16 independent paths; all use physical exposure 8000. Output ratios average the 12 fixed probes within each path.

Figure 3: Predictions beyond parameter covariance. Percentage deviations of (a) nonlinear output variance from

(a/B)Df_{2}(m)S_{1}Df_{2}(m)^{T}

and (b) actual objective gap from

(a/4B)\operatorname{tr}C_{1}(m)

share the range

[-5,5]\%

. (c) Signed mean-coefficient errors, with

T=1600/a

. Bars are one standard error across 16 paths in (a,b) and 32 in (c); the two outputs share paths. Zero denotes the prediction.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A sharp symmetric finite-variance endpoint. Analytic computations for standardized

t_{3}

innovations. (a) The cosine gap divided by

\sqrt{a}

approaches

e^{-1/4}/36

. (b) The symmetric tail modulus divided by

\sqrt{a}

approaches

8/\pi

. Dashed lines are the exact limits. The characteristic function uses 100 terms of a convergent logarithmic series at 80-digit arithmetic precision; the modulus uses ( 67 ). No trajectories or sampling error bars enter these calculations.

Figure 5: Local fluctuations with matched innovation covariance. Variance deviations in (a) are percentages relative to the OU target on a zero-centered axis. (a) Median variance deviation and interquartile range across 16 runs; the continuous OU prediction is zero. (b) Pooled first-coordinate Q–Q curves at

a=0.0025

. The intervals describe between-run dispersion.

Figure 6: Global occupation and transition rates. (a) Per-run left occupation, with paired positions for

a=0.08

and

0.04

; circles start left and crosses start right. The dotted line is the heavy-tail limit ( 75 ). (b) Heavy-tail completed transitions divided by exposure to the last visited well core, compared with one-jump probabilities per iteration. Run lengths are reported in Table 3 .

Noise	$a$	$N$ per run	Left occupation	Raw	Completed
Gaussian	0.08	2,000,000	$0.500^{*}$	0	0
Gaussian	0.04	2,000,000	$0.500^{*}$	0	0
Rademacher	0.08	2,000,000	$0.500^{*}$	0	0
Rademacher	0.04	2,000,000	$0.500^{*}$	0	0
Student t3	0.08	6,600,000	$0.852\;(0.010)$	477	413
Student t3	0.04	51,500,000	$0.845\;(0.011)$	522	400

Appendix

Table 3: Occupation experiment: 12 runs per row.

N

includes burn-in; counts sum post-burn-in events over runs. Heavy-tail occupation is the mean with one across-run standard error in parentheses. ∗ The value

0.500

is fixed by six starts in each well and no observed crossings; it is not a stationary-weight estimate.

Figure 7: Additional local measurements. Coverage in (b) is shown as percentage-point deviation from 95 percent on a zero-centered axis. (a) Bounded characteristic-function discrepancy ( 76 ), shown as median and interquartile range across 16 runs. (b) Nominal

95\%

local fluctuation coverage for the nonlinear observable, shown as mean and one standard error across runs. These finite-horizon measurements describe distributional agreement and coverage; the asymptotic rates are established by the analytic results.

Figure 8: Distributional shape and covariance at

t=1

. Normalized cosine discrepancies (a), skew sine bias divided by

\sqrt{a}

(b), and covariance error (c). Curves are deterministic characteristic-function calculations with analytic truncation error below

10^{-12}

. The nonzero skew limit distinguishes shape error from covariance discretization.

Figure 9: Loss distributions beyond their means. (a) Actual nonlinear losses divided by

a

in the heterogeneous rectangles at

a=0.0025

, with the shared

0.01\chi_{2}^{2}

limit. (b) Actual finite-sum losses multiplied by

B/a

a=0.005

, with the generalized chi-square limit computed by deterministic quadrature. Bands are pointwise one standard error across 16 independent runs. The corresponding moment curves appear in Figures 1 and 3 .

Problem	$B$	Full	Diagonal noise	Isotropic noise	Discrete linear
Finite sum	1	$2.32\pm 0.37$	$52.40\pm 0.37$	$34.70\pm 0.30$	–
Finite sum	4	$1.74\pm 0.23$	$51.58\pm 0.36$	$34.03\pm 0.30$	–
Finite sum	16	$1.34\pm 0.26$	$52.36\pm 0.27$	$34.79\pm 0.23$	–
Neural	1	$6.97\pm 0.47$	$228.29\pm 4.15$	$184.92\pm 3.49$	$6.98\pm 0.47$
Neural	4	$6.31\pm 0.36$	$233.11\pm 3.50$	$189.61\pm 2.99$	$6.31\pm 0.37$
Neural	16	$6.09\pm 0.36$	$227.63\pm 2.49$	$185.48\pm 2.16$	$6.09\pm 0.36$

Appendix

Table 4: Covariance-reference comparison. Output-variance MARE (percent), mean

\pm

one standard error across 16 runs, at

a=0.005

for the two-dimensional finite sum and

a\lambda_{\max}(H)=0.000625

for the neural model. Each problem uses its fixed root Hessian and output Jacobian. The discrete-linear reference is evaluated for the neural model.

Figure 10: Small-step neural predictions. At

a\lambda_{\max}(H)=0.000625

, (a) logit-variance and (c) actual objective-gap deviations use a common percentage scale. (b) Per-run MARE measures absolute relative error across the 12 probes. Bars summarize 16 paths; dashed lines and bands show the mean and one standard error of 64 matched stationary OU paths. The OU loss is quadratic. The main-text Figure 2 reports all four steps.

Figure 11: Finite-second-moment evidence in the nonlinear nongradient system. (a) All-root truncated moments at

a=0.005

, with one standard error across 16 runs and a deterministic Gaussian reference. (b) Median relative covariance error and interquartile range. (c) All individual stable root-centered trace ratios; blue is Gaussian noise and orange is

t_{3}

. Raw quadratic measurements use dispersion summaries because

t_{3}

has no fourth moment.

$a$	$(\overline{f}_{1}-f_{1}(m))/a$	$(\overline{f}_{2}-f_{2}(m))/a$
$0.04$	$-0.05988\pm 0.00455$	$-0.028306\pm 0.000644$
$0.02$	$-0.05769\pm 0.00409$	$-0.027889\pm 0.000539$
$0.01$	$-0.05470\pm 0.00429$	$-0.027249\pm 0.000498$
Theory	$-0.05560$	$-0.027255$

Appendix

Table 5: Signed mean coefficients from 32 independent confirmation paths per step, with one standard error. Both outputs use the same paths.

Figure 12: Independent mean measurements and error sources. (a,b) Signed coefficients for both preselected observables. Gray paths use

R=16,aT=400

; the independent confirmation uses

R=32,aT=1600

, with paired burn-in windows. Dashed lines are root-derived coefficients. (c,d) The three terms in ( 78 ) for the confirmation’s primary window. Bars are one standard error across independent paths; outputs and windows within a path are dependent.

Augmentation	$B=1,a_{s}$	$B=1,a_{l}$	$B=16,a_{s}$	$B=16,a_{l}$
Tilt + dither	$0.986\pm 0.012$	$1.473\pm 0.011$	$0.985\pm 0.007$	$1.009\pm 0.006$
Tilt only	$0.986\pm 0.012$	$1.507\pm 0.015$	$0.986\pm 0.007$	$1.008\pm 0.006$
Dither only	$0.986\pm 0.012$	$1.469\pm 0.011$	$0.985\pm 0.007$	$1.009\pm 0.006$
Neither	$0.986\pm 0.012$	$1.502\pm 0.014$	$0.986\pm 0.007$	$1.008\pm 0.006$

Appendix

Table 6: Neural augmentation ablation. Mean logit-variance ratios (one standard error across 16 trajectories) use each arm’s root-derived reference. The common steps are

a_{s}=0.002603

and

a_{l}=0.020825

Figure 13: Directional effect of dither. Its fraction of theoretical variance in six Hessian eigenvectors (a) and 12 fixed logit directions (b). Both panels use the same 0–100 percent scale; the output range is annotated. These are deterministic contributions to the local Lyapunov covariance.

Figure 14: Augmentation ablation across step and batch. Logit-variance deviations from prediction (top) and MARE in six Hessian directions (bottom). Each row shares its vertical range across batches. Each arm uses its own root-derived reference; bars are one standard error across 16 paths. Comparable small-step output accuracy coexists with substantial directional errors at the largest step and

B=1

Figure 15: Neural finite-step decomposition on paired states. Actual and quadratic objective gaps (a), scaled drift residual and linear drift norm (b), and relative state-dependent covariance variation (c). Bars are one standard error across 16 paths at each step. The covariance norm is Frobenius; drift norms are Euclidean.

Figure 16: Observation-window sensitivity. Per-run logit MARE (a) and mean variance ratio (b) for paired short and full windows of 16 independent neural paths per setting. The stationary OU references use 64 paths at each window length. Bars are one standard error across paths; shared paths make the two neural estimates dependent.

Model	$a\lambda_{\max},B$	Variance ratio	MARE (%)	Loss ratio
Digits 256	$0.000625,1$	$1.012\pm 0.005$	$3.018\pm 0.225$	$1.030\pm 0.005$
Digits 256	$0.000625,16$	$0.998\pm 0.003$	$3.245\pm 0.203$	$0.997\pm 0.003$
Digits 256	$0.005,1$	$1.520\pm 0.006$	$52.034\pm 0.571$	$1.927\pm 0.010$
Digits 256	$0.005,16$	$1.001\pm 0.004$	$3.004\pm 0.207$	$1.012\pm 0.003$
Digits 1024	$0.000625,1$	$1.024\pm 0.003$	$3.689\pm 0.273$	$1.027\pm 0.003$
Digits 1024	$0.000625,16$	$0.996\pm 0.005$	$3.288\pm 0.185$	$0.999\pm 0.003$

Appendix

Table 7: Independent dimension/data study. Each row uses 16 paths with retained physical exposure 8000. Values are mean

\pm

one standard error. MARE is a percentage.

Figure 17: Moment predictions across dimension and data. Per-run logit MARE (top) and actual objective-gap ratio (bottom) for 16 independent paths per setting. Exact stationary OU references use 64 paths per case and step, matching the save and loss schedules and sharing the scaled reference across batches. All bars are one standard error across paths. The OU loss is quadratic.

Figure 18: Long-horizon communication at a smaller step. (a) Mean occupation and one standard error over 12 paths at nested horizons; the dashed line is the heavy-tail limit. (b) Total completed transitions against retained updates scaled by

a^{3}

. (c) Directional rates relative to the finite-step one-jump proxy, with 95% path-bootstrap intervals. Nested windows are dependent.

Explore similar work

Sep 30, 2026cs.LG

Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD

We prove a sharp Gaussian approximation for the invariant law of constant-stepsize SGD with bounded additive noise generated by an exogenous uniformly ergodic Markov chain. For a smooth, strongly convex objective with a Lipschitz Hessian and nondegenerate long-run noise covariance, the centered iterate normalized by the square root of the stepsize is

O(\sqrtα)

-close in 1-Wasserstein distance to its limiting Gaussian. The proof combines blockwise Gaussian comparison with long-run contraction. A four-state example gives a matching lower bound although the one-time noise marginal is symmetric and every nonzero-lag autocovariance vanishes. In this example, an adjacent third-order mixed moment produces the leading correction.

Junghoon Seo

PIT IN Corp., South Korea

Stochastic Gradient Descent Gaussian Primitives Hessian Covariance Markov

Ask PDF

Feb 15, 2026cs.LG

Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds

Constant-stepsize stochastic approximation (SA) is widely used in learning for computational efficiency, yet the distribution of the iterates is typically intractable. Classical asymptotics results give

X_k^{(α)} \approx X^{(α)} \approx x^\star+\sqrtαY

, where

X^{(α)}

is the steady state and

Y

is an appropriate Gaussian limit, by progressively taking the time

k\uparrow\infty

and stepsize

α\downarrow0

. Such limit results, however, do not quantify finite-time, finite-stepsize errors. We develop an explicit pre-limit characterization for SA with i.i.d.\ and Markovian noise. We establish existence and uniqueness of the stationary law, a geometric Wasserstein convergence to stationarity, and almost-sure and

L^3

convergence of the steady state to the root

x^\star

, identifying the scale

\sqrtα

as first-order fluctuation. At this scale, we derive a higher-order quantitative Gaussian approximation with a Wasserstein error, using Stein's method and Poisson equation techniques. We further obtain non-uniform Berry--Esseen-type tail bounds, incorporating both steady-state approximation and finite-time convergence errors. We instantiate the theory for strongly convex smooth SGD, linear SA, and nonlinear contractive SA. Beyond strong convexity, for general convex SGD, we identify a Gibbs limiting law and prove a pre-limit Wasserstein approximation error under stability and Stein-equation hypothesis, which are validated numerically.

Zedong Wang, Yuyang Wang, Ijay Narang +3

H. Milton Stewart School of Industrial & Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Brown University, Providence, RI 02912, USA · School of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA +2

Stochastic Approximation Finite Time Analysis Convergence Stationarity Wasserstein Distance Gaussian Primitives+1

Ask PDF

Oct 4, 2026stat.ML

Gaussian Limits for SGD Without Stationary Moments

Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with

o(a)

error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Xiaoli Li, Wei Biao Wu

University of Chicago

Stochastic Gradient Descent

Ask PDF