Geometric Moment Contraction for Stochastic Nesterov Acceleration

Wei Biao Wu

stat.MLSep 25, 2026

Geometric Moment Contraction for Stochastic Nesterov Acceleration

Authors: Wei Biao Wu

Abstract

We study geometric moment contraction (GMC) of the constant-parameter stochastic Nesterov recursion

Y_k=Θ_k+β(Θ_k-Θ_{k-1}),\qquad Θ_{k+1}=Y_k-γG(Y_k,X_{k+1}).

Under mean strong monotonicity and stochastic $L^p$ Lipschitz continuity, an explicit Perron comparison proves synchronous $L^p$ contraction when $βγL_p<(1-β)(1-q_{γ,p})$ . This direct criterion includes infinite-variance gradients for $1<p<2$ , but its small-step regime requires $β<μ/(μ+L_p)$ . A complementary power-Lyapunov argument establishes a positive, generally much smaller, step-size interval for every fixed $β<1$ and every $p>1$ , using only a finite $p$ th gradient moment. At $p=2$ , a simpler explicit certificate gives

0<γ<\frac{2μ(1-β)^2}{L_2^2(1-β+2β^2)}.

Its quadratic high-momentum scaling is a limitation of the chosen metric, not a sharp stability boundary. We quantify this loss, provide a general mean-only quadratic $S$ -procedure, and exploit endpoint Lyapunov inequalities under stronger samplewise sector information. Verified endpoint certificates can be orders of magnitude less conservative than the explicit metric.

Figures & tables

Reference	Algorithm/regime	Main object	Difference from the present paper
Chen and Wu (2016)	Nonlinear AR( $\infty$ ); finite-lag special cases	Stochastic-Lipschitz stability, functional dependence, CLT and scalar strong approximation	Direct predecessor; the AR(2) specialization gives $q(1+2\beta)<1$ . Signed Nesterov metrics yield additional regimes.
Can et al. (2019)	HB, NAG, APG; constant	Wasserstein convergence to an invariant law	Structured oracle and finite-variance control; no direct nonlinear synchronous $L^{p}$ GMC or physical-dependence limit theory.
Assran and Rabbat (2020)	NAG; constant	Optimization error and steady-state neighborhood	Smooth strongly convex objectives with bounded variance; does not identify a causal GMC process or arbitrary-initialization CLT.
Gitman et al. (2019)	Unified momentum; local quadratic	Stability and stationary covariance	Local linearization rather than global nonlinear contraction.
Gadat et al. (2018) ; Barakat et al. (2021)	Heavy ball/general momentum; decreasing gains	Convergence, fluctuations, trap avoidance	Vanishing-step asymptotics rather than a fixed-parameter invariant process.
Li et al. (2023) ; Tang et al. (2023)	SGD, momentum, NAG	CLTs and averaged-iterate asymptotics	Primarily decreasing gains/local asymptotics; no GMC construction for the constant NAG state.

Table 1: Representative comparison with the closest literature. “Constant” refers to a fixed learning rate and momentum parameter.

Figure 1: Deterministic curvature,

\mu=L=h=1

. The exact quadratic curve is also the supremum obtainable by a freely chosen single-mode quadratic norm. The much smaller slow-coordinate curve is not an intrinsic stability limit.

$L$	$\beta$	$\Gamma_{\rm sc}$	$\Gamma_{\rm NAG}$	Ratio
1.5	0.50	0.566242	0.222222	2.5
1.5	0.80	0.404160	0.024024	16.8
1.5	0.90	0.334987	0.005168	64.8
1.5	0.95	0.289680	0.001198	241.8
1.5	0.99	0.239403	0.000045	5306.3
3	0.50	0.119297	0.055556	2.1

Table 2: Exact scalar random-curvature boundaries versus the explicit slow-coordinate sufficient bound,

\mu=1

. Ratios compare model-specific stability with a universal sufficient certificate.

Figure 2: Mean-only comparison for

\mu=1,L=1.5

. The scalar exact boundary tends to

2/9

\beta\uparrow 1

, whereas the explicit slow-coordinate bound tends to zero.

Figure 3: Mean-only comparison for

\mu=1,L=3

. The exact scalar boundary decays linearly in

1-\beta

, not quadratically. No sufficiency claim for general nonlinear models is made for the upper curve.

Figure 4: Samplewise sector

[1,1.5]

. Dots are the largest strictly verified steps located by the specified grid and local refinement; connecting lines are visual guides, not certification of every intervening parameter value. The upper curve is the per-mode Schur bound.

Figure 5: Samplewise sector

[1,3]

. The optimized common metric substantially improves the fixed slow-coordinate metric. These dots do not assert that every smaller step is feasible for a common quadratic norm; no monotonicity of feasible steps is assumed.

Figure 6: Nonlinear model,

\beta=0.8

\gamma=0.015

. Each observed squared metric is normalized by its initial value and compared with the corresponding certified factor. Late flattening near machine precision is floating-point round-off, not a nonzero mathematical coupling error.

Figure 7: The same nonlinear sector model at

\gamma=0.9

, certified by the endpoint LMI rather than the explicit slow-coordinate bound. Values below the plotting range or equal to zero after floating-point coalescence are not used to estimate a decay rate.

Figure 8: Mean-only example with a negative sample slope of positive probability. The exact second-moment recursion is available independently of Monte Carlo sampling and gives a model-specific benchmark for the theorem’s bound.

Figure 9: Fractional-moment synchronous coupling under infinite-variance Pareto noise. The finite

p

th moment is

p=1.4<1.5

; no variance estimate is used in the displayed product distance.

Initial pair	Mean	Standard deviation	Exact initial shift	Coverage
$(-30,-30)$	-0.0165	1.0080	-0.0225	0.9463
$(0,0)$	0.0060	1.0080	0.0000	0.9483
$(30,30)$	0.0285	1.0080	0.0225	0.9477

Table 3: Full-orbit quadratic CLT experiment:

n=40000

3000

replications, and known long-run variance one. Coverage uses

\pm z_{0.975}

on the standardized statistic.

Figure 10: Normal quantile comparison for full-orbit averages from three separated initial pairs. The innovations are non-Gaussian; the normal comparison concerns the iterate average. No initial iterates are discarded.

$\beta$	Exact marginal variance	Estimated marginal variance	Estimated long-run variance
0.00	0.25000	0.24997 (0.00029)	0.9931 (0.0163)
0.30	0.29368	0.29371 (0.00028)	1.0110 (0.0220)
0.50	0.33766	0.33746 (0.00046)	1.0196 (0.0199)
0.70	0.40136	0.40157 (0.00033)	1.0003 (0.0141)
0.90	0.49968	0.50043 (0.00066)	1.0475 (0.0172)
0.95	0.53302	0.53283 (0.00047)	0.9540 (0.0187)

Table 4: Quadratic marginal and long-run variance estimates. Parentheses contain Monte Carlo standard errors across

12

independent runs; the exact long-run variance equals one.

Figure 11: Marginal stationary variance for the additive quadratic model: exact formula and Monte Carlo means with

1.96

standard errors across independent runs.

Figure 12: Long-run variance for the same parameters, estimated by nonoverlapping batch means. The exact value is one for every momentum, although the marginal variance in Figure 11 changes substantially.

Stochastic Optimization Convergence

Stochastic optimization convergence studies how noisy, sampled, or delayed updates approach solutions and how fast they do so.

See all 200 papers

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuNewton-Schulz Iteration Gradient Descent
Pengcheng Laboratory, Shenzhen, China
Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework
May 29, 2026Zaiwei Chen, Siva Theja MaguluriControl Lyapunov Functions Lyapunov Stability
Purdue IE · Georgia Tech IE
Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients
Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network Stability Neural Network Training Dynamics
Chandar Research Lab · Mila – Quebec AI Institute · Université de Montréal +4

Sep 30, 2026J/K move · Enter open · S save

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Hanyng Peng, Hui Wang, Yue Yu

Pengcheng Laboratory, Shenzhen, China

Newton-Schulz Iteration Gradient Descent Nonconvex Stochastic Optimization Muon Optimizer Stochastic Optimization Convergence

Open paper AskPDF arXiv

Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one global orthogonalization. Under gradient-dependent

(\mathcal L_0,\mathcal L_1,q)

-smoothness and conditionally unbiased stochastic gradients with bounded layer-wise variance, we establish an

\mathcal O(T^{-1/4})

bound on the expected average Frobenius gradient norm. The analysis retains the Nesterov recursion and requires neither bounded stochastic gradients, symmetric noise, nor a uniform positive lower bound on the nonzero output singular values. Its constants contain no explicit matrix-dimension or rank factors when the number of blocks and problem constants are fixed. The proof follows a descent inequality and a decomposition of the momentum tracking error into initialization, noise, and drift. For the original five-step quintic, we verify the required scalar-map bounds analytically; the result also allows step-dependent coefficients satisfying the same bounds. A complementary nuclear-norm result quantifies rank dependence under a stronger spectral condition. The vanishing rate uses coupled learning-rate and momentum schedules, including the standard single-coefficient Nesterov rule.

Abstract

Stochastic Optimization Convergence

Explore similar work

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework

Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients