SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

Yaxin Yu; Long Chen; Minfu Feng

math.OCMar 10, 2026

SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

Authors: Yaxin Yu, Long Chen, Minfu Feng

Organizations: School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, China · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA

Abstract

Under the multiplicative noise scaling (MNS) condition, original Nesterov acceleration is provably sensitive to noise and may diverge when gradient noise overwhelms the signal. In this paper, we develop two accelerated stochastic gradient descent methods by discretizing the Hessian-driven Nesterov accelerated gradient flow. We first derive SHANG, a direct semi-implicit discretization that already improves stability under MNS. We then introduce SHANG++, which adds a damping correction and achieves faster convergence with greater noise robustness. We establish convergence guarantees for both convex and strongly convex objectives under MNS, together with explicit parameter choices. In our experiments, SHANG++ performs consistently well across convex problems and applications in deep learning. In a dedicated noise experiment on ResNet-34, a single hyperparameter configuration maintains accuracy within one percentage point of the noise-free setting. Across all experiments, SHANG++ outperforms existing accelerated methods in robustness and efficiency, with minimal parameter sensitivity.

Figures & tables

Panel of Figure 1 — Figure 1: Performance of different algorithms under varying noise levels.

Panel of Figure 2 — Figure 2: Training and test loss (log scale, running average with decay $0.99$ ) on MNIST with LeNet-5 (batch size $50$ ).

Panel of Figure 3 — Figure 3: Training loss (left) and test loss (right) in log scale (running average with decay $0.99$ ) on CIFAR-10 with ResNet-34, for batch sizes $32$ (top row), $50$ (middle row), and $256$ (bottom row).

Panel of Figure 4 — Figure 4: Training and test loss (log scale, running average with decay $0.99$ ) on CIFAR-100 with ResNet-50 (batch size $50$ ).

	SGD	SHB	NAG	Adam	AGNES	SNAG	SHANG	SHANG++
LeNet-5	$91.07$	$98.98$	$98.9$	$\underline{99.07}$	$98.88$	$\underline{99.07}$	$99.06$	$\mathbf{99.11}$
$(b=50)$	$\pm 0.11$	$\pm 0.05$	$\pm 0.08$	$\pm 0.07$	$\pm 0.09$	$\pm 0.08$	$\pm 0.02$	$\pm 0.03$
ResNet-34	$81.74$	$78.9$	$81.28$	$\mathbf{86.99}$	$67.45$	$75.58$	$84.5$	$\underline{85.36}$
$(b=32)$	$\pm 0.38$	$\pm 1.67$	$\pm 1.58$	$\pm 0.14$	$\pm 7.7$	$\pm 6.02$	$\pm 2$	$\pm 1.42$
ResNet-34	$79.91$	$84.59$	$86.43$	$\underline{87.38}$	$70.49$	$77.65$	$87.15$	$\mathbf{87.4}$
$(b=50)$	$\pm 0.11$	$\pm 2.62$	$\pm 0.81$	$\pm 0.26$	$\pm 2.51$	$\pm 2.7$	$\pm 0.82$	$\pm 0.5$

Table 1: Test accuracy of SGD, SHB, NAG, Adam, AGNES, SNAG, SHANG, and SHANG++ on MNIST (LeNet-5), CIFAR-10 (ResNet-34), and CIFAR-100 (ResNet-50). Here

b

is batch size. Larger values are better. Best is in bold and second best is underlined.

Panel of Figure 6 — Figure 6: Training and test loss (log scale, running average with decay $0.99$ ) on CIFAR-10 using U-Net with batch size $5$ .

Accelerated Gradient Methods

Accelerated gradient methods use momentum, curvature, or adaptive updates to reduce the iterations needed for optimization.

See all 43 papers

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning Optimization Gradient Descent
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong
Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGD
Apr 11, 2026Konstantinos Emmanouilidis, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Rene VidalRandom Reshuffling Richardson Extrapolation
Department of Computer Science, University of Pennsylvania · University of Wisconsin-Madison, Archimedes Research Unit
Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization
May 13, 2026Yaxin Yu, Long Chen, Minfu FengStochastic Optimization Momentum Methods
School of Mathematics Sichuan University Chengdu, China · Department of Mathematics University of California, Irvine Irvine, CA, USA

May 25, 2026J/K move · Enter open · S save

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

Chung-Yiu Yau, Dawei Li, Athanasios Glentis, Valentyn Boreiko+2

University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong

Deep Learning Optimization Gradient Descent Accelerated Gradient Methods

Open paper AskPDF arXiv

Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochastic gradient noise and non-convex loss landscapes. In particular, standard lookahead relies on short-horizon update signals (e.g., differences between consecutive iterates), which are inherently noisy and can lead to unstable extrapolation directions. This work revisits Nesterov's acceleration from a trajectory perspective and argues that effective acceleration in deep learning should harness the low-frequency trends of optimization trajectories rather than extrapolating noisy one-step updates. Leveraging this insight, we propose EMA-Nesterov, a simple modification that replaces the standard Nesterov's lookahead direction with an exponential moving average (EMA) of parameter updates. This yields a stabilized lookahead direction that captures and harnesses the evolving trend of the training trajectory through a low-pass filter, while remaining adaptive to progressive changes via the geometric weighting structure of EMA. We show that EMA-Nesterov retains a theoretical accelerated convergence rate in convex problems that is analogous to Nesterov's accelerated gradient method. Furthermore, we provide empirical evidence on language model pre-training to verify that EMA-Nesterov is broadly applicable across a range of fine-tuned base optimizers, including Adam, SOAP, Muon, as well as complex optimizers that achieve state-of-the-art performance on optimization benchmarks (NanoGPT). Compared to prior lookahead methods, EMA-Nesterov achieves better performance by avoiding the instability of short-horizon lookahead and the non-adaptivity of long-horizon lookahead.

Abstract

Accelerated Gradient Methods

Explore similar work

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGD

Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization