math.OCNov 8, 2026

Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning

Authors: Yaxin Yu, Long Chen, Minfu Feng

Organizations: School of Mathematics, Sichuan University, Chengdu, China · Department of Mathematics, University of California, Irvine, CA, USA

Abstract

Stochastic first-order methods use sampled gradients to reduce per-iteration costs, but estimation errors can slow convergence and, in some settings, cause divergence. To improve gradient accuracy while retaining acceleration, we develop scalar and diagonal versions of VR-Adam-SHANG for smooth stochastic convex optimization. The methods combine STORM-type recursive estimation, Adam-style preconditioning, and SHANG's accelerated interpolation and correction, using at most two stochastic gradient evaluations per iteration without full-gradient refreshes. Under unbiased sampling, bounded variance, and mean-square smoothness of gradient noise, polynomial steps with prescribed initialization yield O(1/K)O(1/K) mean-square gradient-estimation error and expected objective-gap bounds combining O(1/K2)O(1/K^2) initial-energy decay with an O(1/K)O(1/\sqrt K) stochastic term after KK iterations. These guarantees require neither monotonicity of the preconditioner nor a uniform positive lower bound on its eigenvalues. The analysis also establishes polynomial-step convergence of projected nonrecursive Adam-SHANG-s, filling a gap in its original theory. Experiments compare the methods at matched sample-gradient budgets. In every tested setting, at least one of the two versions achieves the lowest mean terminal objective gap among all compared methods.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization

    May 13, 2026Yaxin Yu, Long Chen, Minfu FengStochastic OptimizationMomentum Methods

  2. Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

    Apr 9, 2026Yaxin Yu, Long Chen, Zeyi XuSmooth Convex OptimizationAccelerated Gradient Methods

  3. SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

    Mar 10, 2026Yaxin Yu, Long Chen, Minfu FengAccelerated Gradient MethodsStochastic Optimization Convergence