Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning
Organizations: School of Mathematics, Sichuan University, Chengdu, China · Department of Mathematics, University of California, Irvine, CA, USA
Abstract
Stochastic first-order methods use sampled gradients to reduce per-iteration costs, but estimation errors can slow convergence and, in some settings, cause divergence. To improve gradient accuracy while retaining acceleration, we develop scalar and diagonal versions of VR-Adam-SHANG for smooth stochastic convex optimization. The methods combine STORM-type recursive estimation, Adam-style preconditioning, and SHANG's accelerated interpolation and correction, using at most two stochastic gradient evaluations per iteration without full-gradient refreshes. Under unbiased sampling, bounded variance, and mean-square smoothness of gradient noise, polynomial steps with prescribed initialization yield mean-square gradient-estimation error and expected objective-gap bounds combining initial-energy decay with an stochastic term after iterations. These guarantees require neither monotonicity of the preconditioner nor a uniform positive lower bound on its eigenvalues. The analysis also establishes polynomial-step convergence of projected nonrecursive Adam-SHANG-s, filling a gap in its original theory. Experiments compare the methods at matched sample-gradient budgets. In every tested setting, at least one of the two versions achieves the lowest mean terminal objective gap among all compared methods.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Accelerated Gradient Methods
Accelerated gradient methods use momentum, curvature, or adaptive updates to reduce the iterations needed for optimization.
See all 48 papers