Accelerated Gradient Methods

Follow this topic Ask about this topic

Accelerated gradient methods seek faster optimization by improving on ordinary gradient descent, often using momentum or estimates of curvature. Classical approaches include Nesterov acceleration and heavy-ball momentum, while FISTA extends acceleration to problems with nonsmooth regularizers; newer variants combine these ideas with adaptive preconditioning, stochastic mini-batches, or distributed communication. They are used in machine learning and in structured tasks such as sparse optimization, matrix problems, and linear programming. A central challenge is that acceleration can be sensitive to noise, conditioning, and problem structure, so researchers study when fast convergence guarantees hold and when momentum or predetermined step sizes can fail.

Latest papers

44 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

Deterministic Adam-Inspired Methods with Accelerated Convergence Rate
Apr 9, 2026Yaxin Yu, Long Chen, Zeyi XuAccelerated Gradient Methods Adaptive Gradient Methods
School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, CHINA · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA
Stochastic Auto-conditioned Fast Gradient Methods with Optimal Rates
Apr 7, 2026Yao Ji, Guanghui LanStochastic Optimization Accelerated Gradient Methods
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, 225 North Ave, Atlanta, GA 30332, USA
Scalable Mean-Variance Portfolio Optimization via Subspace Embeddings and GPU-Friendly Nesterov-Accelerated Projected Gradient
Apr 3, 2026Yi-Shuai Niu, Yajuan WangPortfolio Optimization Randomized Sketching
Beijing Institute of Mathematical Sciences and Applications (BIMSA)
Optimal Projection-Free Adaptive SGD for Matrix Optimization
Apr 2, 2026Dmitry KovalevAdaptive Gradient Methods Matrix Optimization
Yandex Research
Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
Mar 30, 2026Ivan PasichnykGradient-Based Attribution Momentum Methods
We Label Data Inc.
Improving Infinitely Deep Bayesian Neural Networks with Nesterov's Accelerated Gradient Method
Mar 26, 2026Chenxu Yu, Wenqi FangBayesian Neural Networks Neural Network Optimization
Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen, P.R.China
SHANG++: Robust Stochastic Acceleration under Multiplicative Noise
Mar 10, 2026Yaxin Yu, Long Chen, Minfu FengAccelerated Gradient Methods Stochastic Optimization Convergence
School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, China · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA
Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation
Mar 9, 2026Adam Rozzio, Rafael Athanasiades, O. Deniz AkyildizMaximum Marginal Likelihood Estimation Expectation-Maximization
Department of Mathematics, Ecole Normale Supérieure Paris-Saclay, France · Department of Mathematics, Imperial College London, UK
FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
Mar 5, 2026Junkang Liu, Fanhua Shang, Yuanyuan Liu +3Federated Learning Accelerated Gradient Methods
School of Artificial Intelligence, Xi’an, Xidian University, China · College of Intelligence and Computing, Tianjin, Tianjin University, China · Medical College, Tianjin University; Peng Cheng Laboratory, Tianjin, China +1
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
Feb 26, 2026Shuchen Zhu, Rizhen Hu, Mingze Wang +4Accelerated Gradient Methods Language Model Pretraining
Peking University · Zhejiang Lab, Zhejiang · Meituan, Beijing
Complexity of Classical Acceleration for $\ell_1$ -Regularized PageRank
Feb 24, 2026Kimon Fountoulakis, David Martínez-RubioAccelerated Gradient Methods
University of Waterloo, Canada · IMDEA Software Institute, Madrid, Spain
Accelerated Gradient Methods for Nonconvex Optimization: Escape Trajectories From Strict Saddle Points and Convergence to Local Minima
Jul 13, 2023Rishabh Dixit, Mert Gurbuzbalaban, Waheed U. BajwaNonconvex Optimization Accelerated Gradient Methods
Department of Electrical and Computer Engineering, Rutgers University–New Brunswick, NJ 08854 USA · Departments of Electrical & Computer Engineering, Management Science and Information Systems, and Statistics, Rutgers University–New Brunswick, NJ 08854 USA · Departments of Electrical & Computer Engineering and Statistics, Rutgers University–New Brunswick, NJ 08854 USA
NAG-GS: A Semi-Implicit Stochastic Optimizer with Large-Step Stability
Sep 29, 2022Valentin Leplat, Daniil Merkulov, Aleksandr Katrutsa +3Stochastic Optimization Accelerated Gradient Methods
Skolkovo Institute of Science and Technology, Moscow, Russia · Moscow Institute of Physics and Technology, Moscow, Russia · AIRI, Moscow, Russia +1
Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant
Jan 26, 2022John ChiangLogistic Regression Gradient Descent

Apr 9, 2026J/K move · Enter open · S save

Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

Yaxin Yu, Long Chen, Zeyi Xu

School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, CHINA · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA

Accelerated Gradient Methods Adaptive Gradient Methods

Open paper AskPDF arXiv

Adam is widely used, but its convergence theory remains incomplete even in the deterministic full-batch setting because momentum and adaptive preconditioning are tightly coupled. For smooth convex objectives, we split the momentum variable through variable-and-operator splitting, which reveals the acceleration mechanism. We then combine a Hessian-driven correction with Adam-style feedback based on the gradient magnitude. The resulting Adam-HNAG (Hessian-driven Nesterov accelerated gradient with Adam-style adaptive preconditioning) flow admits a nonnegative energy that decays exponentially. Its discretization yields two methods, Adam-HNAG and the synchronous variant Adam-HNAG-s. Under the stated trajectory-bound and consistency conditions, both methods satisfy a discrete Lyapunov contraction. If the exact adaptive steps are accepted, this contraction gives an

O(k^{-2})

objective-value bound. Numerical experiments illustrate their behavior. These results apply to the proposed methods, not to the original Adam recursion.