Accelerated Gradient Methods

Follow this topic Ask about this topic

Accelerated gradient methods seek faster optimization by improving on ordinary gradient descent, often using momentum or estimates of curvature. Classical approaches include Nesterov acceleration and heavy-ball momentum, while FISTA extends acceleration to problems with nonsmooth regularizers; newer variants combine these ideas with adaptive preconditioning, stochastic mini-batches, or distributed communication. They are used in machine learning and in structured tasks such as sparse optimization, matrix problems, and linear programming. A central challenge is that acceleration can be sensitive to noise, conditioning, and problem structure, so researchers study when fast convergence guarantees hold and when momentum or predetermined step sizes can fail.

Latest papers

44 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

When Are Nesterov Trajectories Eventually Locally Minimizing?
Oct 23, 2026Peyman Morteza, Stephen WrightAccelerated Gradient Methods
Department of Computer Sciences University of Wisconsin–Madison
Weak- $L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
Oct 20, 2026Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov +4Gradient Clipping Heavy-Tailed Stochastic Optimization
Independent researcher · Yaroslavl State University · Moscow Independent Research Institute of Artificial Intelligence +2
Non-Acceleration of the Heavy-Ball Method in 1D
Oct 18, 2026Konstantinos Fotopoulos, Michael Helcig, Karl Deck +1Momentum Methods Accelerated Gradient Methods
Department of CS, ETH Zürich, Switzerland
Acceleration and its Limits in Gaussian Variational Inference
Oct 14, 2026Jaesung R. Park, Uijeong Jang, Yifan ChenOracle Complexity Variational Inference
Department of Mathematics, University of California, Los Angeles
Neural Algorithmic Reasoning for Graph Saddle Point Problems
Oct 5, 2026Samantha Chen, Jesse He, Coleman Clougherty +2Primal-Dual Optimization Graph Neural Networks
Oberlin College · Department of Computer Science, Oberlin College · University of California, San Diego +2
Reinforcement Learning to Accelerate Primal-Dual Hybrid Gradient for Linear Programming
Oct 1, 2026Jinhwan Sul, Alex Oshin, Evangelos A. TheodorouPrimal-Dual Optimization Reinforcement Learning
Georgia Institute of Technology, USA
Accelerated Algorithm for Sparse Regularized Partial Optimal Transport
Sep 30, 2026Khoa Nguyen, Dung T. Nguyen, Thong Huynh +4Inverse Optimal Transport Optimal Transport
Aalto University, Konemiehentie 2, 02150 Espoo, Finland · Linköping University, 581 83 Linköping, Sweden · Hanoi University of Science and Technology, Hanoi, Vietnam +4
Geometric Moment Contraction for Stochastic Nesterov Acceleration
Sep 25, 2026Wei Biao WuStochastic Optimization Convergence Accelerated Gradient Methods
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Sep 25, 2026Miao Lu, Zeyu Bian, Kaiyue Wen +4Momentum Methods Gradient Descent Dynamics
Stanford University · University of California, San Diego · University of Chicago +2
Convergence guarantees for Muon: New parameter regimes and generalizations
Sep 24, 2026Arthur C. B. de Oliveira, Dhruv D. Jatkar, Guilherme S. Vicinansa +1Muon Optimizer Optimization Convergence Analysis
Department of Electrical and Computer Engineering Northeastern University Boston, MA 02115 · Department of Telecommunications and Control Escola Politecnica, University of São Paulo São Paulo, SP 05508-010
Near-Optimal Acceleration for Smooth $\ell_p$ / $\ell_q$ Nondual Convex First-Order Oracle Optimization
Sep 18, 2026David Martínez-Rubio, Brian Bullins, Cristóbal Guzmán +1Accelerated Gradient Methods Constrained Optimization
IMDEA Software Institute, Madrid, Spain · Purdue University, West Lafayette, IN, USA · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Cat´olica de Chile, Santiago, Chile +1
Silver Rate Is (Almost) Optimal for Gradient Descent
Sep 8, 2026Yuhan Ye, Kaizhao LiuLearning Rate Scheduling Optimization Convergence Analysis
MIT
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Aug 28, 2026Shuchen Zhu, Yuxin Fang, Mingze Wang +1Curvature-Aware Optimization Momentum Methods
ByteDance Seed · Peking University
Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
Aug 19, 2026Euijin Hong, Guannan QuDeep Learning Optimization Momentum Methods
Electrical and Computer Engineering, Carnegie Mellon University
Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
Aug 13, 2026Zhixin Ren, Yao Lyu, Congrong Li +2Deep Learning Optimization Momentum Methods
Department of Mathmatical Science, Tsinghua University · College of AI, Tsinghua University · School of Vehicle and Mobility, Tsinghua University
A lower bound for stepsize-based acceleration of gradient descent
Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate Scheduling Last-Iterate Convergence
Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Aug 5, 2026Thang Do, Steffen Dereich, Arnulf JentzenStochastic Optimization Newton-Schulz Iteration
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistics, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam · Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany +2
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
Jun 17, 2026Depen Morwani, Alexandru Meterez, Pranav Nair +1Stochastic Optimization Momentum Methods
1Harvard University · 2Kempner Institute at Harvard University · †: Work done while at Harvard.
When to use what Schatten- $p$ norm in deep learning?
Jun 13, 2026Thomas PethickDeep Learning Optimization Muon Optimizer
Predictive Coding with Bayesian Priors via Proximal Gradients
Jun 6, 2026Francesco BulloPredictive Coding Markov Random Fields
Mechanical Engineering, 2325 Engineering Bldg II, UCSB, Santa Barbara, CA 93106-5070, USA
Accelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization
Jun 5, 2026Ming Sun, Kun YuanGradient Descent Accelerated Gradient Methods
Center for Machine Learning Research Peking University
Near-Optimal Decentralized Stochastic Convex Optimization over Networks
Jun 3, 2026Nitai Kluger, Amit Attia, Tomer KorenStochastic Optimization Convergence Accelerated Gradient Methods
Blavatnik School of Computer Science, Tel Aviv University · Google Research Tel Aviv
Stochastic Gradient Descent with Momentum is Algorithmically Stable
May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanGeneralization Bounds Algorithmic Stability
Department of Mathematics, The University of Hong Kong, Hong Kong, China · Department of Mathematics and Mathematical Statistics, Umeå University, Sweden
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
May 26, 2026Jiacheng Li, Jianchao Tan, Hongtao Xu +5Language Model Pretraining Gradient Descent
Meituan, Beijing, China
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning Optimization Gradient Descent
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong
Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration
May 18, 2026Sachin Garg, Michał DerezińskiMomentum Methods Stochastic Optimization Convergence
University of Michigan
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
May 15, 2026Manuel Graca, L. Miguel Silveira, Arlindo Oliveira +1Deep Learning Optimization Gradient Descent
IST, ULisboa Lisbon, Portugal · Old Dominion University Norfolk, VA USA
A Unified Lyapunov-IQC Framework for Uniform Stability of Smooth Quadratic First-Order Accelerated Optimizers
May 8, 2026Don Li, Dacian DaescuLyapunov Stability Algorithmic Stability
Department of Mathematics & Statistics, Portland State University, Portland, OR 97201, USA
A Parameter-Free First-Order Algorithm for Non-Convex Optimization with $\tilde{\mkern1mu O}(ε^{-5/3})$ Global Rate
May 4, 2026Sichao Xiong, Sadok Jerad, Coralia CartisOptimization Convergence Analysis Gradient Descent
Department of Computer Science University of Oxford Oxford, OX1 3QG · Mathematical Institue University of Oxford Oxford, OX2 6GG
Randomized Subspace Nesterov Accelerated Gradient
May 1, 2026Gaku Omiya, Pierre-Louis Poirion, Akiko TakedaGradient Descent Randomized Sketching
Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan

Oct 23, 2026J/K move · Enter open · S save

When Are Nesterov Trajectories Eventually Locally Minimizing?

Peyman Morteza, Stephen Wright

Department of Computer Sciences University of Wisconsin–Madison

Accelerated Gradient Methods

Open paper AskPDF arXiv

We revisit the variational characterization of Nesterov's celebrated method and study when Nesterov trajectories are eventually locally minimizing in a variational sense, a notion that we define precisely. We show that this question is equivalent to the nonnegativity of an associated Schrödinger operator acting on variations of the trajectory. For power objectives of the form

f(x)=\frac{1}{p}\|x\|^p

(where

\| \cdot\|

denotes the Euclidean norm), we give a complete characterization: Among all exponents

p\ge 2

, the Nesterov trajectory with non-zero initial position is eventually locally minimizing if and only if

p=4

. We further interpret this exceptional exponent through the stability theory of semilinear elliptic equations.