Momentum Methods

Follow this topic Ask about this topic

Momentum methods study how accumulated gradients or optimizer states can speed learning and stabilize progress, from heavy-ball and Nesterov updates to exponential moving averages in Adam and orthogonalized methods such as Muon. Researchers analyze convergence, stability, generalization, and compute costs in settings including stochastic and nonconvex optimization, min-max problems, federated and asynchronous training, and large-batch neural-network training. Practical approaches include adaptive momentum schedules, variance reduction, sign-based updates, and matrix-aware preconditioning, with evaluations ranging from convex models to vision and language-model training. A central tension is that momentum can enable faster progress and larger batches, but its benefits depend on noise, curvature, memory timescales, and hyperparameter choices that may become unreliable at larger scales.

Latest papers

58 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

When Momentum Helps Polyak--Ruppert Averaging
Oct 20, 2026Alexandra Suvorikova, Darina Dvinskikh, Yuriy Dorn +1Stochastic Optimization Stochastic Optimization Convergence
Independent researcher · HSE University · MSU AI Institute +1
Hybrid Momentum Methods for Optimizing Deep Learning Models
Oct 20, 2026Haifeng Xu, Yijin Ren, Chenyu Zhou +3Deep Learning Optimization Momentum Methods
Shanghai University of Finance and Economics · Columbia University · Shanghai Jiao Tong University +2
Non-Acceleration of the Heavy-Ball Method in 1D
Oct 18, 2026Konstantinos Fotopoulos, Michael Helcig, Karl Deck +1Momentum Methods Accelerated Gradient Methods
Department of CS, ETH Zürich, Switzerland
Adding Heavy-Ball Momentum to Distributed Constrained Optimization for Energy Resource Management Subject to Time-Delays and Quantization
Oct 17, 2026Mohammadreza Doostmohammadian, Sergio PequitoDistributed Optimization Constrained Optimization
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
Oct 14, 2026Kaifei Wang, Zikang Shan, Liwei Wang +2Bandit Learning Momentum Methods
Peking University · Shanghai Jiao Tong University, SIMIS, and Stanford University · Shanghai Jiao Tong University
Randomized Momentum Transport with Cheap Predictors
Oct 9, 2026El Mahdi Chayti, Martin JaggiMomentum Methods Stochastic Optimization Convergence
Machine Learning and Optimization Laboratory (MLO), EPFL
Muon meets Tamed Langevin: Momentum Preconditioning beyond Convex and gradient-Lipschitz Potentials
Oct 1, 2026Nikolaos Makras, Sotirios SabanisMarkov Chain Monte Carlo Momentum Methods
School of Mathematics, University of Edinburgh, UK · National Technical University of Athens, Greece · Athena/Archimedes Research Centre, Greece
Optimal Momentum Methods for Stochastic Multilevel Compositional Optimization
Oct 1, 2026Wei Jiang, Rui Yan, Sifan Yang +3Stochastic Optimization Momentum Methods
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Software Technology, Zhejiang University, Ningbo, China
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Sep 25, 2026Miao Lu, Zeyu Bian, Kaiyue Wen +4Momentum Methods Gradient Descent Dynamics
Stanford University · University of California, San Diego · University of Chicago +2
COREM: Cosine-Relation Momentum Reshaping with Stateful Writeback
Sep 18, 2026Yan Wang, Xiaochuan Wang, Yuxiang SunNeural Network Optimization Momentum Methods
Aalto University · Helsinki University · City University of Hong Kong
A pullback-corrected scalar auxiliary variable optimizer with momentum and adaptive mobility
Sep 11, 2026Jiahao Zhang, Shiheng Zhang, Guang LinNeural Network Optimization Adaptive Gradient Methods
School of Mechanical Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Mathematics, University of Washington, Seattle, WA 98195, USA · Department of Mathematics, Purdue University, West Lafayette, IN 47906, USA
Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency
Sep 2, 2026Jia-Nan Wang, Zixun Huang, Kairui Li +2Momentum Methods Gradient Descent
Peking University · University of Pennsylvania · AI for Science Institute, Beijing
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Aug 28, 2026Shuchen Zhu, Yuxin Fang, Mingze Wang +1Curvature-Aware Optimization Momentum Methods
ByteDance Seed · Peking University
A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization
Aug 24, 2026Yong Zhao, Chunlin You, Minh N. Dao +1Multi-Objective Optimization Federated Learning
College of Mathematics and Statistics, Chongqing Jiaotong University, Chongqing 400074, China. · Key Laboratory of Complex Systems Optimization and Intelligent Control of Chongqing Municipal Education Commission, Chongqing 400074, China. · School of Science, RMIT University, Melbourne, VIC 3000, Australia. +2
Variational Structure at the Edge of Stability
Aug 21, 2026Eric RegisEdge of Stability Gradient Descent Dynamics
Yale University
Enhanced Artificial Neural Networks Using QHAdamW in Air Quality Forecasting
Aug 20, 2026Mary Joy Daniel VinasAir Quality Prediction Neural Network Optimization
Computer Studies Department, Technological University of the Philippines, City of Manila, Philippines.
Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
Aug 19, 2026Euijin Hong, Guannan QuDeep Learning Optimization Momentum Methods
Electrical and Computer Engineering, Carnegie Mellon University
Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
Aug 13, 2026Zhixin Ren, Yao Lyu, Congrong Li +2Deep Learning Optimization Momentum Methods
Department of Mathmatical Science, Tsinghua University · College of AI, Tsinghua University · School of Vehicle and Mobility, Tsinghua University
An Inertial Block Proximal Linearized Method with Adaptive Momentum for Nonconvex and Nonsmooth Optimization
Aug 6, 2026Weifeng YangMomentum Methods Nonconvex Optimization
Yunnan Earthquake Administration, Kunming 650224, China
Nesterov acceleration in optimizing over probability measures
Jul 25, 2026Jiaqi Tang, Qin Li, Wilfrid GangboMomentum Methods Gradient Descent
Department of Statistics, University of Wisconsin–Madison, Madison, WI 53706, USA · Department of Mathematics, University of Wisconsin–Madison, Madison, WI 53706, USA · Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA
Muse: Representation Geometry of Muon Beyond Normalized Momentum
Jul 16, 2026Da Chang, Qiankun Shi, Lvgang Zhang +4Representation Geometry Momentum Methods
1Pengcheng Laboratory · 2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +3
Heavy-Ball Q-Learning with Residual Weighting Correction
Jun 25, 2026Donghwan LeeMomentum Methods Q-Learning
Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, South Korea
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
Jun 17, 2026Depen Morwani, Alexandru Meterez, Pranav Nair +1Stochastic Optimization Momentum Methods
1Harvard University · 2Kempner Institute at Harvard University · †: Work done while at Harvard.
CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
Jun 15, 2026Bishnu Dev, Sushil Bohara, Martin Takáč +1Momentum Methods Neural Network Optimization
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Mohamed bin Zayed University of Artificial Intelligence
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
Jun 7, 2026Ganzhao YuanMomentum Methods Nonconvex Stochastic Optimization
Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarNewton-Schulz Iteration Momentum Methods
MIT
Fast Generalization after Interpolation via Critically Damped Momentum Optimization
Jun 1, 2026Luca Muscarnera, Silas Ruhrberg Estévez, Yuanzhang Xiao +1Neural Network Generalization Momentum Methods
University of Cambridge · University of Hawaii at Manoa
Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
May 27, 2026Katie Everett, Elliot PaquetteHigh-Dimensional Asymptotics Momentum Methods
Google DeepMind & MIT · McGill University & Mila
Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
May 27, 2026Kristi Topollai, Allan Ma, Tolga Dimlioglu +2Momentum Methods Communication-Efficient Distributed Training
New York University, New York, USA

Oct 20, 2026J/K move · Enter open · S save

When Momentum Helps Polyak--Ruppert Averaging

Alexandra Suvorikova, Darina Dvinskikh, Yuriy Dorn, Alexander Gasnikov

Independent researcher · HSE University · MSU AI Institute+1

Stochastic Optimization Stochastic Optimization Convergence Polyak-Ruppert Averaging Momentum Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

Polyak--Ruppert averaging gives SGD and momentum methods the same asymptotic covariance. We characterize how fast finite runs reach it and how far they can be parallelized. For strongly convex quadratics with additive noise, we prove an online--offline bridge: the empirical risk minimizer on the samples of a tail average is one Newton step from the tail-averaged query point of any online procedure. For SGD, heavy ball, and Nesterov's method with constant momentum, this step is a boundary term at the two ends of the tail and splits the gap to the offline estimator into center, covariance, and shape errors. Necessary and sufficient conditions show that, on the least curved direction, the center error sets the pace: SGD needs

m\ggκ^2

rounds but momentum only

m\ggκ

, where

κ

is the condition number, so at sample size

N

the critical batch size grows from

Θ(\sqrt{N}/κ)

Θ(\sqrt{N/κ})

. The covariance error separates the momentum methods: heavy ball has relative variance excess

Θ(κ^{3/2}/m)

along the stiffest direction, whereas Nesterov has relative error

O(\sqrtκ/m)

in every direction. The shape error is method-independent by a Berry--Esseen bound uniform in

κ

and batch size, so confidence intervals are valid when the other two errors vanish. We further examine how the conclusions change under heavy-tailed or state-dependent noise and clipping, and establish a first-order comparison with ERM for nonlinear Nesterov averaging under explicit conditions.