Adaptive Gradient Methods

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 113

All topics

A Rod Flow Model for Adam at the Edge of Stability
May 7, 2026Eric Regis, Sinho ChewiEdge of Stability Momentum Methods
Yale University
Revisiting Adam for Streaming Reinforcement Learning
May 7, 2026Florin Gogianu, Adrian Catalin Lutu, Razvan PascanuDistributional RL Reinforcement Learning
Bitdefender, Bucharest, Romania · Mila, Montreal, Canada
Directional Consistency as a Complementary Optimization Signal: The GONO Framework
May 7, 2026Victor Daniel GeraDeep Learning Optimization Stochastic Optimization Convergence
Department of Artificial Intelligence Anurag University, Hyderabad, India
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
May 7, 2026Ziqing Wen, Zhouyang Liu, Jiahuan Wang +4Deep Learning Optimization Learning Rate Scheduling
College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China
On Adaptivity in Zeroth-Order Optimization
May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order Optimization LLM Fine-Tuning
Qualcomm AI Research
FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction
May 5, 2026Duc Dm, Thao Do, Minh Son Hoang +3Differentially Private Stochastic Gradient Descent Privacy-Preserving ML
School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Center for Environmental Intelligence (CEI), VinUniversity, Hanoi, Vietnam · Khoury College of Computer Sciences, Northeastern University, USA
Anon: Extrapolating Adaptivity Beyond SGD and Adam
May 4, 2026Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5Nonconvex Stochastic Optimization Adaptive Gradient Methods
Wuhan University · University of Macau · University of Calgary
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
May 1, 2026Zhijie Cai, Haolong Chen, Guangxu ZhuMemory-Efficient Fine-Tuning Zeroth-Order Optimization
Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong-Shenzhen · Shenzhen Loop Area Institute
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution
Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning Optimization Fine-Tuning
School of AI, Jilin International Sciences University, Changchun, China.
NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics
Apr 29, 2026Douglas Jiang, Yuechen Wang, Jiayi Wang +3Neural Network Optimization Adaptive Gradient Methods
Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02115, USA · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA · Equal contributions +2
Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair
Apr 24, 2026Yuelin Hu, Zhenbo Yu, Zhengxue Cheng +2Continual Learning for LLMs Adaptive Gradient Methods
Shanghai Jiao Tong University · Shanghai Maritime University
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo
Apr 19, 2026S. Gratton, Ph. L. TointMomentum Methods Nonconvex Stochastic Optimization
Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
Mar 19, 2026Steffen Dereich, Thang Do, Arnulf JentzenStochastic Optimization Stochastic Optimization Convergence
Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistic, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam +2
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
Jan 30, 2026Cheng Ge, Caitlyn Heqi Yin, Hao Liang +1RL for Language Model Reasoning Group Relative Policy Optimization
Department of Aeronautics and Astronautics, MIT · Department of Statistics, University of Wisconsin–Madison · Department of Informatics, King’s College London +1
Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum Methods Adaptive Gradient Methods
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
ZENITH: Automated Gradient Norm Informed Stochastic Optimization
Jan 21, 2026Dhrubo SahaDeep Learning Optimization Learning Rate Scheduling
Georgia Institute of Technology
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
Nov 10, 2025Steffen Dereich, Thang Do, Arnulf Jentzen +1Stochastic Optimization Stochastic Optimization Convergence
Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China +1
Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning Optimization Neural Network Optimization
1Vector Institute, Toronto, Canada · University of Cambridge, Cambridge, United Kingdom · 3Microsoft, United States +1
On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
May 17, 2025Huan Li, Yiming Dong, Zhouchen LinOptimization Convergence Analysis Stochastic Optimization Convergence
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, China. · National Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China.
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
Jun 20, 2024Steffen Dereich, Arnulf Jentzen, Adrian RiekertStochastic Optimization Convergence Adaptive Gradient Methods
Institute for Mathematical Stochastics, University of Münster, Germany · School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany
Generalizing Adam to Manifolds for Efficiently Training Transformers
May 26, 2023Benedikt BrantnerStiefel Manifold Optimization Riemannian Optimization
Max-Planck-Institut für Plasmaphysik Boltzmannstraße 2, 85748 Garching, Deutschland · Technische Universität München, Zentrum Mathematik Boltzmannstraße 3, 85748 Garching, Deutschland
Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant
Jan 26, 2022John ChiangLogistic Regression Gradient Descent
Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
Date pendingSharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic Optimization Stochastic Optimization Convergence
Department of Statistics and Data Science Cornell University

May 7, 2026, cs.LGJ/K move · Enter open · S save

A Rod Flow Model for Adam at the Edge of Stability

Eric Regis, Sinho Chewi

Yale University

Edge of Stability Momentum Methods Adaptive Gradient Methods Gradient Descent Dynamics

Open paper AskPDF arXiv

Cohen et al. (arXiv:2207.14484) observed that adaptive gradient methods such as Adam operate at the edge of stability. While there has been significant work on continuous-time modeling of gradient descent at the edge of stability, extending these models to momentum methods remains underdeveloped. In the gradient descent setting, Regis et al. (arXiv:2602.01480) introduced rod flow, which models consecutive iterates as an extended one-dimensional object -- a "rod." Here we extend rod flow to Adam by working in the joint phase space of parameters and first moment

(w, m)

and treating the second moment

ν

as a smooth auxiliary variable. We also develop rod flows for heavy ball momentum, Nesterov momentum, and scalar and per-component versions of RMSProp, Adam, and NAdam. For all eight optimizers, we empirically evaluate rod flow on representative machine learning architectures, where it tracks the discrete iterates through the edge-of-stability regime significantly more accurately than the corresponding stable flow.

Momentum

Related topics

Latest papers 113