Adaptive Gradient Methods

Ask about this topic

Overview

Adaptive gradient methods use information from current and past gradients to choose update sizes and directions, aiming to train more reliably than fixed-step gradient descent. Common approaches include momentum and learning-rate schedules, coordinate-wise methods such as Adam and AdaGrad, and block or matrix preconditioners such as Shampoo, often combined with orthogonalized updates. Researchers study their convergence and stability as well as their practical use in large language models, vision, scientific machine learning, and privacy-sensitive training. A central tension is that adaptivity can improve speed or robustness but costs memory and computation, and no optimizer consistently wins across tasks; guarantees and generalization depend on data, noise, geometry, and hyperparameter choices.

Latest papers

CardsList

157 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo

    Apr 19, 2026S. Gratton, Ph. L. TointMomentum MethodsNonconvex Stochastic Optimization

    Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.

  2. A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

    Apr 15, 2026Damek Davis, Dmitriy DrusvyatskiyGradient DescentAdaptive Gradient Methods

    Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104

  3. Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

    Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning OptimizationAdaptive Gradient Methods

    Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1

  4. Muon2^2: Boosting Muon via Adaptive Second-Moment Preconditioning

    Apr 11, 2026Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4Neural Network OptimizationMuon Optimizer

    University of California at Santa Barbara · University at Albany, SUNY

  5. Delve into the Applicability of Advanced Optimizers for Multi-Task Learning

    Apr 10, 2026Zhipeng Zhou, Linxiao Cao, Pengcheng Wu +2Orthogonal OptimizationMulti-Task Learning

    Nanyang Technological University · Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Shanghai Jiao Tong University

  6. Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

    Apr 9, 2026Yaxin Yu, Long Chen, Zeyi XuAccelerated Gradient MethodsAdaptive Gradient Methods

    School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, CHINA · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA

  7. FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection

    Apr 8, 2026Devender Singh, Tarun SheelHypergradient OptimizationAdaptive Gradient Methods

    Department of Computer Science Memorial University of Newfoundland St. John’s, NL, Canada · Department of Mathematics and Statistics Memorial University of Newfoundland St. John’s, NL, Canada

  8. Optimal Projection-Free Adaptive SGD for Matrix Optimization

    Apr 2, 2026Dmitry KovalevAdaptive Gradient MethodsMatrix Optimization

    Yandex Research

  9. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

    Mar 19, 2026Steffen Dereich, Thang Do, Arnulf JentzenStochastic OptimizationStochastic Optimization Convergence

    Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistic, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam +2

  10. Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

    Mar 16, 2026Pratyush Acharya, Habish DhakalGrokkingNeural Network Training Dynamics

  11. OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

    Mar 10, 2026Ganzhao YuanStochastic OptimizationAdaptive Gradient Methods

    Shenzhen University of Advanced Technology (SUAT), China

  12. Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts

    Mar 10, 2026Yuze Dong, Jinsong WuAdaptive Gradient MethodsNon-Stationary Time Series Forecasting

    School of Artificial Intelligence, Guilin University of Electronic Technology 1 Jinji Road, Guilin, 541004, Guangxi, China · Department of Electrical Engineering, University of Chile Av. Tupper 2007, Santiago, 8370451, Chile

  13. Towards Understanding Adam Convergence on Highly Degenerate Polynomials

    Mar 10, 2026Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou +2Optimization Convergence AnalysisAdaptive Gradient Methods

    Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Mathematical Sciences, Shanghai Jiao Tong University · Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China

  14. Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

    Mar 7, 2026Tao Shi, Liangming Chen, Long Jin +1Neural Network OptimizationAdaptive Gradient Methods

    School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

  15. Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

    Mar 3, 2026Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov +2Differentially Private Stochastic Gradient DescentDifferential Privacy

    University of Basel, Switzerland · University of Zürich, Switzerland

  16. Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

    Mar 3, 2026Ruinan Jin, Yingbin Liang, Shaofeng ZouStochastic Optimization ConvergenceAdaptive Gradient Methods

    Department of Electrical and Computer Engineering, The Ohio State University, USA · School of Electrical, Computer and Energy Engineering, Arizona State University, USA

  17. HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

    Mar 3, 2026Feihu Huang, Guanyi Zhang, Songcan ChenAdaptive Gradient MethodsNeural Network Generalization

    College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

  18. Adam Converges Without Any Modification On Update Rules

    Mar 2, 2026Yushun Zhang, Bingran Li, Congliang Chen +2Optimization Convergence AnalysisAdaptive Gradient Methods

    The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Research Institute of Big Data

  19. Never Saddle for Reparameterized Steepest Descent as Mirror Flow

    Mar 2, 2026Tom Jacobs, Chao Zhou, Rebekka BurkholzNeural Network Training DynamicsGradient Descent Dynamics

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  20. Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training

    Feb 27, 2026Yongzhong XuTransformer Training DynamicsNeural Network Training Dynamics

  21. Extending μμP: Spectral Conditions for Feature Learning Across Optimizers

    Feb 24, 2026Akshita Gupta, Marieme Ngom, Sam Foreman +1Hyperparameter TransferAdaptive Gradient Methods

    Purdue University · Argonne National Laboratory

  22. Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

    Feb 19, 2026Minxin Zhang, Yuxuan Liu, Hayden SchaefferDeep Learning OptimizationAdaptive Gradient Methods

    Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA

  23. Optimizer choice matters for the emergence of Neural Collapse

    Feb 18, 2026Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk +1Neural CollapseNeural Network Training Dynamics

    University of Basel · Warsaw University of Technology IDEAS Research Institute

  24. The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks

    Feb 18, 2026Eitan Gronich, Gal VardiGradient DescentAdaptive Gradient Methods

    Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel

  25. POP: Prior-Fitted First-Order Optimization Policies

    Feb 17, 2026Jan Kobiolka, Christian Frey, Gresa Shala +3Learning to OptimizeAdaptive Gradient Methods

    Department of Computer Science University of Technology Nuremberg Germany · Department of Computer Science Albert Ludwig University of Freiburg Germany · Faculty of Natural Sciences University of Elbasan Albania

  26. Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

    Jan 30, 2026Cheng Ge, Caitlyn Heqi Yin, Hao Liang +1RL for Language Model ReasoningGroup Relative Policy Optimization

    Department of Aeronautics and Astronautics, MIT · Department of Statistics, University of Wisconsin–Madison · Department of Informatics, King’s College London +1

  27. Why β1=β2β_1 = β_2 Is Dynamically Special in Adam

    Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum MethodsAdaptive Gradient Methods

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  28. ZENITH: Automated Gradient Norm Informed Stochastic Optimization

    Jan 21, 2026Dhrubo SahaDeep Learning OptimizationLearning Rate Scheduling

    Georgia Institute of Technology

  29. Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

    Nov 10, 2025Steffen Dereich, Thang Do, Arnulf Jentzen +1Stochastic OptimizationStochastic Optimization Convergence

    Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China +1

  30. Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

    Sep 22, 2025Doğay AltınelDeep Learning OptimizationNeural Network Optimization

    Istanbul Medeniyet University