Adaptive Gradient Methods

Ask about this topic

Overview

Adaptive gradient methods use information from current and past gradients to choose update sizes and directions, aiming to train more reliably than fixed-step gradient descent. Common approaches include momentum and learning-rate schedules, coordinate-wise methods such as Adam and AdaGrad, and block or matrix preconditioners such as Shampoo, often combined with orthogonalized updates. Researchers study their convergence and stability as well as their practical use in large language models, vision, scientific machine learning, and privacy-sensitive training. A central tension is that adaptivity can improve speed or robustness but costs memory and computation, and no optimizer consistently wins across tasks; guarantees and generalization depend on data, noise, geometry, and hyperparameter choices.

Latest papers

CardsList

157 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. A Statistical Viewpoint on Adam, Shampoo and Muon

    Oct 17, 2026Tetiana Parshakova, Robert M. Gower, Paulo OrensteinStochastic OptimizationAdaptive Gradient Methods

  2. Efficient Preconditioned Optimization Methods with Frequent Directions

    Oct 12, 2026Sifan Yang, Yuanyu Wan, Yibo Wang +1Adaptive Gradient MethodsOnline Convex Optimization

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Software Technology, Zhejiang University, Ningbo, China +1

  3. Adaptive Variance Reduction without Knowing Oracle Smoothness

    Oct 9, 2026Wei Jiang, Yibo Wang, Zechao LiNonconvex Stochastic OptimizationVariance Reduction

    School of Computer Science and Engineering, Nanjing University of Science and Technology · National Key Laboratory for Novel Software Technology, Nanjing University

  4. Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

    Oct 9, 2026Ran Tian, Courtney PaquetteRandom Feature MethodsNeural Network Training Dynamics

    Google Deepmind

  5. From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

    Oct 8, 2026Jihwan Kim, Dogyoon Song, Chulhee YunAdaptive Gradient MethodsClassification

    Seoul National University · KAIST InnoCORE LLM · University of California, Davis +1

  6. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient MethodsHyperparameter Optimization

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  7. DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon

    Oct 5, 2026Wenpeng Zhang, Runsheng YuMuon OptimizerAdaptive Gradient Methods

    Independent Reasearcher

  8. Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

    Oct 3, 2026Mahalakshmi Sabanayagam, Simon LuceyGradient Descent DynamicsSpectral Bias

    Australian Institute for Machine Learning, Adelaide University

  9. Exact information accounting for SGD methods

    Sep 30, 2026Akshay BalsubramaniNeural Network GeneralizationStochastic Optimization Convergence

  10. Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

    Sep 29, 2026Ruinan Jin, Difei Cheng, Ling Chen +3Convergence AnalysisStochastic Optimization Convergence

    Mohamed bin Zayed University of Artificial Intelligence · Aerospace Information Technology University · The Ohio State University +2

  11. Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

    Sep 29, 2026Hyunseok Lee, Mihir Basil, Yizhou Liu +1Neural Network Training DynamicsNeural Scaling Laws

    Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · Department of Physics, University of Pennsylvania, Philadelphia, PA

  12. Why Adaptive Optimizers Underestimate Rare Tokens

    Sep 29, 2026Sangsidhya KarAdaptive Gradient MethodsSign-Based Optimization

    Department of Statistics Presidency University, Kolkata

  13. Second-Moment Stochastic Approximation Methods

    Sep 29, 2026Tao Jiang, Lin XiaoStochastic ApproximationMuon Optimizer

    Fundamental AI Research (FAIR), Meta Superintelligence Labs

  14. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic OptimizationSign-Based Optimization

    Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  15. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriImplicit BiasClassification

    University of Alberta

  16. Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Bounds

    Sep 26, 2026Weiming Ou, Xiao WangConvergence AnalysisAdaptive Gradient Methods

    Shanghai University of Finance and Economics · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics

  17. Arithmetic Simplicity in Stochastic Gradient Methods

    Sep 26, 2026Bin Fu, Pengfei Gu, Jose Nunez +1Adaptive Gradient MethodsStochastic Optimization Convergence

    Department of Computer Science University of Texas Rio Grande Valley Edinburg, TX 78539, USA

  18. VCMM: Variance-Calibrated Momentum for Multimodal Learning

    Sep 23, 2026Zhongjing Gu, Chenyang Huang, Yufa Feng +3Multimodal OptimizationAdaptive Gradient Methods

  19. AURA: Angular Update Rate Adaptation for training complex-valued neural networks

    Sep 22, 2026Enrico Ballini, Allan Peter Engsig-Karup, Tito AndriolloNeural Network OptimizationComplex-Valued Neural Networks

    Aarhus University, Aarhus, Denmark. · Technical University of Denmark, Kongens Lyngby, Denmark.

  20. Adaptive Forgetting for Nonstationary Optimization: Towards Robust EEG Decoding

    Sep 21, 2026Hongyu Zhu, Lin Chen, Jing Chen +2Cross-Subject EEG DecodingAdaptive Gradient Methods

    Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · School of Resources and Environment, University of Electronic Science and Technology of China

  21. Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods

    Sep 18, 2026Wenpeng Zhang, Runsheng Yu, Peilin ZhaoAdaptive Gradient MethodsNeural Network Optimization

    Independent Reasearcher · School of Artificial Intelligence Shanghai Jiao Tong University

  22. Beyond Quadratic Loss: The Stability Phase Diagram of Adam

    Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

  23. A Full Adam Theorem for Spectral Heavy-Tail Onset

    Sep 14, 2026Zongmin LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    Stanford University

  24. A pullback-corrected scalar auxiliary variable optimizer with momentum and adaptive mobility

    Sep 11, 2026Jiahao Zhang, Shiheng Zhang, Guang LinNeural Network OptimizationAdaptive Gradient Methods

    School of Mechanical Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Mathematics, University of Washington, Seattle, WA 98195, USA · Department of Mathematics, Purdue University, West Lafayette, IN 47906, USA

  25. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresDeep Learning OptimizationAdaptive Gradient Methods

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  26. When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

    Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate SchedulingWeight Decay

    Purdue University

  27. Equivariance Breaks the Learning Rate

    Sep 8, 2026Andrei Manolache, Mathias NiepertEquivariant Neural NetworksEquivariant Representation Learning

    University of Stuttgart, Germany · Bitdefender, Romania

  28. Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

    Sep 2, 2026Sai Niranjan Ramachandran, Suvrit SraPhase Transitions in Neural NetworksStochastic Gradient Descent

    School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)

  29. Convergence rates for the RMSprop optimizer with full control of the hyperparameters

    Aug 31, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationStochastic Optimization Convergence

  30. SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

    Aug 29, 2026Guangyuan Wang, Mads Toftrup, Sebastian Loeschcke +2Neural Network OptimizationPhysics-Informed ML

    McGill University · California Institute of Technology · Aarhus University +1