Adaptive Gradient Methods

Ask about this topic

Overview

Adaptive gradient methods use information from current and past gradients to choose update sizes and directions, aiming to train more reliably than fixed-step gradient descent. Common approaches include momentum and learning-rate schedules, coordinate-wise methods such as Adam and AdaGrad, and block or matrix preconditioners such as Shampoo, often combined with orthogonalized updates. Researchers study their convergence and stability as well as their practical use in large language models, vision, scientific machine learning, and privacy-sensitive training. A central tension is that adaptivity can improve speed or robustness but costs memory and computation, and no optimizer consistently wins across tasks; guarantees and generalization depend on data, noise, geometry, and hyperparameter choices.

Latest papers

CardsList

157 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization

    Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning OptimizationNeural Network Optimization

    1Vector Institute, Toronto, Canada · University of Cambridge, Cambridge, United Kingdom · 3Microsoft, United States +1

  2. On the O(dK1/4)O(\frac{\sqrt{d}}{K^{1/4}}) Convergence Rate of AdamW Measured by ℓ1\ell_1 Norm

    May 17, 2025Huan Li, Yiming Dong, Zhouchen LinOptimization Convergence AnalysisStochastic Optimization Convergence

    Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, China. · National Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China.

  3. Correcting Before Adapting: Gradient Correction for Federated Adaptive Optimization

    Feb 4, 2025Evan Chen, Shiqiang Wang, Dong-Jun Han +2Federated LearningHeterogeneous Federated Learning

    Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN · IBM T. J. Watson Research Center, Yorktown Heights, NY, USA · Department of Computer Science and Engineering, Yonsei University, Seoul, Korea

  4. BEAT: Balanced Frequency Adaptive Tuning for Long-Term Time-Series Forecasting

    Jan 31, 2025Zhixuan Li, Naipeng Chen, Seonghwa Choi +2Long-Term Time Series ForecastingFrequency-Domain Time Series Modeling

    College of Computing and Data Science, Nanyang Technological University, Singapore · Department of Electrical and Electronic Engineering, Yonsei University, Korea

  5. Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

    Jun 20, 2024Steffen Dereich, Arnulf Jentzen, Adrian RiekertAdaptive Learning RateStochastic Optimization Convergence

    Institute for Mathematical Stochastics, University of Münster, Germany · School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany

  6. Generalizing Adam to Manifolds for Efficiently Training Transformers

    May 26, 2023Benedikt BrantnerStiefel Manifold OptimizationRiemannian Optimization

    Max-Planck-Institut für Plasmaphysik Boltzmannstraße 2, 85748 Garching, Deutschland · Technische Universität München, Zentrum Mathematik Boltzmannstraße 3, 85748 Garching, Deutschland

  7. Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant

    Jan 26, 2022John ChiangLogistic RegressionGradient Descent