Overview

Adaptive gradient methods use information from current and past gradients to choose update sizes and directions, aiming to train more reliably than fixed-step gradient descent. Common approaches include momentum and learning-rate schedules, coordinate-wise methods such as Adam and AdaGrad, and block or matrix preconditioners such as Shampoo, often combined with orthogonalized updates. Researchers study their convergence and stability as well as their practical use in large language models, vision, scientific machine learning, and privacy-sensitive training. A central tension is that adaptivity can improve speed or robustness but costs memory and computation, and no optimizer consistently wins across tasks; guarantees and generalization depend on data, noise, geometry, and hyperparameter choices.

Latest papers

CardsList

180 papers20 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

  1. On the Convergence of Continuous-Time AdamW

    Nov 16, 2026Bangjie Wang, Ning NingOptimization Convergence AnalysisAdaptive Gradient Methods

  2. Near-Optimal Methods for Generalized Lipschitz Convex Optimization

    Nov 15, 2026Dmitry Kovalev, Grigory MalinovskyNonsmooth Convex OptimizationStochastic Optimization Convergence

  3. High-dimensional limit theorems for adaptive variants of SGD I: Ballistic limits

    Nov 14, 2026Aukosh Jagannath, Varnan SarangianSGD with MomentumAdaptive Gradient Methods

  4. Orthogonal Updates for the Win: Towards Accelerated Adaptive Minimax Optimization

    Nov 8, 2026Zhiwei Zhai, Xinyu Wang, Wenjing Yan +2Min-Max OptimizationAdaptive Gradient Methods

  5. Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning

    Nov 8, 2026Yaxin Yu, Long Chen, Minfu FengAccelerated Gradient MethodsSmooth Convex Optimization

  6. Full Fine-Tuning is not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation

    Nov 8, 2026Nairouz Mrabah, Ismail Ben Ayed, Eric GrangerFew-Shot VLM AdaptationAdaptive Gradient Methods

  7. Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

    Nov 6, 2026Jeonseong Kim, Chenglin FanStochastic Optimization ConvergenceAdaptive Gradient Methods

  8. Task-Contribution Dynamics of Adam in Physics-Informed Neural Networks: Slow-Manifold Analysis and Gradient Modulation

    Nov 6, 2026Enzhan Zheng, Xiaoliang SongNeural Network Training DynamicsAdaptive Gradient Methods

  9. ZEMA: confidence-calibrated signed descent with only one state

    Nov 3, 2026Ali Dadsetan, Domenic Rosati, Frank RudziczAdaptive Gradient MethodsSign-Based Optimization

  10. Non-Interfering Asymmetric AdamW Updates for Constrained Exploration of Equifinality in Hybrid Models

    Nov 3, 2026Ekaterina Gikalo, Zavud Baghirov, Christian Reimers +3Neural Network OptimizationAdaptive Gradient Methods

  11. Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy

    Oct 31, 2026Sunyoung An, Xiaoming HuoTemporal-Difference LearningPolicy Evaluation

  12. Adaptive Federated Optimization Across Smooth and Nonsmooth Regimes

    Oct 29, 2026Zimeng Wang, Ali Dadras, Alp YurtseverFederated LearningAdaptive Gradient Methods

  13. Efficient Preconditioned Optimization Methods with Frequent Directions

    Oct 12, 2026Sifan Yang, Yuanyu Wan, Yibo Wang +1Adaptive Gradient MethodsOnline Convex Optimization

  14. Adaptive Variance Reduction across Smoothness and Noise Conditions

    Oct 9, 2026Wei Jiang, Yibo Wang, Sifan Yang +2Nonconvex Stochastic OptimizationVariance Reduction

  15. From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

    Oct 8, 2026Jihwan Kim, Dogyoon Song, Chulhee YunAdaptive Gradient MethodsClassification

  16. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient MethodsHyperparameter Optimization

  17. Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

    Oct 3, 2026Mahalakshmi Sabanayagam, Simon LuceyGradient Descent DynamicsSpectral Bias