Adaptive Gradient Methods

Follow this topic Ask about this topic

Adaptive gradient methods use information from current and past gradients to choose update sizes and directions, aiming to train more reliably than fixed-step gradient descent. Common approaches include momentum and learning-rate schedules, coordinate-wise methods such as Adam and AdaGrad, and block or matrix preconditioners such as Shampoo, often combined with orthogonalized updates. Researchers study their convergence and stability as well as their practical use in large language models, vision, scientific machine learning, and privacy-sensitive training. A central tension is that adaptivity can improve speed or robustness but costs memory and computation, and no optimizer consistently wins across tasks; guarantees and generalization depend on data, noise, geometry, and hyperparameter choices.

Latest papers

157 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

Apr 19, 2026

A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo

A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as adpative variants of Shampoo andMuon. This framework also allows combining heterogeneous geometriesacross different groups of variables while preserving a unifiedconvergence analysis. A fully stochastic global rate-of-convergenceanalysis is conducted for all methods in the framework, with andwithout two types of momentum, using reasonable assumptions on thevariance of the gradient oracle and without assuming boundedstochastic gradients or small enough stepsize.

S. Gratton, Ph. L. Toint

Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.

Momentum Methods Nonconvex Stochastic Optimization Stochastic Optimization Convergence Adaptive Gradient Methods

Apr 15, 2026

A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.

Damek Davis, Dmitriy Drusvyatskiy

Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104

Gradient Descent Adaptive Gradient Methods

Apr 14, 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Tong Zhang, Jiangning Zhang, Zhucun Xue +9

Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1

Deep Learning Optimization Adaptive Gradient Methods Stochastic Gradient Descent Second-Order Optimization Zeroth-Order Optimization

Apr 11, 2026

Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning

Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, the orthogonalization quality of Muon hinges on the number of Newton--Schulz (NS) iterations performed, which poses efficiency challenges due to its non-trivial computation and communication cost. We propose Muon

^2

, an extension of Muon, to improve both quality and efficiency by applying Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon

^2

, leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon

^2

demonstrates dramatic improvement over Muon at each polar step. Across GPT, LLaMA, and Mixture-of-Experts pre-training experiments up to 13B parameters, Muon

^2

(and its memory-efficient variant Muon

^2

-F that preserves most of its benefits) consistently outperforms Muon and its variants while reducing NS iterations by 40%, and saves up to 1/4 training time over Muon when achieving the same loss.

Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4

University of California at Santa Barbara · University at Albany, SUNY

Neural Network Optimization Muon Optimizer Adaptive Gradient Methods

Apr 10, 2026

Delve into the Applicability of Advanced Optimizers for Multi-Task Learning

Multi-Task Learning (MTL) is a foundational machine learning problem that has seen extensive development over the past decade. Recently, various optimization-based MTL approaches have been proposed to learn multiple tasks simultaneously by altering the optimization trajectory. Although these methods strive to de-conflict and re-balance tasks, we empirically identify that their effectiveness is often undermined by an overlooked factor when employing advanced optimizers: the instant-derived gradients play only a marginal role in the actual parameter updates. This discrepancy prevents MTL frameworks from fully releasing its power on learning dynamics. Furthermore, we observe that Muon-a recently emerged advanced optimizer-inherently functions as a multi-task learner, which underscores the critical importance of the gradients used for its orthogonalization. To address these issues, we propose APT (Applicability of advanced oPTimizers), a framework featuring a simple adaptive momentum mechanism designed to balance the strengths between advanced optimizers and MTL. Additionally, we introduce a light direction preservation method to facilitate Muon's orthogonalization. Extensive experiments across four mainstream MTL datasets demonstrate that APT consistently augments existing MTL approaches, yielding substantial performance improvements.

Zhipeng Zhou, Linxiao Cao, Pengcheng Wu +2

Nanyang Technological University · Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Shanghai Jiao Tong University

Orthogonal Optimization Multi-Task Learning Multi-Task Optimization Adaptive Gradient Methods

Apr 9, 2026

Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

Adam is widely used, but its convergence theory remains incomplete even in the deterministic full-batch setting because momentum and adaptive preconditioning are tightly coupled. For smooth convex objectives, we split the momentum variable through variable-and-operator splitting, which reveals the acceleration mechanism. We then combine a Hessian-driven correction with Adam-style feedback based on the gradient magnitude. The resulting Adam-HNAG (Hessian-driven Nesterov accelerated gradient with Adam-style adaptive preconditioning) flow admits a nonnegative energy that decays exponentially. Its discretization yields two methods, Adam-HNAG and the synchronous variant Adam-HNAG-s. Under the stated trajectory-bound and consistency conditions, both methods satisfy a discrete Lyapunov contraction. If the exact adaptive steps are accepted, this contraction gives an

O(k^{-2})

objective-value bound. Numerical experiments illustrate their behavior. These results apply to the proposed methods, not to the original Adam recursion.

Yaxin Yu, Long Chen, Zeyi Xu

School of Mathematics, Sichuan University, Chengdu, Sichuan, 610065, CHINA · Department of Mathematics, University of California, Irvine, Irvine, CA, 92697, USA

Accelerated Gradient Methods Adaptive Gradient Methods

Apr 8, 2026

FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection

Adaptive moment methods such as Adam use a diagonal, coordinate-wise preconditioner based on exponential moving averages of squared gradients. This diagonal scaling is coordinate-system dependent and can struggle with dense or rotated parameter couplings, including those in matrix factorization, tensor decomposition, and graph neural networks, because it treats each parameter independently. We introduce FlowAdam, a hybrid optimizer that augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE). When EMA-based statistics detect landscape difficulty, FlowAdam switches to clipped ODE integration. Our central contribution is Soft Momentum Injection, which blends ODE velocity with Adam's momentum during mode transitions. This prevents the training collapse observed with naive hybrid approaches. Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22% on low-rank matrix/tensor recovery and 6% on Jester (real-world collaborative filtering), also surpassing tuned Lion and AdaBelief, while matching Adam on well-conditioned workloads (CIFAR-10). MovieLens-100K confirms benefits arise specifically from coupled parameter interactions rather than bias estimation. Ablation studies show that soft injection is essential, as hard replacement reduces accuracy from 100% to 82.5%.

Devender Singh, Tarun Sheel

Department of Computer Science Memorial University of Newfoundland St. John’s, NL, Canada · Department of Mathematics and Statistics Memorial University of Newfoundland St. John’s, NL, Canada

Hypergradient Optimization Adaptive Gradient Methods Implicit Regularization

Apr 2, 2026

Optimal Projection-Free Adaptive SGD for Matrix Optimization

Recently, Jiang et al. [2026] developed Leon, a practical variant of One-sided Shampoo [Xie et al., 2025a, An et al., 2025] algorithm for online convex optimization, which does not require computing a costly quadratic projection at each iteration. Unfortunately, according to the existing analysis, Leon requires tuning an additional hyperparameter in its preconditioner and cannot achieve dimension-independent convergence guarantees for convex optimization problems beyond the bounded gradients assumption. In this paper, we resolve this issue by proving certain stability properties of Leon's preconditioner. Using our improved analysis, we show that tuning the extra hyperparameter can be avoided and, more importantly, develop the first practical variant of One-sided Shampoo with Nesterov acceleration, which does not require computing projections at each iteration. As a side contribution, we obtain improved dimension-independent rates in the non-smooth non-convex setting and develop a unified analysis of the proposed algorithm, which yields accelerated projection-free adaptive SGD with (block-)diagonal preconditioners.

Dmitry Kovalev

Yandex Research

Adaptive Gradient Methods Matrix Optimization Accelerated Gradient Methods Nonconvex Stochastic Optimization

Mar 19, 2026

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Steffen Dereich, Thang Do, Arnulf Jentzen

Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistic, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam +2

Stochastic Optimization Stochastic Optimization Convergence Adaptive Gradient Methods Stochastic Gradient Descent

Mar 16, 2026

Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a

Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The

delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) Capacity Collapse (

P < 23

), where rank-deficiency prevents structural learning; (2) The Variance-Limited Regime (

P \approx 41

), where generalization waits for the spectral gate to open; and (3) Stability Override (

P > 67

), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the anisotropic rectification unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.

Pratyush Acharya, Habish Dhakal

Grokking Neural Network Training Dynamics Neural Network Generalization Adaptive Gradient Methods Algorithmic Reasoning

Mar 10, 2026

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam. However, existing analyses of Adam-style methods often yield suboptimal guarantees in the zero-noise regime, rely on open-loop parameter schedules, or require prior knowledge of smoothness constants. Motivated by these limitations, we introduce OptEMA and analyze two complementary variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first moment with a fixed second-moment decay, and OptEMA-V, which swaps these roles. At the heart of these variants is a Corrected AdaGrad-Norm coefficient schedule. This formulation renders OptEMA algorithmically closed-loop and Lipschitz-free, meaning its effective stepsizes are trajectory-dependent and require no parameterization via the Lipschitz constant. Under lower-boundedness, unbiasedness, bounded variance, average smoothness, and a bounded stochastic-gradient condition used to control the adaptive normalizers, we prove that both variants achieve the unified noise-adaptive rate

\tilde{\mathcal{O}} \left(T^{-1/2}+σ^{1/2}T^{-1/4}\right)

for the averaged gradient norm. In the zero-noise regime, these bounds automatically reduce to the nearly optimal deterministic rate

\widetilde{\mathcal{O}}(T^{-1/2})

without manual hyperparameter retuning.

Ganzhao Yuan

Shenzhen University of Advanced Technology (SUAT), China

Stochastic Optimization Adaptive Gradient Methods Nonconvex Stochastic Optimization

Mar 10, 2026

Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts

Time-series forecasting often faces challenges from non-stationarity, particularly distributional drift, where the data distribution evolves over time. This dynamic behavior can undermine the effectiveness of adaptive optimizers, such as Adam, which are typically designed for stationary objectives. In this paper, we revisit Adam in the context of non-stationary forecasting and identify that its second-order bias correction limits responsiveness to shifting loss landscapes. To address this, we propose TS_Adam, a lightweight variant that removes the second-order correction from the learning rate computation. This simple modification improves adaptability to distributional drift while preserving the optimizer core structure and requiring no additional hyperparameters. TS_Adam integrates easily into existing models and consistently improves performance across long- and short-term forecasting tasks. On the ETT datasets with the MICN model, it achieves an average reduction of 12.8% in MSE and 5.7% in MAE compared to Adam. These results underscore the practicality and versatility of TS_Adam as an effective optimization strategy for real-world forecasting scenarios involving non-stationary data. Code is available at: https://github.com/DD-459-1/TS_Adam.

Yuze Dong, Jinsong Wu

School of Artificial Intelligence, Guilin University of Electronic Technology 1 Jinji Road, Guilin, 541004, Guangxi, China · Department of Electrical Engineering, University of Chile Av. Tupper 2007, Santiago, 8370451, Chile

Adaptive Gradient Methods Non-Stationary Time Series Forecasting Distribution Shift Robustness Long-Term Time Series Forecasting

Mar 10, 2026

Towards Understanding Adam Convergence on Highly Degenerate Polynomials

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and

β_2

near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment

v_t

and squared gradient

g_t^2

, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou +2

Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Mathematical Sciences, Shanghai Jiao Tong University · Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China

Optimization Convergence Analysis Adaptive Gradient Methods

Mar 7, 2026

Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance. A widely accepted explanation for its defect in generalization is that it often tends to converge to sharp minima. To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam). The key improvement of InvAdam lies in its parameter update mechanism, which is opposite to that of Adam. Specifically, it computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments. This modification aims to increase the step size of the parameter update when the elements in the second-order moments are large and vice versa, which helps the parameter escape sharp minima and stay at flat ones. However, InvAdam's update mechanism may face challenges in convergence. To address this challenge, we propose dual Adam (DualAdam), which integrates the update mechanisms of both Adam and InvAdam, ensuring convergence while enhancing generalization performance. Additionally, we introduce the diffusion theory to mathematically demonstrate InvAdam's ability to escape sharp minima. Extensive experiments are conducted on image classification tasks and large language model (LLM) fine-tuning. The results validate that DualAdam outperforms Adam and its state-of-the-art variants in terms of generalization performance. The code is publicly available at https://github.com/LongJin-lab/DualAdam.

Tao Shi, Liangming Chen, Long Jin +1

School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

Neural Network Optimization Adaptive Gradient Methods Neural Network Generalization

Mar 3, 2026

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with adaptivity in optimization through the lens of stochastic differential equations, providing the first SDE-based analysis of private optimizers. Focusing on DP-SGD and DP-SignSGD under per-example clipping, we show a sharp contrast under fixed hyperparameters: DP-SGD converges at a Privacy-Utility Trade-Off of

\mathcal{O}(1/\varepsilon^2)

with speed independent of

\varepsilon

, while DP-SignSGD converges at a speed linear in

\varepsilon

with an

\mathcal{O}(1/\varepsilon)

trade-off, dominating in high-privacy or large batch noise regimes. By contrast, under optimal learning rates, both methods achieve comparable theoretical asymptotic performance; however, the optimal learning rate of DP-SGD scales linearly with

\varepsilon

, while that of DP-SignSGD is essentially

\varepsilon

-independent. This makes adaptive methods far more practical, as their hyperparameters transfer across privacy levels with little or no re-tuning. Empirical results confirm our theory across training and test metrics, and empirically extend from DP-SignSGD to DP-Adam.

Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov +2

University of Basel, Switzerland · University of Zürich, Switzerland

Differentially Private Stochastic Gradient Descent Differential Privacy Adaptive Gradient Methods

Mar 3, 2026

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In this paper, we uncover a key second-moment normalization in Adam and develop a stopping-time/martingale analysis that provably distinguishes Adam from SGD under the classical bounded variance model (a second moment assumption). In particular, we establish the first theoretical separation between the high-probability convergence behaviors of the two methods: Adam achieves a

δ^{-1/2}

dependence on the confidence parameter

δ

, whereas corresponding high-probability guarantee for SGD necessarily incurs at least a

δ^{-1}

dependence.

Ruinan Jin, Yingbin Liang, Shaofeng Zou

Department of Electrical and Computer Engineering, The Ohio State University, USA · School of Electrical, Computer and Energy Engineering, Arizona State University, USA

Stochastic Optimization Convergence Adaptive Gradient Methods Stochastic Gradient Descent

Mar 3, 2026

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error

O(\frac{1}{\sqrt{N}})

also is larger than

O(\frac{1}{N})

of SGD, where

N

denotes training sample size. Recently, although some variants of Adam have been proposed to improve its generalization, their improved generalizations are still unexplored in theory. To fill this gap, in the paper, we restudy generalization of Adam and AdamW via algorithmic stability, and first prove that Adam and AdamW without square-root (i.e., Adam(W)-srf) have a generalization error

O(\frac{\hatρ^{-2T}}{N})

, where

T

denotes iteration number and

\hatρ>0

denotes the smallest element of second-order momentum plus a small positive number. To improve generalization, we propose a class of efficient clever Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD. Moreover, we prove that our HomeAdam(W) have a smaller generalization error

O(\frac{1}{N})

than

O(\frac{\hatρ^{-2T}}{N})

of Adam(W)-srf, since

\hatρ

is generally very small. In particular, it is also smaller than the existing

O(\frac{1}{\sqrt{N}})

of Adam(W). Meanwhile, we prove our HomeAdam(W) have a faster convergence rate of

O(\frac{1}{T^{1/4}})

than

O(\frac{\breveρ^{-1}}{T^{1/4}})

of the Adam(W)-srf, where

\breveρ\leq\hatρ

also is very small. Extensive numerical experiments demonstrate efficiency of our HomeAdam(W) algorithms.

Feihu Huang, Guanyi Zhang, Songcan Chen

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

Adaptive Gradient Methods Neural Network Generalization Stochastic Optimization Convergence

Mar 2, 2026

Adam Converges Without Any Modification On Update Rules

Adam is the default algorithm for training neural networks, including large language models (LLMs). However, \citet{reddi2019convergence} provided an example that Adam diverges, raising concerns for its deployment in AI model training. We identify a key mismatch between the divergence example and practice: \citet{reddi2019convergence} pick the problem after picking the hyperparameters of Adam, i.e.,

(β_1,β_2)

; while practical applications often fix the problem first and then tune

(β_1,β_2)

. In this work, we prove that Adam converges with proper problem-dependent hyperparameters. First, we prove that Adam converges when

β_2

is large and

β_1 < \sqrt{β_2}

. Second, when

β_2

is small, we point out a region of

(β_1,β_2)

combinations where Adam can diverge to infinity. Our results indicate a phase transition for Adam from divergence to convergence when changing the

(β_1, β_2)

combination. To our knowledge, this is the first phase transition in

(β_1,β_2)

2D-plane reported in the literature, providing rigorous theoretical guarantees for Adam optimizer. We further point out that the critical boundary

(β_1^*, β_2^*)

is problem-dependent, and particularly, dependent on batch size. This provides suggestions on how to tune

β_1

and

β_2

: when Adam does not work well, we suggest tuning up

β_2

inversely with batch size to surpass the threshold

β_2^*

, and then trying

β_1< \sqrt{β_2}

. Our suggestions are supported by reports from several empirical studies, which observe improved LLM training performance when applying them.

Yushun Zhang, Bingran Li, Congliang Chen +2

The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Research Institute of Big Data

Optimization Convergence Analysis Adaptive Gradient Methods Hyperparameter Optimization

Mar 2, 2026

Never Saddle for Reparameterized Steepest Descent as Mirror Flow

How does the choice of optimization algorithm shape a model's ability to learn features? To address this question for steepest descent methods --including sign descent, which is closely related to Adam --we introduce steepest mirror flows as a unifying theoretical framework. This framework reveals how optimization geometry governs learning dynamics, implicit bias, and sparsity and it provides two explanations for why Adam and AdamW often outperform SGD in fine-tuning. Focusing on diagonal linear networks and deep diagonal linear reparameterizations (a simplified proxy for attention), we show that steeper descent facilitates both saddle-point escape and feature learning. In contrast, gradient descent requires unrealistically large learning rates to escape saddles, an uncommon regime in fine-tuning. Empirically, we confirm that saddle-point escape is a central challenge in fine-tuning. Furthermore, we demonstrate that decoupled weight decay, as in AdamW, stabilizes feature learning by enforcing novel balance equations. Together, these results highlight two mechanisms how steepest descent can aid modern optimization.

Tom Jacobs, Chao Zhou, Rebekka Burkholz

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Neural Network Training Dynamics Gradient Descent Dynamics Adaptive Gradient Methods Gradient Descent Mirror Descent Representation Learning

Feb 27, 2026

Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training

We analyze cumulative parameter trajectories of transformer training under AdamW and identify a dominant low-dimensional drift direction ("backbone") that captures 60--80% of long-horizon displacement from initialization. This direction is highly stable over rolling training windows yet reorients gradually across phases, particularly following objective reweighting. Per-batch gradients exhibit near-noise-floor alignment with the backbone, whereas optimizer-integrated updates align strongly with it, indicating that the structure emerges from accumulated optimizer dynamics rather than instantaneous gradient geometry. Replacing AdamW with SGD-family optimizers eliminates this structure, and reducing

β_2

smoothly degrades backbone dominance and reheating recoverability. Reheating experiments show that transverse probe modes can be transiently re-excited without substantially altering accumulated backbone drift. These results provide a trajectory-level characterization of optimizer-induced geometric structure in transformer training and shift attention from instantaneous gradient properties to cumulative update dynamics.

Yongzhong Xu

Transformer Training Dynamics Neural Network Training Dynamics Transformer Adaptive Gradient Methods Intrinsic Dimensionality

Feb 24, 2026

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization

(μ

P

)

is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving

μ

P for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive

μ

P for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our

μ

P derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.

Akshita Gupta, Marieme Ngom, Sam Foreman +1

Purdue University · Argonne National Laboratory

Hyperparameter Transfer Adaptive Gradient Methods

Feb 19, 2026

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers' matrix structure via orthogonalized momentum, showing superior performance in large language model training. We propose a new optimizer and a diagonal extension, NAMO and NAMO-D, providing the first principled integration of orthogonalized momentum with norm-based Adam-type noise adaptation. NAMO scales orthogonalized momentum using a single adaptive stepsize, preserving orthogonality while improving upon Muon at negligible additional cost. NAMO-D instead right-multiplies orthogonalized momentum by a diagonal matrix with clamped entries. This design enables neuron-wise noise adaptation and aligns with the common near block-diagonal Hessian structure. Under standard assumptions, we establish optimal convergence rates for both algorithms in the deterministic setting and show that, in the stochastic setting, their convergence guarantees adapt to the noise level of stochastic gradients. Experiments on pretraining GPT-2 models demonstrate improved performance of both NAMO and NAMO-D compared to the AdamW and Muon baselines, with NAMO-D achieving further gains over NAMO via an additional clamping hyperparameter that balances the competing goals of maintaining a well-conditioned update direction and leveraging fine-grained noise adaptation.

Minxin Zhang, Yuxuan Liu, Hayden Schaeffer

Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA

Deep Learning Optimization Adaptive Gradient Methods Muon Optimizer Stochastic Optimization Convergence

Feb 18, 2026

Optimizer choice matters for the emergence of Neural Collapse

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.

Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk +1

University of Basel · Warsaw University of Technology IDEAS Research Institute

Neural Collapse Neural Network Training Dynamics Adaptive Gradient Methods Weight Decay Stochastic Gradient Descent

Feb 18, 2026

The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks

We study the implicit bias of momentum-based optimizers on smooth homogeneous models. We show that momentum steepest descent algorithms like Muon (spectral norm), MomentumGD (

\ell_2

norm), and Signum (

\ell_\infty

norm) are approximate steepest descent trajectories under a decaying learning rate schedule, proving that these algorithms have a bias towards KKT points of the corresponding margin maximization problem. We extend the analysis to Adam (without the stability constant), which maximizes the

\ell_\infty

margin, and to Muon-Signum and Muon-Adam, which maximize a hybrid norm. Our experiments corroborate the theory and show that the identity of the margin maximized depends on the choice of optimizer. Overall, our results extend earlier lines of work on steepest descent in homogeneous models and momentum-based optimizers in linear models.

Eitan Gronich, Gal Vardi

Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel

Gradient Descent Adaptive Gradient Methods Algorithmic Bias

Feb 17, 2026

POP: Prior-Fitted First-Order Optimization Policies

Gradient-based optimizers are highly sensitive to design choices in their adaptive learning rate mechanisms. To address this limitation, we introduce POP, a meta-learned Reinforcement Learning (RL) policy that predicts adaptive learning rates for gradient descent, conditioned on the contextual information provided in the optimization trajectory. Our method introduces a novel RL reward formulation, a new function-scaling strategy for in-distribution generalization, and a novel prior that is used to sample millions of synthetic optimization problems. We evaluate POP on an established benchmark including 43 optimization functions of various complexity, where it significantly outperforms gradient-based methods. Our evaluation demonstrates strong generalization capabilities without task-specific tuning.

Jan Kobiolka, Christian Frey, Gresa Shala +3

Department of Computer Science University of Technology Nuremberg Germany · Department of Computer Science Albert Ludwig University of Freiburg Germany · Faculty of Natural Sciences University of Elbasan Albania

Learning to Optimize Adaptive Gradient Methods Policy Learning

Jan 30, 2026

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we answer both questions through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, we prove that under mild conditions GRPO attains an improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. We further introduce IS-GRPO, an importance-sampling variant of GRPO whose expected update remains aligned with the full gradient, and prove a convergence guarantee of the same form as REINFORCE's, which is tighter in practice during real training runs by a factor we measure. Empirically, on GSM8K and MATH we validate the curvature--variance link and measure the quantities that govern our bounds along real training runs. At 1.5B scale, per-prompt normalization outperforms both unnormalized and globally normalized baselines, with gains that emerge in a middle phase of training where per-prompt variances become heterogeneous. At 7B scale, the normalization schemes become statistically indistinguishable while IS-GRPO retains a small lead, in line with the theory's prediction that the gains require heterogeneity.

Cheng Ge, Caitlyn Heqi Yin, Hao Liang +1

Department of Aeronautics and Astronautics, MIT · Department of Statistics, University of Wisconsin–Madison · Department of Informatics, King’s College London +1

RL for Language Model Reasoning Group Relative Policy Optimization Adaptive Gradient Methods

Jan 29, 2026

Why $β_1 = β_2$ Is Dynamically Special in Adam

Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying

β_{1}=β_{2}

can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when

β_{1}=β_{2}

, making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Momentum Methods Adaptive Gradient Methods Neural Network Training Dynamics Gradient Descent Dynamics

Jan 21, 2026

ZENITH: Automated Gradient Norm Informed Stochastic Optimization

Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.

Dhrubo Saha

Georgia Institute of Technology

Deep Learning Optimization Learning Rate Scheduling Adaptive Gradient Methods Stochastic Gradient Descent

Nov 10, 2025

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training of deep neural networks in artificial intelligence (AI) systems. Despite the popularity and the success of Adam it remains an open research problem to provide a rigorous convergence analysis for Adam even for the class of strongly convex SOPs. In one of the main results of this work we establish convergence rates for Adam in terms of the number of gradient steps (convergence rate \nicefrac{1}{2} w.r.t. the size of the learning rate), the size of the mini-batches (convergence rate 1 w.r.t. the size of the mini-batches), and the size of the second moment parameter of Adam (convergence rate 1 w.r.t. the distance of the second moment parameter to 1) for the class of strongly convex SOPs. In a further main result of this work, which we refer to as Adam symmetry theorem, we illustrate the optimality of the established convergence rates by proving for a special class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity to the solution of the SOP (the unique minimizer of the strongly convex objective function) if and only if the random variables in the SOP (the data in the SOP) are symmetrically distributed. In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we disprove that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity. We also complement the conclusions of our convergence analysis and the Adam symmetry theorem by several numerical simulations that indicate the sharpness of the established convergence rates and that illustrate the practical appearance of the phenomena revealed in the Adam symmetry theorem.

Steffen Dereich, Thang Do, Arnulf Jentzen +1

Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China +1

Stochastic Optimization Stochastic Optimization Convergence Adaptive Gradient Methods

Sep 22, 2025

Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.

Doğay Altınel

Istanbul Medeniyet University

Deep Learning Optimization Neural Network Optimization Adaptive Gradient Methods Adaptive Learning Rate Stochastic Gradient Descent