A Stability Analysis of AdamW: Unstable Equilibria and Non-Convergence
Organizations: The University of Texas at Austin
Abstract
AdamW is ubiquitous in deep learning, yet its behavior remains poorly understood. We analyze its dynamics through the lens of dynamical systems and show that AdamW admits an implicit fixed-point objective: its fixed points coincide with the stationary points of a constrained and regularized optimization problem. However, not all of these fixed points are stable under AdamW's dynamics, and stability depends sensitively on curvature, weight decay, and momentum parameters. Even in one-dimensional settings, equilibria may be unstable, and trajectories can exhibit persistent oscillations rather than convergence to them. We further extend the analysis to higher dimensions, deriving sufficient conditions for local stability of the continuous-time dynamics, and propose a convergent modification that preserves AdamW's fixed points. These results clarify what optimization problem AdamW is associated with, when its convergence can be expected, and how its dynamics could inspire more reliable optimizers.
Figures & tables
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Crossing | Perturbation if violated | ODE limit ( ) |
|---|---|---|---|
| Implicit-minimum floor ( 19 ) | real, at | grows monotonically | |
| Moment-mismatch bound ( 20 ) | complex pair | grows while oscillating | condition ( 8 ) |
| Step-flip ceiling ( 21 ) | real, at | flips sign every step | vacuous |
Adaptive Gradient Methods
Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.
See all 180 papers