math.OCNov 16, 2026

On the Convergence of Continuous-Time AdamW

Authors: Bangjie Wang, Ning Ning

Organizations: Department of Mathematics, Texas A&M University · Department of Statistics, Texas A&M University

Abstract

Continuous-time AdamW provides a framework for understanding whether an optimizer widely used to train large-scale neural networks settles to an equilibrium or sustains persistent motion. We study this long-time behavior while retaining bias correction and fixed positive weight decay. Under suitable assumptions, we establish that the dynamics are well defined for all time and remain bounded when the decay is sufficiently small. Our main result connects their limiting behavior to the geometry of the objective: trajectories approach a locally invariant manifold whose dimension equals the Hessian nullity along the associated critical component. When critical points are isolated and have at most one such direction, the trajectory converges to a single equilibrium. With two degenerate directions, however, bounded persistent motion can occur even for a smooth, strictly convex objective with a unique minimum. We also quantify how close the long-time trajectory remains to the critical set as weight decay decreases, without assuming convergence to a point. These results identify objective geometry as a key to understanding when continuous-time AdamW converges and when persistent motion is possible.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

    Apr 9, 2026Yaxin Yu, Long Chen, Zeyi XuSmooth Convex OptimizationAccelerated Gradient Methods

  2. On the O(dK1/4)O(\frac{\sqrt{d}}{K^{1/4}}) Convergence Rate of AdamW Measured by ℓ1\ell_1 Norm

    May 17, 2025Huan Li, Yiming Dong, Zhouchen LinOptimization Convergence AnalysisStochastic Optimization Convergence