Towards Understanding Adam Convergence on Highly Degenerate Polynomials

Zhiwei Bai; Jiajie Zhao; Zhangchen Zhou; Zhi-Qin John Xu; Yaoyu Zhang

cs.LGMar 10, 2026

Towards Understanding Adam Convergence on Highly Degenerate Polynomials

Authors: Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou, Zhi-Qin John Xu, Yaoyu Zhang

Organizations: Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Mathematical Sciences, Shanghai Jiao Tong University · Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China

Abstract

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$ , which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Figures & tables

Panel of Figure 1 — Figure 1: Convergence behavior differs between strongly convex and degenerate polynomials. Adam uses $\beta_{1}=0.9,\beta_{2}=0.99$ .

Work	Proposed Mechanism	Key Assumption	Requires Stochasticity
Zhang et al. (2020)	Heavy-tailed noise	Heavy-tailed gradient noise	Yes
Zhang et al. (2024)	Hessian block heterogeneity	Block-diagonal Hessian structure	No
Tomihari & Sato (2025)	Gradient heterogeneity	Heterogeneous gradients and SignGD approx.	No
Xie et al. (2024)	$\ell_{\infty}$ geometry exploitation	Coordinate-wisely smooth w.r.t. $\ell_{\infty}$ norm	No
Kunstner et al. (2024)	Heavy-tailed class imbalance	Imbalanced data distribution	No
Ours	$v_{t}$ - $g_{t}^{2}$ decoupling	Loss landscape degeneracy	No

Table 1: Comparison of theoretical mechanisms explaining Adam’s advantages over SGD. While prior works often rely on specific assumptions regarding noise, geometry, or data distribution, our proposed mechanism operates solely on loss landscape degeneracy—a ubiquitous structural property in deep neural networks that does not require stochasticity.

Panel of Figure 2 — Figure 2: (a) Training loss of Adam ( $\beta_{1}=0.9,\beta_{2}=0.93$ ) for different $L(x)$ . The text online shows the slope of the exponential fit. (b) Evolution of effective curvature $\lambda_{t}$ . The red shaded region highlights instability where $\lambda_{t}$ exceeds the theoretical stability threshold $\frac{2}{\eta}\frac{1+\beta_{1}}{1-\beta_{1}}$ ( Cohen et al., 2023 ; Bai et al., 2025 ) .

Panel of Figure 3 — Figure 3: (a) Theoretical convergence phase diagram for $k=4$ , partitioned according to the stability conditions derived in Eq. ( 8 ) and Eq. ( 9 ). (b) Experimental validation using Adam on $L(x)=\frac{1}{4}x^{4}$ with $x_{0}=1.0,\eta=0.001$ . The heatmap displays the final training loss after $100,000$ steps. Additional results for $k=6$ can be found in Fig. 11 . For a detailed discussion and a magnified view of the lower-left corner region, please refer to Appendix G.1 .

Figure 4: Bifurcation diagram of effective sharpness $u_{t}$ for $k=4$ . The system Eq: ( 18 ) exhibits three regimes: (I) Stable Fixed Point ( $u_{t}\to u^{*}$ ), leading to exponential convergence of $x_{t}$ with a fixed rate; (II) Period-Doubling ( $u_{t}$ oscillates within $(0,2)$ ), preserving convergence albeit with oscillating rates; (III) Divergence ( $u_{t}>2$ ), where numerical instability occurs.

Figure 5: Phase diagram for

k=4

according to Thm. 6.1 .

Panel of Figure 6 — Figure 6: Three dynamical behaviors of Adam on $L(x)=\frac{1}{4}x^{4}$ with $\eta=0.001$ . Top Row: Evolution of training loss, gradient and first moment. The text online shows the slope of the exponential fit. Middle Row: Evolution of the second moment estimate $v_{t}$ versus the squared gradient $g_{t}^{2}$ . Insets show the evolution of the stability metric $\omega_{t}\lambda_{t}$ relative to the theoretical threshold $2/\eta$ . Bottom Row: Vector fields of $\omega_{t}$ and $\lambda_{t}$ ; purple lines represent evolutionary trajectories. (Left) Regime I: Stable Exponential Convergence. $v_{t}$ decouples from $g_{t}^{2}$ , enabling stable exponential acceleration. (Middle) Regime II: Exponential Convergence with Spike. $v_{t}$ initially decouples, driving acceleration toward the fixed point. However, drastic violation of the stability condition ( $\omega_{t}\lambda_{t}>2/\eta$ ) due to fixed-point instability triggers a loss spike. (Right) Regime III: SignGD-like Oscillation. $v_{t}$ tracks $g_{t}^{2}$ closely (tight coupling), preventing exponential convergence. Violations of the stability threshold trigger instantaneous corrections, resulting in oscillations.

Panel of Figure 7 — Figure 7: Empirical phase diagram on $L(x)=\frac{1}{4}x^{4}$ . Sliding average filter (window size $10$ ) applied for smoothing. (a) Minimum loss during training. (b) Final loss shows Regime II instability (blue sub-region of (a)). (c-d) Coupling ratios confirm exponential regions coincide with $v_{t}$ - $g_{t}^{2}$ decoupling.

Panel of Figure 8 — Figure 8: Optimization trajectories under different couplings of quadratic and quartic loss functions.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Training dynamics on quadratic loss $L(x)=\frac{1}{2}x^{2}$ . (a) Loss curves for Adam and RMSProp. (b-c) Comparison of second moment $v_{t}$ and squared gradient $g_{t}^{2}$ for RMSProp and Adam. (d) Comparison of first moment $m_{t}$ and gradient $g_{t}$ for Adam.

Panel of Figure 10 — Figure 10: Empirical phase diagrams of Adam optimizing $f(x)=\frac{x^{2}}{2}$ . A sliding average filter with a window size of $100$ is applied to smooth numerical fluctuations. (a) and (b) show the minimum and final loss of the optimization process, respectively. (c) and (d) display the maximum and final values of the ratio $\frac{v_{t}}{g_{t}^{2}}$ .

Panel of Figure 11 — Figure 11: Phase diagrams for degree $k=6$ . (a) Theoretical convergence region partitioned by the stability conditions in Eq. ( 8 ). (b) Experimental validation using Adam on $L(x)=\frac{1}{6}x^{6}$ ( $x_{0}=1.0,\eta=0.001$ ). The heatmap displays the final training loss after $100,000$ steps.

Panel of Figure 12 — Figure 12: Zoom-in analysis for $k=4$ . (a) Theoretical stability boundary for the full parameter range. (b) Magnified theoretical boundary for the lower-left corner ( $\beta_{2}\in[0,0.1]$ ). (c-d) Corresponding experimental loss heatmaps. Note the convergence observed in theoretically unstable areas in (d).

Panel of Figure 13 — Figure 13: Dynamics and Bifurcation Analysis. (a-c) Loss trajectories for fixed $\beta_{1}=0.001$ with decreasing $\beta_{2}$ . (d-e) Bifurcation diagrams of effective sharpness $u_{t}$ versus $\beta_{2}$ . The blue dashed line indicates the stability threshold $2/\eta$ . Low $\beta_{1}$ allows the system to remain bounded (below threshold) despite local instability, whereas high $\beta_{1}$ leads to immediate divergence.

Panel of Figure 14 — Figure 14: Three distinct dynamical regimes of Adam on the degenerate objective $L(x)=\frac{1}{4}x^{4}$ with learning rate $\eta=0.01$ . Top Row: Evolution of training loss (red, left axis) and parameter trajectory $x_{t}$ (blue, right axis). The text online shows the slope of the exponential fit. Middle Row: Evolution of the second moment estimate $v_{t}$ versus the squared gradient $g_{t}^{2}$ . Insets display the evolution of the stability metric $\omega_{t}\lambda_{t}$ relative to the theoretical threshold $2/\eta$ . Bottom Row: Vector fields of $\omega$ and $\lambda$ ; purple lines represent evolutionary trajectories. (Left Column) Regime I: Stable Exponential Convergence. $v_{t}$ completely decouples from $g_{t}^{2}$ , facilitating stable exponential acceleration. (Middle Column) Regime II: Exponential Convergence followed by Spike. $v_{t}$ initially decouples, driving acceleration. However, the stability condition is persistently violated ( $\omega_{t}\lambda_{t}>2/\eta$ ) due to response lag, triggering a violent loss spike. (Right Column) Regime III: SignGD-like Oscillation. $v_{t}$ tracks $g_{t}^{2}$ closely (tight coupling), preventing the exponential convergence phase. Violations of the stability threshold ( $\omega_{t}\lambda_{t}>2/\eta$ ) trigger an instantaneous correction, resulting in oscillations.

Panel of Figure 15 — Figure 15: Evolution of training loss across different architectures and optimizers, using learning rates tuned for optimal performance. (a, d) Adam and SGD perform comparably on the two-layer FNN with ReLU and the CNN. (b, e) Adam outperforms SGD on the two-layer FNN with Softmax and Transformers. (c, f) Hessian spectral densities of different model architectures at initialization. The densities are smoothed using KDE. (c) A two-layer FNN on MNIST with Softmax (blue) and ReLU (orange) activations. (f) GPT-2 (blue) and a two-layer CNN on CIFAR-10 (orange).

Panel of Figure 16 — Figure 16: Raw loss curves of a two-layer FNN trained with Adam and SGD across various learning rates. (a) ReLU activation. (b) Softmax activation.

Optimization Convergence Analysis

Optimization convergence analysis studies whether and how quickly iterative algorithms approach solutions, and which assumptions ensure reliable progress.

See all 74 papers

Adam Converges in Nonsmooth Nonconvex Optimization
Jun 21, 2026Zijian LiuHeavy-Tailed Noise Nonconvex Optimization
Department of Technology, Operations, and Statistics Leonard N. Stern School of Business New York University New York, NY 10012, USA
The Convergence Behavior of Adam under Heavy-Tailed Noise
Jul 29, 2026Yijiang PangNonconvex Stochastic Optimization Heavy-Tailed Noise
Michigan State University, East Lansing, USA
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
Nov 10, 2025Steffen Dereich, Thang Do, Arnulf Jentzen +1Stochastic Optimization Stochastic Optimization Convergence
Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China +1

Jun 21, 2026J/K move · Enter open · S save

Adam Converges in Nonsmooth Nonconvex Optimization

Zijian Liu

Department of Technology, Operations, and Statistics Leonard N. Stern School of Business New York University New York, NY 10012, USA

Heavy-Tailed Noise Nonconvex Optimization Stochastic Optimization Convergence Adaptive Gradient Methods Nonsmooth Convex Optimization

Open paper AskPDF arXiv

Adam is one of the most widely implemented and influential modern optimizers. Why is it effective across different optimization problems in practice? This question arguably lies at the center of the optimization community over the last decade and has motivated a substantial body of work aimed at understanding its convergence behavior. However, existing studies have mainly focused on the convergence rate of Adam in smooth nonconvex optimization, which unfortunately does not adequately capture practical settings, since many real-world problems are nonsmooth, such as those arising in training neural networks. Thus, these studies cannot fully explain the popularity and empirical success of Adam. Recently, an insightful and powerful framework called Online-to-Nonconvex Conversion has opened a new way to analyze Adam for nonsmooth nonconvex optimization. Unfortunately, prior works along this line share two common limitations. First, all of them ignore the important bias-correction term in the original Adam algorithm. Second and more importantly, many of them require extra operations that are not used in Adam, such as a clipping step. Therefore, the convergence guarantee for the original Adam method still remains unclear. In this work, we present the first finite-time analysis for the classical form of Adam, i.e., with the bias-correction step and without further algorithmic modifications, and prove that a randomly scaled learning rate ensures a convergence rate of

1/T^{\frac{2}{13}}

for nonsmooth nonconvex optimization. Moreover, our result provably applies to the modern heavy-tailed noise regime, which is closer to practice. Interestingly, our theory is established under the parameter choice

β_1=β_2

, aligning with the recent empirical studies.

Abstract

Optimization Convergence Analysis

Explore similar work

Adam Converges in Nonsmooth Nonconvex Optimization

The Convergence Behavior of Adam under Heavy-Tailed Noise

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer