The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as ×21.1. Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.
Figures & tables
Figure 1: Edge of stability across different families of optimizers. We train a fully-connected network on CIFAR-10 using various full-batch gradient-based optimizers: gradient descent (GD), Adam, GD with a pole-zero filter (QHM) and GD with a two-pole filter (Grokfast), and AdaGrad. Theoretical prediction of the edge of stability χ:=ηSΓ/2=1.0 is occasionally violated in practice, and the offsets χ are heavily dependent on the type of optimizers in use. This systematic deviations motivate a new formulation of the realized edge of stability ζ:=χς/τ with ς and τ being the spatial and temporal calibrations, respectively. This new formulation is more consistent with the empirical observations and removes the optimization-dependent offsets.
Figure 2: Root locus of optimizers. It is possible to exit the unit circle at z=−1 , i.e., Γ=Q(−1) .
Figure 3: Family-wise ranges of per-setting interval medians. Medians of χ taken from the interval between first χ>0.75 and the final downward exit. The previous formulation of the edge of stability χ≤1 ( Cohen et al., 2021 ; Cohen et al., 2023 ) is occasionally violated by large, optimizer-dependent offsets. Our realized EoS ζ has its median universally upper bounded by near 1.0, which is more consistent with the name “edge” of stability.
Figure 4: EoS stack plot. Decomposition of the realized EoS ζ into factors in log-scale, involving the (relative) edge of stability χ ( red area and line), the spatial participation factor ς ( blue area and line), and the temporal calibration factor τ ( grey overlay and black line). Note that logτ operates in the opposite direction by subtraction.
Table 5
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Optimizer
Q(z)
Q(1)
mixture (wj;βj)
Γ=Q(−1)
GD
1
1
(1;0)
1
Heavy Ball
1−βz−11
1−β1
—
1+β1
normalized HB
1−βz−11−β
1
(1;β)
1+β1−β
Nesterov
1+1−βz−1β
1−β1
—
1+β1+2β
normalized Nesterov
(1−β)+1−βz−1β(1−β)
1
(1−β,β;0,β)
1+β(1−β)(1+2β)
QHM, ν∈[0,1]
(1−ν)+1−βz−1ν(1−β)
1
(1−ν,ν;0,β)
1+β1+β(1−2ν)
Appendix
Table 2: Root-locus gauge of the one-pole and parallel optimizer filters. Every dc-normalized entry is a positive one-pole mixture ( 25 ), so Lemma B.3 certifies Ω+={π} and Γ=Q(−1) . Cascaded two-pole momenta are not in this table. The last column recovers the results from Cohen et al. (2023) .
Figure 6: We optimize a quadratic loss f(x)=21x2 with four types of optimizers: GD, Adam, QHM, and cascade two-pole at various learning rates η . Observe that the stability bound of χ=ηSΓ/2≤1 is always satisfied for this quadratic loss.
Filter
Γexp / 2
2/Γexp
2/Q(−1)
ω⋆/π
observed η
GD
0.5
2
2
1.000
[2,2.01]
EMA-HB β=0.9
0.02632
38
38
1.000
[37.81,38]
QHM β=0.9,ν=0.7
0.1684
5.9375
5.9375
1.000
[5.908,5.938]
two-pole 0.4/0.7
0.3889
2.571
26.44
0.263
[2.571,2.584]
two-pole 0.7/0.85
1.469
0.681
139.8
0.088
[0.681,0.684]
Appendix
Table 3: Predicted and observed quadratic thresholds. Γexp is the experiment-report gauge. The observed η is the last stable and first unstable η on a 0.005 -relative grid. 605 sweep runs in total.
Figure 7: Complete root locus for the filters studied in this paper, extending Figure 2 in the main text to all filters.
Filter
Families
Q(1)
Q(−1)
Γ
k⋆
ω⋆/π
GD
GD, RMSProp, AdaGrad
1
1
1
2
1
EMA β=0.9
Adam, AdamW, Adafactor, AMSGrad, PAdam
1
0.05263
0.05263
38
1
NAdam β=0.9
NAdam
1
0.1474
0.1474
13.57
1
HB β=0.9
Heavy Ball
10
0.5263
0.5263
3.8
1
NAG β=0.9
Nesterov
10
1.474
1.474
1.357
1
QHM β=0.9,ν=0.7
QHM
1
0.3368
0.3368
5.938
1
Appendix
Table 4: Temporal filters of the executed families, in the paper-normalized convention χ=ηSΓ/2 of Appendix C.1 . k⋆ is the unit-curvature learning-rate edge. ω⋆ is the first unit-circle crossing in radians. Adam, AdamW, Adafactor, AMSGrad and PAdam share one EMA filter; RMSProp, AdaGrad and GD share the memoryless filter. Note that two-pole momentum cases have Γ=Q(−1) .
Family
η grid
β1 / β
β2
ε
BC / p
wd
Adam
3×10−5 – 10−3
0.9
0.999
10−7
yes / 0.5
0
AdamW
3×10−5 – 10−3
0.9
0.999
10−7
yes / 0.5
0.01
Adafactor
10−5 – 10−3
0.9
0.8
10−7
no / 0.5
0
AMSGrad
10−5 – 10−3
0.9
0.999
10−7
no / 0.5
0
PAdam
10−3 – 10−1
0.9
0.999
10−7
no / 0.25
0
NAdam
10−5 – 10−3
0.9
0.999
10−7
no / 0.5
0
Appendix
Table 5: Optimizer grids on used in the paper. Learning rates are five geometrically spaced values. The displayed endpoints are the grid minima and maxima. “BC” is bias correction. p is the preconditioner exponent on the second moment. wd is decoupled weight decay.
Family
η1
η2
η3
η4
η5
Center
n
χ range
ζ range
Adafactor
—
1.052
1.103
1.102
1.191
1.084
204/350
1.052–1.191
0.202442–0.965959
AdaGrad
1.424
1.556
1.845
4.047
21.070
1.248∗
3/350
1.424–21.070
0.982240–1.011240
Adam
0.927
0.980
0.971
0.976
0.974
0.960
259/350
0.927–0.980
0.177652–0.948673
AdamW
0.956
0.981
0.975
1.023
0.971
0.971
247/350
0.956–1.023
0.115807–0.936750
AMSGrad
0.789
0.931
0.979
0.946
0.949
0.960
203/350
0.789–0.979
−0.000206–0.836529
GD
0.904
1.130
1.246
1.478
1.624
1.124
47/140
0.904–1.624
0.000101–0.991144
Appendix
Table 6: Measured relative edge χ=ηSΓ/2 offsets for different optimizers. Each cell is the median of χ between first entry above 0.75 and last exit below 0.75 (open windows remain open at the horizon). A dash means the trajectory never exceeded 0.75 . Extreme cases χ>1.2 are highlighted in boldface. ∗ The manuscript marks AdaGrad and RMSProp as sparse edge-touch statistics.
Figure 8: Comparison of worst-case EoS χ and realized EoS ζ on the primary CIFAR-10 experiments.
Figure 9: Comparison of worst-case EoS χ and realized EoS ζ on the primary CIFAR-10 experiments.
Figure 10: Decomposition of the realized EoS ζ into factors in log-scale, involving the (relative) edge of stability χ ( red area and line), the spatial participation factor ς ( blue area and line), and the temporal calibration factor τ ( grey overlay and black line). Note that logτ operates in the opposite direction by subtraction.
Figure 11: Decomposition of the realized EoS ζ into factors in log-scale, involving the (relative) edge of stability χ ( red area and line), the spatial participation factor ς ( blue area and line), and the temporal calibration factor τ ( grey overlay and black line). Note that logτ operates in the opposite direction by subtraction.
Figure 12: Decomposition of the realized EoS ζ into factors in log-scale, involving the (relative) edge of stability χ ( red area and line), the spatial participation factor ς ( blue area and line), and the temporal calibration factor τ ( grey overlay and black line). Note that logτ operates in the opposite direction by subtraction.
Figure 13: Decomposition of the realized EoS ζ into factors in log-scale, involving the (relative) edge of stability χ ( red area and line), the spatial participation factor ς ( blue area and line), and the temporal calibration factor τ ( grey overlay and black line). Note that logτ operates in the opposite direction by subtraction.