It has recently been found that deep learning often occurs at the "edge of stability (EoS)," where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal to the learning rate. However, what happens before we reach that regime? We fix a deep learning problem and vary first order optimization methods with dense learning rate sweeps. We then track the characterizing quantities of a learning trajectory: the loss, the sharpness, and the alignment between consecutive gradients. To our surprise, if we scale the learning rate by the dc gain of the optimizer, these traces from the sweeps from different optimizers almost perfectly overlap across a large range of learning rates. The dc-normalized optimizers have another role that only becomes apparent in high learning rates: they select when the sharpness value detaches from this universal curve and enters the edge of stability. Upon this discovery, we specify three distinct regimes with respect to the dc-adjusted learning rate: the low-LR regime where the trajectory is nearly insensitive to the optimizer, the high-LR regime, where the optimizer governs the sharpness according to the EoS reciprocal rule, and the in-between mid-LR regime where so-called progressive sharpening originates independently of the optimizer. This distinguishes the role of the optimizer, the learning rate, and the model in shaping the learning progress.
Figures & tables
Figure 1: Main results. All results drawn in the same figure to visualize coincidence. Each line connects a learning rate sweep over a single optimizer. In the effective lr h coordinate, the operating range of this learning system is naturally partitioned into three distinct regimes. In the low and mid-LR regimes, the results coincide in an optimizer-independent curves. In the edge regime, the optimizer plays a role in determining the edge of stability, marked by the strict oblique lines S∝1/h in Figure 1 a. Consecutive gradient alignment ct and first order alias ρt marks the natural transition boundaries between these three regimes, marked by the triangle (between low and mid-LR) and square (between mid-LR and edge) markers.
Figure 2: Geometry repsonses of CNN and ViT. We observe common structures of low-LR softening dS/dh<0 and mid-LR sharpening dS/dh>0 in the sharpness S curve. However, the curve shapes are unique to each architecture, suggesting that this common curves are the model intrinsic material properties independent of optimization algorithms.
Figure 3: Geometry responses of MLP with full training traces. Instead of picking only the final values as in Figures 1 a and 1 b, we sample intermediate values at every 50 iterations for each of the 338 non-blow-up runs, collecting a total of 1.352M entries. We plot each run as a connected line drawn in the same plot, aligned in the training progress τ=ht=ηQ(1)t coordinate. The low-LR traces almost perfectly overlap, whereas the high-LR traces show clearly different trajectories.
Figure 4: Progress sharpening occurrence depends on the effective learning rate h . We plot traces of the sharpness S over the true training iterations t (not τ this time!) using a single exemplar case of a 30-degree escape Grokfast optimizer (GF30). Progress sharpening only occurs when the stabilized sharpness is larger than the initial sharpness S/S0>1 , condition of which is governed by the effective learning rate h .
Figure 5: A journey to the edge. A subset of Figure 1 showing the SGD curves.
Figure 6: The edge of stability. Eq. ( ♠ ‣ 1 ) holds for all optimizers and lrs.
Figure 7
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Optimizer
Q(1)
Q′(1)
αQ=21−Q(1)Q′(1)
θ⋆
k⋆
ΓQ
SGD
1
0
0.5
180∘
2
1
SGDM50
2
-2
1.5
180∘
6
0.33333333
SGDM90
10
-90
9.5
180∘
38
0.052631579
SGDN50
2
-1
1
180∘
3
0.66666667
SGDN90
10
-81
8.6
180∘
13.571429
0.14736842
QHM01
1
-8.8888889
9.3888889
180∘
31.090909
0.064327485
Appendix
Table 1: Characteristic quantities of the first order optimizers used in the experiments. The parameters of the filters Q(z) and the corresponding edge gains. The edge gain ΓQ is twice the reciprocal of the first positive gain k⋆ for which a root of z−1+kQˉ(z)=0 reaches the unit circle.
Optimizer
hlow|mid
hmid|high
ηedge
Nlow
Nmid
Nhigh
SGD
5e-05
0.003
0.003
16
11
6
SGDM50
5e-05
0.01
0.005
15
13
5
SGDM90
5e-05
0.05
0.005
10
18
5
SGDN50
5e-05
0.004
0.002
15
11
6
SGDN90
5e-05
0.02
0.002
10
16
5
QHM01
5e-05
0.05
0.05
16
18
4
Appendix
Table 2: Regime boundaries of MLP experiments. All optimizers have the same low-LR/mid-LR boundaries hlow|mid , since their curves coincide into a single curve. However, their mid-LR/high-LR boundaries hmid|high differ. We also provide the edge position in the raw learning rate coordinate ηedge=hmid|high/Q(1) . Finally, we also provide the number of runs fall into each category of the regimes Nlow , Nmid , and Nhigh for each optimizer.
Figure 9: Individual plots of MLP sharpness S . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 10: Individual plots of MLP loss L . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 11: Individual plots of MLP alignment ct . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 12: Individual plots of MLP aliasing ρt . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 13: Individual plots of CNN sharpness S . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 14: Individual plots of CNN alignment ct . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 15: Individual plots of CNN loss L . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 16: Individual plots of CNN aliasing ρt . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 17: Individual plots of ViT sharpness S . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 18: Individual plots of ViT loss L . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 19: Individual plots of ViT alignment ct . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
Figure 20: Individual plots of ViT aliasing ρt . Plots are drawn with respect to the training iteration t coordinate. Different learning rates are colored in different colors.
The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as ×21.1. Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.
Jaerin Lee, Kyoung Mu Lee
Computer Vision Lab, ASRI Seoul National University Seoul 08826, Korea
Empirically, modern deep learning training often occurs at the Edge of Stability (EoS), where the sharpness of the loss exceeds the threshold below which classical convergence analysis applies. Despite recent progress, existing theoretical explanations of EoS either rely on restrictive assumptions or focus on specific squared-loss-type objectives. In this work, we introduce and study a structural property of loss functions that we term product-stability. We show that for losses with product-stable minima, gradient descent applied to objectives of the form (x,y)↦l(xy) can provably converge to the local minimum even when training in the EoS regime. This framework substantially generalizes prior results and applies to a broad class of losses, including binary cross entropy. Using bifurcation diagrams, we characterize the resulting training dynamics, explain the emergence of stable oscillations, and precisely quantify the sharpness at convergence. Together, our results offer a principled explanation for stable EoS training for a wider class of loss functions.
Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold 2/η, where η is the learning rate. Classical analysis tools such as gradient flow and the descent lemma do not apply here, motivating the search for a continuous-time model valid at EoS. We propose Edge Flow, a system of three coupled ordinary differential equations that provides a tractable, faithful, and predictive model of gradient descent dynamics at EoS. Edge Flow decomposes the dynamics into a center, an oscillation direction, and an oscillation magnitude. The center follows a modified gradient flow on a symmetrized loss; the direction tracks a top eigenvector of the Hessian via Rayleigh quotient dynamics; and the magnitude grows or decays exponentially depending on whether the sharpness exceeds or falls below the threshold 2/η. Crucially, sharpness stabilization emerges from the coupled dynamics via a self-stabilization feedback loop. Discretizing Edge Flow only requires two gradient evaluations and one Hessian--vector product at each iteration. We demonstrate empirically that Edge Flow tracks the dynamics of gradient descent at least as faithfully as previously proposed continuous-time EoS models, while in addition resolving the oscillation of the sharpness at the onset of EoS, and that it provides a principled framework for understanding and mitigating instabilities in this regime.
Pierre Marion
Inria, École Normale Supérieure, PSL Research University