On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials

Rotem Mulayoff; Sebastian U. Stich

cs.LGFeb 16, 2026

On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials

Authors: Rotem Mulayoff, Sebastian U. Stich

Organizations: CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Abstract

The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient descent (GD) correspond to flat minima, which have been associated with favorable features. While prior work often relies on linearization to determine stability, it remains unclear whether linearized dynamics faithfully capture the full nonlinear behavior. Recent work has shown that GD may stably oscillate near a linearly unstable minimum and still converge once the step size decays, indicating that linear analysis can be misleading. In this work, we explicitly study the effect of nonlinear terms. Specifically, we derive an exact criterion for stable oscillations of GD near minima in the multivariate setting. Our condition depends on high-order derivatives, generalizing existing results. Extending the analysis to stochastic gradient descent (SGD), we show that nonlinear dynamics can diverge in expectation even if a single batch is unstable. This implies that stability can be dictated by a single batch that oscillates unstably, rather than an average effect, as linear analysis suggests. Finally, we prove that if all batches are linearly stable, the nonlinear dynamics of SGD are stable in expectation.

Figures & tables

Panel of Figure 1 — Figure 1: Stable vs. unstable oscillations near a minimum. We apply GD to $f_{+}$ and $f_{-}$ from ( 2 ) with various step sizes $\eta\in(1,4)$ . The resulting dynamics ( 3 ) correspond to the normal form of a flip bifurcation. Once the step size exceeds the linear stability threshold $\eta_{\mathrm{lin}}=2$ , stability is determined by the sign of the cubic term in the dynamics. Panel \subfigref fig:Warmup_functions shows $f_{+}$ and $f_{-}$ , whose minima share the same sharpness. Panel \subfigref fig:Warmup_bifurcation1 visualizes GD’s output on $f_{+}$ with various step sizes. When the step size $\eta$ crosses $\eta_{\mathrm{lin}}$ , the minimum $x^{*}=0$ loses stability, resulting in unstable oscillations, which lead to divergence. Panel \subfigref fig:Warmup_bifurcation2 depicts GD’s convergent points on $f_{-}$ for various step sizes. At the threshold, $\eta=\eta_{\mathrm{lin}}$ , the minimizer $x^{*}=0$ loses stability, and the iterates settle into a stable period- $2$ cycle, which then undergoes period doubling, chaos, and eventually divergence for $\eta>4$ .

Figure 2: Demonstration of Thm. 7 . Consider

\mathcal{L}_{\beta}(x_{1},x_{2})=\frac{1}{2}x_{1}^{2}+\frac{1}{10}x_{2}^{2}+\beta x_{1}^{2}x_{2}+\frac{1}{10}x_{1}^{4}

, whose linear stability threshold under GD at the vicinity of the local minimizer

\bm{x}^{*}=(0,0)

\eta_{\mathrm{lin}}=2

. According to Thm. 7 , GD at the edge of stability oscillates stably around

\bm{x}^{*}

if and only if

|\beta|>0.2

(see App. VI ). Panel \subfigref fig:GD_func_graphs plots

\mathcal{L}_{\beta}

near

\bm{x}^{*}

for

\beta=0.1

and

\beta=0.5

, highlighting the asymmetry introduced by the cubic term. Panel \subfigref fig:GD Oscillations shows the long-term value

\bm{x}_{T}

across a range of

\beta

. When

|\beta|>0.2

, GD converges to a stable period-

2

cycle, whereas for

|\beta|<0.2

the iterates diverge. This validates that condition ( 8 ) precisely captures the transition from stability to instability.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$a$	Lower case non-bold letters for scalars
$\bm{a}$	Lower case bold letters for vectors
$\bm{A}$	Upper case bold for matrices
$\bm{A}^{\mathrm{T}}$	Transpose of $\bm{A}$
$\otimes$	Kronecker product
⊗k	$k$ ’th Kronecker power
$f\circ g$	Composition of $f$ and $g$ , i.e. , $f(g(\bm{x}))$

Appendix

Table 1: Table of notations

Gradient Descent Dynamics

Gradient Descent Dynamics studies how iterative gradient-based updates shape optimization trajectories, stability, convergence, and the solutions models learn.

See all 148 papers

SGD at the Edge of Stability: The Stochastic Sharpness Gap
Apr 22, 2026Fangshuo Liao, Afroditi Kolomvaki, Anastasios KyrillidisEdge of Stability Neural Network Optimization
Rice CS · Rice CS, Ken Kennedy Institute
SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates
Jun 29, 2026Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1Edge of Stability Stochastic Optimization Convergence
Department of CIS1 · IDEAS4 · University of Pennsylvania +2
Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
Apr 15, 2026Arseniy Andreyev, Advikar Ananthkumar, Marc Walden +2Stochastic Gradient Descent Gradient Descent Dynamics
Meta. · Work done at Princeton University, Princeton, NJ, USA. · Harvard University, Cambridge, MA, US +1

Apr 22, 2026J/K move · Enter open · S save

SGD at the Edge of Stability: The Stochastic Sharpness Gap

Fangshuo Liao, Afroditi Kolomvaki, Anastasios Kyrillidis

Rice CS · Rice CS, Ken Kennedy Institute

Edge of Stability Neural Network Optimization Neural Network Training Dynamics Gradient Descent Dynamics Stochastic Gradient Descent

Open paper AskPDF arXiv

When training neural networks with full-batch gradient descent (GD) and step size

η

, the largest eigenvalue of the Hessian -- the sharpness

S(\boldsymbolθ)

-- rises to

2/η

and hovers there, a phenomenon termed the Edge of Stability (EoS). \citet{damian2023selfstab} showed that this behavior is explained by a self-stabilization mechanism driven by third-order structure of the loss, and that GD implicitly follows projected gradient descent (PGD) on the constraint

S(\boldsymbolθ)\leq 2/η

. For mini-batch stochastic gradient descent (SGD), the sharpness stabilizes below

2/η

, with the gap widening as the batch size decreases; yet no theoretical explanation exists for this suppression. We introduce stochastic self-stabilization, extending the self-stabilization framework to SGD. Our key insight is that gradient noise injects variance into the oscillatory dynamics along the top Hessian eigenvector, strengthening the cubic sharpness-reducing force and shifting the equilibrium below

2/η

. Following the approach of \citet{damian2023selfstab}, we define stochastic predicted dynamics relative to a moving projected gradient descent trajectory and prove a stochastic coupling theorem that bounds the deviation of SGD from these predictions. We derive a closed-form equilibrium sharpness gap:

ΔS = ηβσ_{\boldsymbol{u}}^{2}/(4α)

, where

α

is the progressive sharpening rate,

β

is the self-stabilization strength, and

σ_{ \boldsymbol{u}}^{2}

is the gradient noise variance projected onto the top eigenvector. This formula predicts that smaller batch sizes yield flatter solutions and recovers GD when the batch equals the full dataset.

Abstract

Gradient Descent Dynamics

Explore similar work

SGD at the Edge of Stability: The Stochastic Sharpness Gap

SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates

Momentum Further Constrains Sharpness at the Edge of Stochastic Stability