Non-Euclidean Gradient Descent Has Many Edges of Stability

Peter Zaika; Zakhar Shumaylov

math.OCOct 10, 2026

Non-Euclidean Gradient Descent Has Many Edges of Stability

Authors: Peter Zaika, Zakhar Shumaylov

Organizations: University of Cambridge

Abstract

Recent observations that deep learning models train at the edge of stability have renewed interest in understanding stability properties of classical optimization algorithms. For quadratic objectives, standard gradient descent has a sharp stability threshold: it converges from every initialization when $η<2/L$ , diverges from almost every initialization when $η>2/L$ , and diverges from every nonzero initialization when $η>2/μ$ . For non-Euclidean gradient descent, the corresponding known guarantees are weaker: convergence holds for every initialization when $η<2/S$ , where $S$ is the generalized sharpness, while every nonzero initialization diverges when $η>2/μ$ . The behavior in the intermediate regime $2/S<η<2/μ$ is not fully characterized. We show that Hilbertian norms are the only ones for which $2/S$ is an almost-everywhere sharp threshold for every positive-definite quadratic objective. Using linear conjugacy between non-Euclidean and Euclidean gradient descent, we show that for general norms the intermediate regime can contain arbitrarily many edges of stability associated with distinct attracting directions. We identify a forward-invariant region of exponentially divergent directions beyond the generalized stability threshold. When the directions of minimum curvature are finite and nondegenerate, we prove that, sufficiently close to $2/μ$ from below, only their invariant directions remain convergent. We give smooth two-dimensional constructions with arbitrarily many distinct thresholds, arbitrarily large relative threshold separation and show these properties to be stable.

Figures & tables

Figure 1: Convergent (blue) and divergent (red) initial directions for the example of Section 7.2 at four step sizes, on one hemisphere of the conjugate $z$ -coordinates, which contains every direction since $u$ and $-u$ coincide. Markers show the local maxima $u_{1}$ (global) and $u_{2}$ of $q_{H}$ : filled if they attract nearby directions, and coloured by the fate of their own line. Grey arrows show the two-step direction of motion; black curves are three trajectories of directions, starting at the open squares. Hatching marks the invariant divergence region $\mathcal{U}_{\eta}$ of Theorem 7 . Computational details are in Section E.5 .

Figure 2: Stability regimes of non-Euclidean gradient descent on a positive-definite quadratic, as a function of the step size

\eta

(positions schematic). For every norm, all initializations converge for

\eta<2/S_{H}

and every nonzero initialization diverges for

\eta>2/\mu_{H}

. (a) For a Hilbertian norm, almost every initialization diverges throughout the intermediate window

2/S_{H}<\eta<2/\mu_{H}

. (b) For non-Hilbertian norms and suitable

H

, different projectively attracting critical directions have distinct radial stability thresholds

2/\lambda_{i}

, allowing open convergent and divergent basins to coexist.

Figure 3: The nonlinear Rayleigh landscape

q_{H}

on the Euclidean unit circle for the smooth norm in ( 23 ). The global and suboptimal maxima are attained at

\theta=3\pi/4

and

\theta=\pi/4

, respectively, with values

S_{H}=3/2

and

\lambda_{\rm sub}=1

. Their radial thresholds are

4/3

and

2

. The minimum is

\mu_{H}=2\sqrt{3}-5/2

Figure 4: Finite-time radial growth $T^{-1}\log(r_{T}/r_{0})$ in the conjugate $z$ -coordinates for the example ( 23 ), over initial directions $\theta_{0}$ and step sizes $\eta$ ( $T=400$ ; colour scale clipped to $[-1.5,1]$ ). Blue denotes negative finite-time growth and red denotes positive growth; the black contour is the finite-time zero-growth level set. For $4/3<\eta<2$ , open sectors of convergent and divergent initializations coexist. For these parameters, the convergent sector occupies approximately one third of initial directions in the displayed range. The dashed levels mark $2/S_{H}$ , $2/\lambda_{\rm sub}$ , and $2/\mu_{H}$ .

Gradient Descent Dynamics

Gradient Descent Dynamics studies how iterative gradient-based updates shape optimization trajectories, stability, convergence, and the solutions models learn.

See all 153 papers

Mar 5, 2026

Non-Euclidean Gradient Descent Operates at the Edge of Stability

The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian approaches and then hovers near the stability threshold

2/η

during gradient descent (GD) with step size

η

. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its theoretical foundations remain incomplete. We provide an interpretation of EoS through the lens of Directional Smoothness [Mishkin et al., 2024]. This interpretation naturally extends to non-Euclidean norms, which we use to define generalized sharpness under an arbitrary norm. Our generalized sharpness measure includes previously studied vanilla GD and preconditioned GD as special cases, as well as methods for which EoS has not been studied, such as

\ell_{\infty}

-descent, Block CD, Spectral GD, and their normalized versions. Through experiments on neural networks, we show that non-Euclidean GD with our generalized sharpness also exhibits progressive sharpening followed by oscillations around or above the threshold

2/η

. Practically, our framework provides a geometry-aware spectral diagnostic that can be applied across a broad class of non-Euclidean gradient methods.

Rustem Islamov, Michael Crawshaw, Jeremy Cohen +1

University of Basel, Switzerland · George Mason University, USA · Flatiron Institute, USA

Edge of Stability Gradient Descent Dynamics

Ask PDF

Jun 14, 2026

A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

The Edge of Stability (EoS) phenomenon, where gradient descent operates with sharpness exceeding the classical convergence threshold yet the loss decreases over long timescales, is ubiquitous in modern deep learning but remains poorly understood in realistic settings. Prior rigorous analyses have been largely confined to scalar or low-dimensional losses with specific structural forms. In this work, we develop a bifurcation theory framework for gradient descent on the edge of stability that applies directly to overparameterized neural networks. By decomposing the training dynamics into components normal and tangent to the manifold of minimizers, we show that stable EoS training arises from a flip bifurcation in the normal direction, governed by the sign of the first Lyapunov coefficient, while the tangent dynamics drift toward regions of decreasing sharpness. Under mild spectral and geometric assumptions on the loss landscape, we prove convergence to the minimizing manifold when training at the EoS threshold. As a corollary, we recover and unify prior results: we show that the product-stability condition of Gan (2026) is an instance of our framework.

Eric Gan

Independent Researcher.

Dynamical Systems Edge of Stability Neural Network Training Dynamics Gradient Descent Dynamics

Ask PDF

Apr 3, 2026

Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability

Empirically, modern deep learning training often occurs at the Edge of Stability (EoS), where the sharpness of the loss exceeds the threshold below which classical convergence analysis applies. Despite recent progress, existing theoretical explanations of EoS either rely on restrictive assumptions or focus on specific squared-loss-type objectives. In this work, we introduce and study a structural property of loss functions that we term product-stability. We show that for losses with product-stable minima, gradient descent applied to objectives of the form

(x,y) \mapsto l(xy)

can provably converge to the local minimum even when training in the EoS regime. This framework substantially generalizes prior results and applies to a broad class of losses, including binary cross entropy. Using bifurcation diagrams, we characterize the resulting training dynamics, explain the emergence of stable oscillations, and precisely quantify the sharpness at convergence. Together, our results offer a principled explanation for stable EoS training for a wider class of loss functions.

Eric Gan

Independent Researcher.

Deep Learning Optimization Edge of Stability Gradient Descent Dynamics Neural Network Stability Loss Landscape Sharpness

Ask PDF

Abstract

Gradient Descent Dynamics

Explore similar work

Non-Euclidean Gradient Descent Operates at the Edge of Stability

A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability