Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

Benjamin Gess; Daniel Heydecker

cs.LGMar 10, 2026

Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

Authors: Benjamin Gess, Daniel Heydecker

Organizations: Technische Universität Berlin, Fakultät II 10623 Berlin, Germany · Max Planck Institute for Mathematics in the Sciences 04103 Leipzig, Germany · University of Oslo, Blindern 0316 Oslo, Norway

Abstract

Large loss spikes in stochastic gradient descent are studied through a rigorous large-deviations analysis for a shallow, fully connected network in the NTK scaling. In contrast to full-batch gradient descent, the catapult phase is shown to split into inflationary and deflationary regimes, determined by an explicit log-drift criterion. In both cases, large spikes are shown to be at least polynomially likely. In addition, these spikes are shown to be the dominant mechanism by which sharp minima are escaped and curvature is reduced, thereby favouring flatter solutions. Corresponding results are also obtained for certain ReLU networks, and implications for curriculum learning are derived.

Large Deviation Principles

See all 8 papers

Flatland: The Adventures of Gradient Descent with Large Step Sizes
Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Edge of Stability Gradient Descent
LMU Munich · Munich Center for Machine Learning · University of British Columbia +1
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Fokker–Planck Equation Neural Network Training Dynamics
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · P.N. Lebedev Physical Institute of the Russian Academy of Sciences · Innopolis University
Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
Apr 15, 2026Arseniy Andreyev, Advikar Ananthkumar, Marc Walden +2Stochastic Gradient Descent Gradient Descent Dynamics
Meta. · Work done at Princeton University, Princeton, NJ, USA. · Harvard University, Cambridge, MA, US +1

Jun 4, 2026J/K move · Enter open · S save

Flatland: The Adventures of Gradient Descent with Large Step Sizes

Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt+1

LMU Munich · Munich Center for Machine Learning · University of British Columbia+1

Edge of Stability Gradient Descent Adaptive Gradient Methods Gradient Descent Dynamics Neural Network Stability Loss Landscape Sharpness

Open paper AskPDF arXiv

The training of neural networks often entails objective functions that are not globally

L

-smooth. For these functions, it is both theoretically and practically difficult to reply to the question: what is the largest possible step size that ensures the convergence of gradient descent (GD)? We address this longstanding open question in deep learning by providing a unifying definition of "large" step sizes that requires only local Lipschitz (or even Hölder) continuity of the gradient. We design first-order adaptive methods that provably yield large step sizes and show that they operate at the edge of stability (EoS) right from the start of the training. In particular, the loss decreases nonmonotonically and the product between the step size and sharpness, i.e., the largest eigenvalue of the Hessian, stays above the EoS threshold of 2 throughout training. Using our method, we are also able to minimize the sharpness all the way down to its global minimum. Contrary to expectation, we find that encountering globally-flat regions too early in the training may both slow down convergence and jeopardize the generalization ability of the network. Exploiting a self-stabilization argument, we allow GD to enter slightly sharper valleys and turn unsuccessful training runs into very successful ones.

Abstract

Large Deviation Principles

Explore similar work

Flatland: The Adventures of Gradient Descent with Large Step Sizes

Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

Momentum Further Constrains Sharpness at the Edge of Stochastic Stability