Functional Central Limit Theorem for Stochastic Gradient Descent

Kessang Flamand; Victor-Emmanuel Brunel

stat.MLFeb 17, 2026

Functional Central Limit Theorem for Stochastic Gradient Descent

Authors: Kessang Flamand, Victor-Emmanuel Brunel

Organizations: CREST-ENSAE

Abstract

We study the asymptotic shape of the trajectory of the stochastic gradient descent algorithm applied to a convex objective function. Under mild regularity assumptions, we prove a functional central limit theorem for the properly rescaled trajectory. Our result characterizes the long-term fluctuations of the algorithm around the minimizer by providing a diffusion limit for the trajectory. In contrast with classical central limit theorems for the last iterate or Polyak-Ruppert averages, this functional result captures the temporal structure of the fluctuations and applies to non-smooth settings such as robust location estimation, including the geometric median.

Figures & tables

Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) uses noisy, data-sampled gradients to train models and solve large-scale optimization problems.

See all 214 papers

Scaling Limits of Constant-Stepsize SGD at Flat Minima
Jul 17, 2026Jingyi Zhang, Cheng Mao, Debankur MukherjeeStochastic Optimization Convergence Flat Minima
Georgia Institute of Technology
Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference
Oct 10, 2026Joseph Keslin, Wei Biao WuStochastic Gradient Descent Heavy-Tailed Stochastic Optimization
Department of Statistics, University of Chicago
The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives
Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate Scheduling Last-Iterate Convergence
School of Data Science, Fudan University · Shanghai Center for Mathematical Sciences, Fudan University

Jul 17, 2026J/K move · Enter open · S save

Scaling Limits of Constant-Stepsize SGD at Flat Minima

Jingyi Zhang, Cheng Mao, Debankur Mukherjee

Georgia Institute of Technology

Stochastic Optimization Convergence Flat Minima Stochastic Gradient Descent

Open paper AskPDF arXiv

For stochastic gradient descent (SGD) with a constant stepsize

α

, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar

\sqrtα

scaling and a Gaussian limit as

α\downarrow 0

. We show that this behavior changes fundamentally for convex objectives

H

with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize

α

, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an

α

-dependent metric. When the minimizer

x_\star

has local flatness exponent

m\ge2

, meaning that

\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d

x\to x_\star

, we obtain a contraction bound with factor

1-cα^{m-1}

, where

c>0

is a constant. This recovers the factor

1-cα

in the quadratic case

m=2

. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale

α^{1/m}

and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation

dY_t=-h_0(Y_t)\,dt+Σ^{1/2}\,dB_t ,

where

h_0

is the limiting drift at the minimizer and

Σ

denotes the asymptotic covariance. This recovers the Gaussian limit when

m=2

and gives generally non-Gaussian stationary limits in the flat case

m>2

. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.

Abstract

Stochastic Gradient Descent

Explore similar work

Scaling Limits of Constant-Stepsize SGD at Flat Minima

Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference

The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives