Precise Convergence Speed of Clipped SGD

David A. R. Robin

cs.LGSep 24, 2026

Precise Convergence Speed of Clipped SGD

Authors: David A. R. Robin

Organizations: LAMSADE, Université Paris-Dauphine, PSL Research University

Abstract

We present a tightened convergence analysis of clipped gradient descent on $(L_0, L_1)$ -smooth functions, with quantitative constants. Building on the ideas of Koloskova et al (2023), we refactor several case disjunctions to reveal the central role of a control of the bias derived from fundamental properties of $\ell_2$ -projection, simplifying proofs. We also extend the domain of validity from $η\leq 1 / (9 β)$ to $η< 1 /β$ where $β= L_0 + c L_1$ for clipping constant $c$ , which matches the more traditional analysis of smooth functions. We strengthen the convergence criterion from $\left( \min_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ to $\left( \frac{1}{T} \sum_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ with matching speed, and lower the final achievable loss from $\mathcal{O}(\min(σ^2/c, σ))$ to the more precise $6 \min(σ^2 /c, 3 σ)$ .

Explore similar work

Robust and Fast Training via Per-Sample Clipping
May 4, 2026Davide Nobile, Philipp GrohsStochastic Gradient Descent Gradient Clipping
Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences
Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H"{o}lder Smoothness
Sep 14, 2026Misbah Uz Zaman, Anirbit MukherjeeStochastic Gradient Descent Lipschitz Continuity
Department of Mathematics and Statistics Indian Institute of Science Education and Research Kolkata · Department of Computer Science The University of Manchester
Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
Sep 10, 2026Amartya Mukherjee, Jun LiuStochastic Gradient Descent Gradient Clipping
Department of Applied Mathematics, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1

May 4, 2026, math.OCJ/K move · Enter open · S save

Robust and Fast Training via Per-Sample Clipping

Davide Nobile, Philipp Grohs

Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences

Stochastic Gradient Descent Gradient Clipping Convergence Convex Optimization

Open paper AskPDF arXiv

We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically. We show that the resulting method, per-sample clipped SGD (PS-Clip-SGD), achieves optimal in-expectation convergence rates for non-convex optimization problems under heavy-tailed gradient noise. Moreover, we establish high-probability convergence guarantees that match the in-expectation rates up to polylogarithmic factors in the failure probability. We complement our theoretical results with multiple numerical experiments. In particular, we demonstrate that PS-Clip-SGD outperforms both vanilla SGD with momentum and standard gradient clipping when training AlexNet on the CIFAR-100 dataset, even after accounting for the additional computational time caused by per-sample clipping. We also empirically show that, in the presence of gradient accumulation, applying clipping at the mini-batch level can improve training performance while incurring virtually no additional computational cost. This finding is particularly interesting, as it contradicts the common practice of applying clipping only after all accumulation steps have been completed.

Abstract

Explore similar work

Robust and Fast Training via Per-Sample Clipping

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H"{o}lder Smoothness

Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise