Last-Iterate Convergence of Clipped SGD with Linear Decay

Wei Jiang; Haomin Bai; Zechao Li

cs.LGOct 13, 2026

Last-Iterate Convergence of Clipped SGD with Linear Decay

Authors: Wei Jiang, Haomin Bai, Zechao Li

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology · National Key Laboratory for Novel Software Technology, Nanjing University

Abstract

In this paper, we study the last-iterate convergence of clipped stochastic gradient descent under heavy-tailed noise. We assume that the stochastic gradients are unbiased and that the noise has a bounded central $p$ -th moment, where $1<p\le2$ . For smooth convex objectives, a linearly decreasing learning rate gives a high-probability error bound of order $O(LR^2\log(1/δ)/T+σR(\log(1/δ)/T)^{(p-1)/p})$ . We further establish last-iterate guarantees for nonsmooth composite objectives and smooth composite objectives with proximal updates. In addition, when a causal adversary replaces at most $q$ gradients, the same clipped update achieves a bound with $\log(1/δ)$ replaced by $\log(1/δ)+q$ . The corresponding lower bounds for the clean and corrupted oracle models are also provided.

Figures & tables

Setting	Objective error	Oracle calls
Smooth, Theorem 1	$LR^{2}\ell/T+\sigma R(\ell/T)^{a}$	$T$
Nonsmooth composite, Theorem 2	$GR/\sqrt{T}+GR\ell/T+\sigma R(\ell/T)^{a}$	$T$
Smooth composite, Theorem 3	$LR^{2}\ell/T+\sigma R(\ell/T)^{a}$	$T+O(\ell)$
$q$ replacements, Theorem 4	$LR^{2}(\ell+q)/T+\sigma R((\ell+q)/T)^{a}$	$T$

Table 1: High-probability guarantees for the final update. Here

a=(p-1)/p

\ell\asymp\log(1/\delta)

Last-Iterate Convergence

Last-iterate convergence studies whether an algorithm’s final update approaches an optimizer or equilibrium, rather than only its average.

See all 42 papers

On the Last-Iterate Convergence of Clipped Gradient Methods
Oct 14, 2026Zijian LiuLast-Iterate Convergence Gradient Clipping
New York University
The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise
Oct 24, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaLast-Iterate Convergence Convergence Analysis
Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China
Weak- $L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
Oct 20, 2026Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov +4Gradient Clipping Heavy-Tailed Stochastic Optimization
Independent researcher · Yaroslavl State University · Moscow Independent Research Institute of Artificial Intelligence +2

Oct 14, 2026J/K move · Enter open · S save

On the Last-Iterate Convergence of Clipped Gradient Methods

Zijian Liu

New York University

Last-Iterate Convergence Gradient Clipping Heavy-Tailed Stochastic Optimization Stochastic Gradient Descent

Open paper AskPDF arXiv

In stochastic (strongly) convex optimization, establishing last-iterate convergence guarantees for first-order methods has recently garnered increasing attention, as this not only deepens the theoretical understanding of these algorithms but also aligns more closely with practice. Notably, people have demonstrated that one of the most famous, simple, and popular methods, Stochastic Gradient Descent (

\mathtt{SGD}

), provably achieves last-iterate convergence. However, the existing results may have limited applicability, since they typically rely on strong conditions on the gradient noise (e.g., an exponentially decaying tail) rather than the more realistic heavy-tailed noise. Unfortunately, when faced with heavy-tailed noise,

\mathtt{SGD}

is known to exhibit undesirable behavior or even fail to converge. To address the challenge of heavy-tailed noise, people have proposed

\mathtt{Clipped}\text{-}\mathtt{SGD}

, an algorithm that combines

\mathtt{SGD}

with a simple mechanism, gradient clipping. Although

\mathtt{Clipped}\text{-}\mathtt{SGD}

performs well in practice, its last-iterate convergence property remains highly underexplored. In this work, we prove the first optimal last-iterate rates in high probability for

\mathtt{Clipped}\text{-}\mathtt{SGD}

under heavy-tailed noise, thereby closing a gap in the literature.

Abstract

Last-Iterate Convergence

Explore similar work

On the Last-Iterate Convergence of Clipped Gradient Methods

The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise

Weak-LαL^αLα Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

Weak- $L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails