Gradient Clipping

Follow this topic Ask about this topic

Gradient clipping caps the size of gradients or gradient components before an optimizer updates model parameters, helping prevent outliers from destabilizing training. Researchers study norm, coordinate-wise, and spectral clipping, as well as adaptive thresholds, often proving convergence for SGD and Adam under heavy-tailed noise and testing methods on neural-network training. Clipping is also used in differential privacy, federated and asynchronous optimization, and large-model pretraining. A central tension is that clipping improves robustness but introduces bias and can hurt accuracy, so theory and practice seek rules that control outliers without discarding useful gradient information.

Latest papers

22 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information
Oct 23, 2026Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo +1Differentially Private Stochastic Gradient Descent Gradient Clipping
Australian National University, Australia · CSIRO, Australia
Weak- $L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
Oct 20, 2026Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov +4Gradient Clipping Heavy-Tailed Stochastic Optimization
Independent researcher · Yaroslavl State University · Moscow Independent Research Institute of Artificial Intelligence +2
On the Last-Iterate Convergence of Clipped Gradient Methods
Oct 14, 2026Zijian LiuLast-Iterate Convergence Gradient Clipping
New York University
Last-Iterate Convergence of Clipped SGD with Linear Decay
Oct 13, 2026Wei Jiang, Haomin Bai, Zechao LiLast-Iterate Convergence Stochastic Gradient Descent
School of Computer Science and Engineering, Nanjing University of Science and Technology · National Key Laboratory for Novel Software Technology, Nanjing University
Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
Oct 7, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedDecentralized Optimization Stochastic Gradient Descent
Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise
Sep 26, 2026Alexandra Suvorikova, Egor Gladin, Darina Dvinskikh +5Polyak-Ruppert Averaging Stochastic Optimization
Independent Researcher · HSE University · MBZUAI +3
Precise Convergence Speed of Clipped SGD
Sep 24, 2026David A. R. RobinGradient Clipping Nonconvex Stochastic Optimization
LAMSADE, Université Paris-Dauphine, PSL Research University
Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and Hölder Smoothness
Sep 11, 2026Misbah Uz Zaman, Anirbit MukherjeeGradient Clipping Nonconvex Stochastic Optimization
Department of Mathematics and Statistics Indian Institute of Science Education and Research Kolkata · Department of Computer Science The University of Manchester
Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
Sep 10, 2026Amartya Mukherjee, Jun LiuStochastic Optimization Gradient Clipping
Department of Applied Mathematics, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1
TruncGradGS: Improved 3D Gaussian Splatting via Truncated Gradient Updates
Sep 3, 2026Theo Morales, Nhat-Quynh Le-Pham, Robin Atkins +13D Gaussian Splatting Gradient Clipping
Trinity College Dublin · Dolby Laboratories
Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness
Aug 12, 2026Alokendu Mazumder, Ayaan Mohd, Harshit Rawat +3Stochastic Optimization Convergence Adaptive Gradient Methods
Robert Bosch Center for Cyber Physical Systems, IISc Bengaluru · Undergraduate Program, IISc Bengaluru · Department of Computational Data Sciences, IISc Bengaluru +4
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
Jul 20, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungNeural Network Stability Gradient Clipping
College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers
Jun 11, 2026Samuel Erickson, Mikael JohanssonGradient Clipping Stochastic Optimization Convergence
School of EECS, KTH Royal Institute of Technology, Stockholm, Sweden.
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient Descent Gradient Clipping
Arizona State University, Tempe, AZ, United States
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
May 26, 2026Zitao Song, Cedar Site Bai, Zhe Zhang +2Gradient Clipping Heavy-Tailed Noise
Department of Computer Science, Purdue University, West Lafayette, IN, USA. · School of Industrial Engineering, Purdue University, West Lafayette, IN, USA
Zeroth-Order Nonconvex Nonsmooth Optimization with Heavy-Tailed Noise
May 23, 2026Zhuanghua Liu, Luo LuoStochastic Optimization Gradient Clipping
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
May 18, 2026Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1Language Model Pretraining Gradient Clipping
University of Minnesota
Stochastic Non-Smooth Convex Optimization with Unbounded Gradients
May 15, 2026Dmitry KovalevStochastic Optimization Gradient Clipping
Yandex Research
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
May 12, 2026Alexander Yukhimchuk, Mladen Kolar, Martin Takáč +1Stochastic Optimization Gradient Clipping
MBZUAI · University of Southern California
From Dual Tracking to Clipping: Provably Faster Distributionally Robust Multi-Objective Optimization
May 7, 2026Yufeng Yang, Fangning Zhuo, Ziyi Chen +2Gradient Clipping Gradient Descent
Texas A&M University · A*STAR · University of Maryland
Robust and Fast Training via Per-Sample Clipping
May 4, 2026Davide Nobile, Philipp GrohsGradient Clipping Nonconvex Stochastic Optimization
Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences
Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theory
Feb 26, 2026Meisam Mohammady, Qin Yang, Nicholas Stout +4Differentially Private Stochastic Gradient Descent Gradient Clipping
Iowa State University · University of Connecticut · University of Kansas

Oct 23, 2026J/K move · Enter open · S save

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo, David Smith

Australian National University, Australia · CSIRO, Australia

Differentially Private Stochastic Gradient Descent Gradient Clipping Curvature-Aware Optimization

Open paper AskPDF arXiv

Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy guarantees. However, the utility of DPSGD is constrained by gradient clipping bias and DP noise. While existing work has tried to address these issues, the geometric misalignment caused by clipping bias remains a critical challenge. To bridge this gap, we propose a novel framework, Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), which leverages approximate second-order information to construct an optimal, curvature-aware clipping space. CA-DPSGD integrates a rectified estimator for the curvature matrix and is coupled with an anisotropic noise mechanism to ensure privacy guarantees. Extensive empirical experiments demonstrate that our approach outperforms state-of-the-art clipping methods.