Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

Xincheng Xu; Thilina Ranbaduge; Thierry Rakotoarivelo; David Smith

cs.LGOct 23, 2026

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

Authors: Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo, David Smith

Organizations: Australian National University, Australia · CSIRO, Australia

Abstract

Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy guarantees. However, the utility of DPSGD is constrained by gradient clipping bias and DP noise. While existing work has tried to address these issues, the geometric misalignment caused by clipping bias remains a critical challenge. To bridge this gap, we propose a novel framework, Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), which leverages approximate second-order information to construct an optimal, curvature-aware clipping space. CA-DPSGD integrates a rectified estimator for the curvature matrix and is coupled with an anisotropic noise mechanism to ensure privacy guarantees. Extensive empirical experiments demonstrate that our approach outperforms state-of-the-art clipping methods.

Figures & tables

Dataset	Model	$(\epsilon,\delta)$	DPSGD	Auto-S	DP-PSAC	CA-DPSGD
MNIST	4-layer CNN	$(3,1\text{e-}5)$	98.07 $\pm$ 0.10	98.13 $\pm$ 0.04	98.17 $\pm$ 0.10	98.32 $\pm$ 0.12
Fashion-MNIST	4-layer CNN	$(3,1\text{e-}5)$	86.45 $\pm$ 0.42	86.46 $\pm$ 0.07	86.56 $\pm$ 0.13	86.72 $\pm$ 0.18
CIFAR-10	SimCLRv2	$(2,1\text{e-}5)$	92.62 $\pm$ 0.03	92.57 $\pm$ 0.10	92.52 $\pm$ 0.12	92.82 $\pm$ 0.09
ImageNette	ResNet9	$(8,1\text{e-}4)$	60.91 $\pm$ 0.71	60.97 $\pm$ 0.57	61.39 $\pm$ 0.53	62.21 $\pm$ 0.47
CelebA	ResNet9	$(8,5\text{e-}6)$	88.49 $\pm$ 0.13	88.61 $\pm$ 0.13	88.55 $\pm$ 0.12	88.67 $\pm$ 0.12
MNLI	RoBERTa-base	$(8,1\text{e-}5)$	80.43 $\pm$ 0.03	80.31 $\pm$ 0.39	80.54 $\pm$ 0.16	80.58 $\pm$ 0.27

Table 1: Test accuracy (%) comparison on various datasets.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Test accuracy (%) of IOM, KFSGD, and DP-AdamBC with/without CA-DPSGD on five image classification datasets. The prefix ”CA-” denotes the method is integrated with CA-DPSGD (e.g., CA-IOM represents IOM combined with CA-DPSGD).

Panel of Figure 2 — Figure 2: Ablation study on test accuracy (%) of CA-DPSGD on three image classification datasets: MNIST, Fashion-MNIST, CIFAR-10. (a) Comparison of CA-DPSGD and CA-DPSGD without Noise Correction (w/o NC). (b) Comparison between the proposed adaptive $\lambda$ mechanism and fixed $\lambda$ values ( $\lambda^{2}\in\{10^{-6},10^{-7},10^{-8}\}$ ).

Panel of Figure 3 — Figure 3: Hyperparameter analysis of $\beta$ on different datasets. (a) Results on Imagenette and (b) Results on Fashion-MNIST. In both cases, the left panels show the stability of $v_{t}$ via $\log_{10}(\|v_{t+1}-v_{t}\|_{1})$ , while the right panels show training loss curves. Larger $\beta$ leads to more stable momentum updates but slower convergence.

Notation	Description
$\epsilon$	Privacy budget
$\delta$	Failure factor
$x_{t}$	Model parameters at iteration $t$
$d$	Dimensionality of model parameters
$N$	Total number of samples in dataset
$B$	Batch size

Appendix

Table 2: Table of Notations

Differentially Private Stochastic Gradient Descent

Differentially private stochastic gradient descent (DP-SGD) protects training data by clipping per-example gradients, adding calibrated noise, and accounting for privacy loss.

See all 52 papers

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient Descent Gradient Clipping
Arizona State University, Tempe, AZ, United States
Differentially Private Natural Gradient Descent
Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Gradient Descent Natural Gradient Methods
Institute of Information Engineering, Chinese Academy of Sciences Beijing, China · Department of Computer Science, Metropolitan College, Boston University Boston, United States · Nanyang Technological University Singapore, Singapore
Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD
Jun 3, 2026Xiaobo Huang, Fang XieDifferentially Private Stochastic Gradient Descent Differential Privacy
Guangdong Provincial Key Laboratory of IRADS, Beijing · Normal-Hong Kong Baptist University · Zhuhai, China

Jun 3, 2026J/K move · Enter open · S save

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

Naima Tasnim, Lalitha Sankar, Oliver Kosut

Arizona State University, Tempe, AZ, United States

Differentially Private Stochastic Gradient Descent Gradient Clipping Differential Privacy Adaptive Gradient Methods

Open paper AskPDF arXiv

Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a bias-free manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DP-Adam baselines, without requiring manual tuning of the clipping threshold.

Abstract

Differentially Private Stochastic Gradient Descent

Explore similar work

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

Differentially Private Natural Gradient Descent

Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD