Differentially Private Stochastic Gradient Descent (DP-SGD) is a cornerstone technique for ensuring privacy in deep learning, widely used in both training from scratch and fine-tuning large-scale language models. While DP-SGD predominantly relies on the Gaussian mechanism, the Laplace mechanism remains underutilized due to its reliance on L1 norm clipping. This constraint severely limits its practicality in high-dimensional models because the L1 norm of an n-dimensional gradient can be up to sqrt(n) times larger than its L2 norm. As a result, the required noise scale grows significantly with model size, leading to poor utility or untrainable models. In this work, we introduce Lap2, a new solution that enables L2 clipping for Laplace DP-SGD while preserving strong privacy guarantees. We overcome the dimensionality-driven clipping barrier by computing coordinate-wise moment bounds and applying majorization theory to construct a tight, data-independent upper bound over the full model. By exploiting the Schur-convexity of the moment accountant function, we aggregate these bounds using a carefully designed majorization set that respects the L2 clipping constraint. This yields a multivariate privacy accountant that scales gracefully with model dimension and enables the use of thousands of moments. Empirical evaluations demonstrate that our approach significantly improves the performance of Laplace DP-SGD, achieving results comparable to or better than Gaussian DP-SGD under strong privacy constraints. For instance, fine-tuning RoBERTa-base (125M parameters) on SST-2 achieves 87.88% accuracy at epsilon=0.54, outperforming Gaussian (87.16%) and standard Laplace (48.97%) under the same budget.
Figures & tables
Fig. 1 : Comparison of ℓ1 and ℓ2 norm clipped spaces.
Fig. 2 : Volume of n -dimensional vector space clipped by C - ℓ1 and C - ℓ2 norms.
Fig. 3 : In Gaussian DP-SGD, the so-called privacy wall leads to sharp increases in the noise multiplier under tight privacy ( ϵ<0.5 ). This arises because the injected noise scales with σ⋅C , which becomes problematic in settings like pretraining where large C is needed to preserve gradient signal—resulting in excessive noise. In contrast, Lap 2 supports larger noise or larger clipping norms (see Appendix B-A ), due to its heavy tails and efficient high-order moments accounting.
Fig. 4 : Illustration of two-sided privacy walls (left and right walls) during CNN training on the MNIST dataset under DP-SGD. The left wall corresponds to the high-privacy regime, where the noise scale saturates and δ(ϵ) approaches 1, indicating limited further privacy gain. The right wall corresponds to the low-privacy regime, where the effective signal-to-noise ratio no longer improves with larger ϵ . Together, these walls define the practical operating range (privacy corridor) in which training remains both private and useful.
Fig. 5 : Accuracy vs. privacy budget ϵ for different clipping norms C using Lap 2 (CNN on MNIST, δ=10−5 ). Increasing C improves utility under moderate ϵ due to stronger signal retention, but high C becomes suboptimal for very tight ϵ as noise grows superlinearly.
Fig. 6 : Accuracy of a CNN trained on MNIST under DP-SGD with Lap 2 for 20 epochs. Configurations (q,b,c) were selected via grid search for each target privacy budget ϵ∈{0.5,1,2} and δ=10−5 , with color indicating the resulting accuracy. The star marks the configuration achieving maximum accuracy for each ϵ .
max values Cmax,bmax ; resolution ΔC , binary tolerance τ
Output: optimal (C∗,b∗) maximizing C under (ϵ,δ)
C∗←0 , b∗←None
for C in [Cmin,Cmax] step ΔC do
blow←bmin,bhigh←bmax
while bhigh−blow>τ do
Algorithm 1 Lap 2 Parameter b Optimizer (via Binary Search)
Dataset
Model
Sampling
Clipping
δ
Learning
Label
Weight
Training
Rate
Threshold
Rate
Smoothing
Decay
Steps
MNIST
CNN
0.0043
1
1×10−5
1×10−3
0.15
1×10−4
5860
Fashion-MNIST
CNN
0.0043
1
1×10−5
1×10−3
0.15
1×10−4
5860
CIFAR-10
ViT
0.01668
1
1×10−5
1×10−3
0
1×10−4
900
TABLE I : Hyperparameter settings for different CV datasets and models.
ϵ
Gaussian (%)
Laplace (%)
Lap 2 (%)
CNN on MNIST
3.42
97.27
51.82
96.82
2.53
97.19
46.67
96.90
1.68
97.42
28.82
95.39
0.88
96.08
16.44
93.29
0.13
87.44
10.40
78.96
TABLE II : Comparison of Gaussian, Laplace ( ℓ1 norm), and Lap 2 ( ℓ2 norm) mechanisms across different ϵ .
ϵ
Gaussian (%)
Laplace (%)
Lap 2 (%)
RoBERTa-base on SST-2
3.7382
90.31
50.34
90.11
1.1584
89.65
50.21
83.32
0.9108
89.23
50.15
89.14
0.5384
87.16
48.97
87.88
RoBERTa-base on QNLI
TABLE III : Comparison of Gaussian, Laplace ( ℓ1 norm), and Lap 2 ( ℓ2 norm) mechanisms across different privacy levels ( ϵ ) for RoBERTa-base on SST-2 and QNLI.
Fig. 7 : Gaussian and Lap 2 mechanisms on the DistilGPT-2 model and E2E dataset for the generation task (batch size B=80 , clipping value C=2 , and ϵ=1 ).
Fig. 8 : Runtime comparison on CNN model with MNIST or FMNIST datasets.
Metric
Noise Type
ϵ=1.2
ϵ=1
ϵ=0.4
BLEU
Gaussian
23.39
22.83
16.16
Lap 2
27.99
27.34
26.90
NIST
Gaussian
3.9474
3.5342
2.7025
Lap 2
4.4001
4.3100
4.1200
METEOR
Gaussian
16.84
16.73
15.20
Lap 2
20.50
20.20
19.80
TABLE IV : Metrics of Gaussian and Lap 2 mechanisms applied to the DistilGPT-2 model on the E2E dataset for the generation task.
Input: Dataset D , steps T , sampling rate ζ , learning rate η ,
target privacy (ϵ,δ)
Output: Final model parameters θ under (ϵ,δ) -DP
// Step 1: Optimize (C,b) using Algorithm 1
(C,b)←H(ϵ,δ,T,ζ)
// Step 2: DP-SGD with ℓ2 clipping and Laplace noise
for t=1 to T do
Appendix
Algorithm 2 Lap 2 DP-SGD: Drop-in Replacement of Gaussian DP-SGD
Fig. 9 : Lap 2 supports larger clipping norms without runaway noise, due to its heavy tails and efficient high-order moments accounting.
Fig. 10 : Accuracy vs batch size for various ϵ . Found by fine-tuning a ViT with CIFAR10 trained for 1 epoch. THis graph shows that there is an optimal batch size, with a slow decay in accuracy afterward. Under gaussian, the decay is slow and consistent across batch sizes [ 14 ] .
Fig. 11 : Accuracy vs batch size for various clippling norms on the MNIST dataset trained on CNN using Lap 2 -DPSGD for 20 epochs for a fixed privacy budget ϵ=2 and δ=10−5 .
Differentially private stochastic gradient descent (DP-SGD) protects training data by clipping per-example gradients, adding calibrated noise, and accounting for privacy loss.