Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Enea Monzio Compagnoni; Alessandro Stanghellini; Rustem Islamov; Aurelien Lucchi; Anastasiia Koloskova

cs.LGMar 3, 2026

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Authors: Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov, Aurelien Lucchi, Anastasiia Koloskova

Organizations: University of Basel, Switzerland · University of Zürich, Switzerland

Abstract

Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with adaptivity in optimization through the lens of stochastic differential equations, providing the first SDE-based analysis of private optimizers. Focusing on DP-SGD and DP-SignSGD under per-example clipping, we show a sharp contrast under fixed hyperparameters: DP-SGD converges at a Privacy-Utility Trade-Off of $\mathcal{O}(1/\varepsilon^2)$ with speed independent of $\varepsilon$ , while DP-SignSGD converges at a speed linear in $\varepsilon$ with an $\mathcal{O}(1/\varepsilon)$ trade-off, dominating in high-privacy or large batch noise regimes. By contrast, under optimal learning rates, both methods achieve comparable theoretical asymptotic performance; however, the optimal learning rate of DP-SGD scales linearly with $\varepsilon$ , while that of DP-SignSGD is essentially $\varepsilon$ -independent. This makes adaptive methods far more practical, as their hyperparameters transfer across privacy levels with little or no re-tuning. Empirical results confirm our theory across training and test metrics, and empirically extend from DP-SignSGD to DP-Adam.

Figures & tables

Figure 1: Empirical validation of the privacy-utility trade-off predicted by Thm. 4.1 and Thm. 4.3 , comparing DP-SGD , DP-SignSGD , and DP-Adam : Our focus is on verifying the functional dependence of the asymptotic loss levels in terms of

\varepsilon

. Left: On a quadratic convex function

f(x)=\tfrac{1}{2}x^{\top}Hx

, the observed empirical loss values perfectly match the theoretical predictions (Eq. 6 , Eq. 9 ). Center and Right: Logistic regressions on the IMDB dataset (center) and the StackOverflow dataset (right), confirm the same pattern: the utility of DP-SGD scales as

\tfrac{1}{\varepsilon^{2}}

, while the utility of DP-SignSGD scales linearly as

\tfrac{1}{\varepsilon}

. Across all settings, we observe that the insights obtained for DP-SignSGD extend to DP-Adam as well as to the test loss (see Figure C.7 ). For experimental details see Appendix C.2 .

Figure 2: Empirical validation of the convergence speeds predicted by Thm. 4.1 and Thm. 4.3 . We compare DP-SGD , DP-SignSGD , and DP-Adam as we train a logistic regression on the IMDB dataset ( Top Row ) and on the StackOverflow dataset ( Bottom Row ). In both tasks, we verify that when DP-SGD converges, its speed is unaffected by

\varepsilon

. As expected, it diverges when

\varepsilon

is too small. Regarding DP-SignSGD and DP-Adam , they are faster when

\varepsilon

is large and never diverge even when this is small. Crucially, Figure C.8 shows that these insights are also verified on the test loss. For experimental details see Appendix C.3 .

Figure 3: Logistic regression on IMDB Dataset: From left to right, we decrease the batch noise, i.e., increase the batch size, taking values

B\in\{48,56,64,72,80\}

: As per Theorem 4.5 , the privacy threshold

\varepsilon^{\star}

that determines when DP-SignSGD is more advantageous than DP-SGD shifts to the left. This confirms that if there is more noise due to the batch size, less privacy noise is needed for DP-SignSGD to be preferable over DP-SGD . For experimental details see Appendix C.4 .

Figure 4: Empirical verification of Thm. 4.6 and Thm. 4.7 under Protocol B on the IMDB dataset (Top Row) and on the StackOverflow dataset (Bottom Row) . We tune

(\eta,C)

of each optimizer for each

\varepsilon

and confirm that:

i)

all methods achieve comparable performance across privacy budgets;

ii)

the optimal

\eta

of DP-SGD scales linearly with

\varepsilon

, while that of adaptive methods is essentially

\varepsilon

-independent;

iii)

failing to sweep over the “best” range of learning rates causes DP-SGD to severely underperform, whereas adaptive methods are resilient. On the left , DP-SGD degrades sharply for small

\varepsilon

. Indeed, the right panels shows that the selected optimal

\eta

flattens out, while the theoretical one would have linearly decayed more: The “best”

\eta

was simply missing from the grid. A posteriori , re-running the sweep with a larger grid ( DP-SGD Tuned ) recovers the scaling law and matches the performance of adaptive methods. For experimental details see Appendix C.5 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Numerical validation of the approximation used in Equation 13 . For several values of

d

, we plot the confluent hypergeometric function as a function of the signal-to-noise ratio

z

. In the realistic range observed in [ Malladi et al., 2022 ] , approximating this function by

1

is extremely accurate.

Dataset	DP-SGD	DP-SignSGD	DP-Adam
IMDB	(5, 0.5)	(0.1, 0.5)	(0.1, 0.5)
StackOverflow	(3, 0.25)	(0.01, 0.5)	(0.01, 0.5)
MovieLens	(0.1, 2)	(0.001, 3)	(0.001, 1)

Appendix

Table C.1 : Tuned hyperparameters for different methods across the two datasets. The values refer to (learning rate, clipping parameter); For DP-Adam we also used

\beta_{1}=0.9,\beta_{2}=0.999

and adaptivity

\epsilon=10^{-8}

in both cases.

Figure C.1 : Consistent with Theorem B.9 and Theorem B.14 , we empirically validate that the SDEs of DP-SGD and DP-SignSGD model their respective optimizers. For a convex quadratic function ( left two panels ) and a nonconvex quartic function ( right two panels ), the SDEs accurately track both the trajectories and the gradient norm of the corresponding algorithms, averaged over 200 runs.

Figure C.2 : Empirical validation of the privacy-utility trade-off predicted by Thm. 4.1 and Thm. 4.3 , comparing DP-SGD , DP-SignSGD , and DP-Adam : Our focus is on verifying the functional dependence of the asymptotic loss levels in terms of

\varepsilon

. Train MSE for matrix factorization on MovieLens-100k, confirm the same pattern: the utility of DP-SGD scales as

\tfrac{1}{\varepsilon^{2}}

, while the utility of DP-SignSGD scales linearly as

\tfrac{1}{\varepsilon}

. We observe that the insights obtained for DP-SignSGD extend to DP-Adam as well as to the test loss.

Figure C.3 : Empirical validation of the convergence speeds predicted by Thm. 4.1 and Thm. 4.3 . We compare DP-SGD , DP-SignSGD , and DP-Adam as we optimize a non-convex matrix factorization problem on the MovieLens dataset. We verify that when DP-SGD converges, its speed is unaffected by

\varepsilon

. As expected, it diverges when

\varepsilon

is too small. Regarding DP-SignSGD and DP-Adam , they are faster when

\varepsilon

is large and never diverge even when this is small.

Figure C.4 : ( Top Row ) StackOverflow and ( Bottom Row ) MovieLens: From left to right, we decrease the batch noise, i.e., increase the batch size, taking values

B=48,\,56,\,64

: As per Theorem 4.5 , the privacy threshold

\varepsilon^{\star}

Figure C.5 : Empirical verification of Thm. 4.6 and Thm. 4.7 under Protocol B on the MovieLens dataset. We tune

(\eta,C)

of each optimizer for each

\varepsilon

and confirm that:

i)

all methods achieve comparable performance across privacy budgets;

ii)

the optimal

\eta

of DP-SGD scales linearly with

\varepsilon

, while that of adaptive methods is essentially

\varepsilon

-independent;

iii)

failing to sweep over the “best” range of learning rates causes DP-SGD to severely underperform, whereas adaptive methods are resilient. On the left , DP-SGD degrades sharply for small

\varepsilon

. Indeed, the right panels shows that the selected optimal

\eta

flattens out, while the theoretical one would have linearly decayed more: The “best”

\eta

was simply missing from the grid. A posteriori , re-running the sweep with a larger grid ( DP-SGD Tuned ) recovers the scaling law and matches the performance of adaptive methods.

Figure C.6 : The empirical dynamics of the first and second moments of the iterates

X_{t}

of DP-SGD (left two panels) and of DP-SignSGD (right two panels) match that prescribed in Theorem B.13 and Theorem B.21 , respectively.

Figure C.7 : Privacy-utility trade-off on the test loss , comparing DP-SGD , DP-SignSGD , and DP-Adam . Left: Logistic regression on the IMDB dataset. Right: Logistic regression on the StackOverflow dataset. In both cases, the empirical scalings predicted by Thm. 4.1 and Thm. 4.3 carry over from training to test: DP-SGD follows the

\tfrac{1}{\varepsilon^{2}}

trend, while adaptive methods follow the

\tfrac{1}{\varepsilon}

trend. This demonstrates that not only do our theoretical insights generalize to the widely used DP-Adam , but also extend from training to test loss.

Figure C.8 : We compare the Test Loss of DP-SGD , DP-SignSGD , and DP-Adam as we train a logistic regression on the IMDB dataset ( Top Row ) and on the StackOverflow dataset ( Bottom Row ).

Explore similar work

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning
May 19, 2026Mohammad Partohaghighi, Roummel MarciaDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Department of Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA · Department of Applied Mathematics University of California, Merced Merced, CA, USA
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient Descent Gradient Clipping
Arizona State University, Tempe, AZ, United States
Revisiting the Provable-Auditable Privacy Gap of DP-SGD
Aug 28, 2026Saloni Modi, Srivi Balaji, Yusong Zhu +2Differential Privacy Differentially Private Stochastic Gradient Descent
University of Texas at Austin · University of Waterloo and Vector Institute

May 19, 2026J/K move · Enter open · S save

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

Mohammad Partohaghighi, Roummel Marcia

Department of Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA · Department of Applied Mathematics University of California, Merced Merced, CA, USA

Deep Learning Optimization Differentially Private Stochastic Gradient Descent Differential Privacy

Open paper AskPDF arXiv

Differentially private stochastic gradient descent (DP-SGD) enables private deep learning through per-example clipping and calibrated Gaussian noise, but its high-variance updates can reduce utility on challenging datasets. We propose SMA-DP-SGD, a Spectral Memory-Aware Differentially Private Stochastic Gradient Descent method that augments DP-SGD with a fractional memory branch built only from previously privatized noisy releases. WeightWatcher-inspired power-law spectral exponents provide group-wise reliability signals, instantiated layer-wise in our experiments, to adapt the decay and effective memory depth. Private-history alignment, norm matching, and warm-up activation stabilize the memory contribution. Privacy remains transparent: conditioned on the private release history, the memory branch is fixed, and the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient

β

. Hence, SMA-DP-SGD preserves a clean conditional sensitivity structure and exactly recovers group-wise DP-SGD when

β=1

. Experiments on CIFAR-100, CIFAR-10, and MNIST show competitive or superior accuracy over several DP optimization baselines, with the largest gains on CIFAR-100 and CIFAR-10. CIFAR-10 ablations show that

β

controls the privacy--utility trajectory, while spectral and memory diagnostics confirm a controlled short-to-moderate effective memory depth and a small memory-branch ratio. Runtime analysis shows that the mechanism incurs additional overhead, about

2.94\times

DP-SGD in our CIFAR-10 implementation, revealing a practical trade-off between adaptive private memory and computational cost.

Abstract

Explore similar work

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

Revisiting the Provable-Auditable Privacy Gap of DP-SGD