Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale

Sabari Shanmugam; Nick Barnes; Kerry Taylor

cs.CVAug 18, 2026

Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale

Authors: Sabari Shanmugam, Nick Barnes, Kerry Taylor

Organizations: School of Computing, Australian National University, Canberra, Australia

Abstract

Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal. Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step that recovers directional signal from the noisy gradient's low-rank structure at zero additional privacy cost. A phase transition governs the utility of this approach: orthogonalization improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery; in low-SNR regimes, the directional bias of the gradient is replaced by a nearly random orthogonal update, and the transformation is harmful. The recovery threshold is determined by the spectral gap of the gradient and is surpassed at large batch sizes. Empirically, the benefit scales with model capacity: spectral orthogonalization achieves a +20.9% improvement over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In the fine-tuning regime, spectral orthogonalization matches the stability of DP-Adam while maintaining a first-order memory footprint. Combining spectral with temporal denoising yields 50.3% on CIFAR-10 (epsilon = 4), the highest accuracy in any tested configuration. These gains are specific to moderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served by DP-SGD or temporal denoising.

Figures & tables

Figure 1: Spectral gradient orthogonalization overview. (a) Gradient singular values from a representative convolutional layer (epoch 1, CIFAR-10): most energy concentrates in a small number of directions (effective rank

6

). (b) After DP noise addition (

\varepsilon=4

), isotropic perturbation raises the effective rank to

483

: the signal becomes indistinguishable from the noise-dominated singular values (Marchenko–Pastur bulk), making the noisy gradient nearly directionless. (c) Direction recovery: cosine similarity between the leading singular vector of the noisy and clean gradients, measured via Monte Carlo simulation (100 trials per batch size) using the measured WRN-16-4 spectrum with noise calibrated to table 1 . High similarity indicates that the gradient’s directional signal persists through the noise, providing the geometric condition for orthogonalization to succeed. Recovery transitions sharply from

0.90

B\!=\!256

{\geq}0.99

B\!\geq\!1024

. (d) Test accuracy on CIFAR-10 (

\varepsilon=4

) follows the same pattern: DP-SGD stagnates while spectral methods scale with batch size past the recovery threshold.

Figure 2: Gradient SNR vs. batch size at

\varepsilon=4

. (a) CIFAR-10 SNR with signal and noise Frobenius norms; the horizontal dashed line marks

\text{SNR}=1

, above which orthogonalization recovers useful gradient structure. (b) CIFAR-10 vs. CIFAR-100: the SNR gap widens with batch size, reaching

{\sim}9\times

B=4096

$B$	$\sigma$	$\\|G\\|_{F}$ (signal)	$\\|N\\|_{F}$ (noise)	SNR (C-10)	SNR (C-100)
256	0.88	0.33	5.71	0.06	0.05
512	1.08	0.34	3.50	0.10	0.07
1024	1.38	0.57	2.23	0.25	0.09
2048	1.83	1.00	1.48	0.68	0.12
4096	2.49	1.76	1.01	$\mathbf{1.75}$	0.20

Table 1: Measured gradient SNR (

\|G\|_{F}/\|N\|_{F}

) for WRN-16-4 at

\varepsilon=4

, averaged over the first epoch.

\sigma

is the noise multiplier (identical for both datasets at

N=50\text{K}

). Signal and noise norms are shown for CIFAR-10. As training progresses, gradient structure typically becomes more low-rank, which lowers the recovery threshold.

Figure 3: Test accuracy vs. batch size at

\varepsilon=4

across three datasets. (a) CIFAR-10: DP-SGD is flat while spectral methods scale upward (

+9.4\%

B=4096

). (b) CIFAR-100: the crossover shifts to larger batch sizes, consistent with the lower SNR. (c) SVHN: DP-SGD degrades with batch size while DP-Muon improves (

+14.9\%

B=4096

). Vertical dotted lines mark the approximate

\text{SNR}\approx 1

threshold. Mean over three seeds.

Method	$B\!=\!256$	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$
DP-SGD	$39.4\pm 1.2$	$39.4\pm 0.7$	$39.7\pm 2.6$	$37.0\pm 2.6$	$39.3\pm 0.3$
DP-Adam	$\mathbf{39.9\pm 2.9}$	$41.0\pm 1.1$	$41.8\pm 1.8$	$42.0\pm 0.5$	$38.8\pm 1.7$
DP-Muon	$39.0\pm 0.5$	$40.5\pm 1.7$	$43.3\pm 1.6$	$47.3\pm 1.8$	$\mathbf{48.7\pm 0.6}$
DP-Muon-S	$38.7\pm 1.0$	$\mathbf{44.0\pm 1.9}$	$\mathbf{44.5\pm 2.9}$	$\mathbf{47.8\pm 1.7}$	$46.3\pm 0.8$

Table 2: Test accuracy (%) on CIFAR-10 at

\varepsilon=4

. Mean

\pm

std over three seeds. Best per column in bold .

Method	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$	$B\!=\!8192$
DP-SGD	$23.5\pm 1.0$	$26.6\pm 2.6$	$\mathbf{29.3\pm 1.2}$	$\mathbf{26.8\pm 0.6}$	$21.6\pm 0.9$
DP-Adam	$\mathbf{25.5\pm 1.3}$	$\mathbf{27.4\pm 1.8}$	$28.9\pm 0.7$	$26.0\pm 0.7$	$18.6\pm 1.2$
DP-Muon	$19.4\pm 1.7$	$22.0\pm 1.0$	$25.8\pm 0.7$	$25.9\pm 0.5$	$\mathbf{25.8\pm 0.6}$
DP-Muon-S	$20.9\pm 1.2$	$24.8\pm 0.3$	$26.5\pm 0.7$	$26.3\pm 0.9$	$22.6\pm 1.0$

Table 3: Test accuracy (%) on CIFAR-100 at

\varepsilon=4

with tuned learning rates (DP-SGD

\eta\!=\!1.0

, DP-Adam

\eta\!=\!0.01

, DP-Muon

\eta\!=\!0.1

, DP-Muon-S

\eta\!=\!0.5

). Mean

\pm

std over three seeds. Best per column in bold .

Figure 4: Test accuracy vs.

\varepsilon

B=2048

across three datasets. DP-SGD stagnates on CIFAR-10 and SVHN (shaded bands) while spectral methods scale with the privacy budget. DP-Muon-S is strongest on CIFAR-100 at all

\varepsilon

. Mean over three seeds.

Figure 5: Temporal vs. spectral denoising on CIFAR-10 at

\varepsilon=4

. Temporal methods (DiSK-SGD, dashed brown) dominate at small batch sizes but decline as fewer gradient steps reduce the filter’s averaging window. Spectral methods (DP-Muon, solid green) improve with batch size as per-step SNR increases. DOPPLER-Muon (dotted pink) combines both mechanisms and reaches the highest accuracy (

50.3\%

) at

B=4096

. Error bars show

\pm

1 std over three seeds.

Denoising	Method	$B\!=\!256$	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$
None	DP-SGD	$39.4\pm 1.2$	$39.4\pm 0.7$	$39.7\pm 2.6$	$37.0\pm 2.6$	$39.3\pm 0.3$
Temporal	DiSK-SGD	$\mathbf{49.0\pm 2.8}$	$\mathbf{47.8\pm 1.0}$	$46.7\pm 1.3$	$43.5\pm 2.0$	$39.9\pm 0.6$
Temporal	DOPPLER-SGD	$37.7\pm 2.0$	$39.3\pm 0.9$	$39.3\pm 2.0$	$34.2\pm 1.8$	$32.3\pm 1.1$
Spectral	DP-Muon	$39.0\pm 0.5$	$40.5\pm 1.7$	$43.3\pm 1.6$	$47.3\pm 1.8$	$48.7\pm 0.6$
Both	DiSK-Muon	$41.7\pm 2.7$	$47.0\pm 1.0$	$\mathbf{48.8\pm 1.0}$	$47.6\pm 0.8$	$45.8\pm 1.2$
Both	DOPPLER-Muon	$40.1\pm 1.6$	$42.6\pm 0.9$	$44.9\pm 2.0$	$\mathbf{49.0\pm 0.5}$	$\mathbf{50.3\pm 1.0}$

Table 4: Temporal vs. spectral denoising on CIFAR-10 (

\varepsilon=4

). DiSK-SGD applies the Kalman filter [ 40 ] (

\kappa=1.5

) to DP-SGD. DOPPLER-Muon combines low-pass filtering [ 41 ] with spectral orthogonalization. Mean

\pm

std over three seeds.

Method	LR	$B\!=\!2048$	$B\!=\!4096$
DP-SGD	0.01	$86.9\pm 1.1$	$79.5\pm 3.4$
DP-Adam	0.0003	$\mathbf{89.4\pm 0.3}$	$\mathbf{89.7\pm 0.1}$
DP-Muon	0.002	$89.1\pm 0.2$	$89.3\pm 0.3$

Table 5: Two-point settings, best per column in bold . (Left) Full fine-tuning of ViT-Small on CIFAR-100 (

\varepsilon{=}4

), mean

\pm

std over three seeds. (Right) From-scratch NF-ResNet-50 on ImageNet-1k (

B{=}32{,}768

, 120 epochs,

K{=}4

, three seeds).

Figure 6: Batch-size sweeps as smooth curves. (a) Tiny-ImageNet (WRN-16-4,

\varepsilon{=}4

). (b) Architecture sweep on CIFAR-10 (

\varepsilon{=}4

); ResNet-18 (solid), WRN-28-10 (dashed). Markers are measured points; curves are monotone cubic (PCHIP) interpolation.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Method	LR	Accuracy
DP-SGD	0.3	$25.2\pm 0.9$
DP-Muon	0.02	$21.5\pm 0.5$
DP-Muon	0.1	$23.2^{\dagger}$
DP-Muon	0.3	$16.8\pm 0.7$
DP-Muon-S	0.3	$\mathbf{25.6\pm 0.6}$

Appendix

Table 6: Learning rate sweep on CIFAR-100 at

B=2048

\varepsilon=4

. Entries with

\pm

are mean over three seeds;

\dagger

denotes a single-seed run from a controlled sweep on the same hardware.

$\varepsilon$	DP-SGD	DP-Adam	DP-Muon	DP-Muon-S
2	$36.7\pm 0.4$	$40.6\pm 2.6$	$41.2\pm 1.7$	$\mathbf{42.6\pm 1.6}$
3	$37.0\pm 1.3$	$40.7\pm 1.3$	$45.7\pm 0.5$	$\mathbf{48.0\pm 3.4}$
4	$37.0\pm 2.6$	$42.0\pm 0.5$	$47.3\pm 1.8$	$\mathbf{47.8\pm 1.7}$
6	$40.0\pm 2.0$	$41.8\pm 0.7$	$\mathbf{50.0\pm 1.1}$	$48.5\pm 0.9$
8	$39.1\pm 1.9$	$43.6\pm 1.0$	$\mathbf{50.8\pm 2.9}$	$49.6\pm 1.3$

Appendix

Table 7: Test accuracy (%) on CIFAR-10 at

B=2048

across privacy budgets. Mean

\pm

std over three seeds. Best per row in bold .

$\varepsilon$	DP-SGD	DP-Adam	DP-Muon	DP-Muon-S	$\Delta$
2	$\mathbf{21.3\pm 0.1}$	$14.3\pm 0.8$	$16.7\pm 0.2$	$20.7\pm 1.0$	$-0.6$
3	$23.5\pm 0.5$	$15.7\pm 0.4$	$19.6\pm 0.3$	$\mathbf{24.0\pm 0.3}$	$+0.5$
4	$25.4\pm 0.6$	$16.6\pm 0.5$	$21.6\pm 0.3$	$\mathbf{26.4\pm 0.3}$	$+1.0$
6	$27.5\pm 1.0$	$18.7\pm 0.2$	$24.7\pm 1.2$	$\mathbf{28.7\pm 1.0}$	$+1.2$
8	$28.1\pm 0.6$	$20.0\pm 0.1$	$26.2\pm 1.6$	$\mathbf{31.1\pm 0.6}$	$+3.0$

Appendix

Table 8: Test accuracy (%) on CIFAR-100 at

B=2048

across privacy budgets. Mean

\pm

std over three seeds.

\Delta

denotes DP-Muon-S minus DP-SGD.

$\varepsilon$	DP-SGD	DP-Adam	DP-Muon
2	$20.9\pm 1.5$	$24.3\pm 3.2$	$\mathbf{28.6\pm 0.4}$
3	$20.2\pm 0.4$	$23.2\pm 3.0$	$\mathbf{29.7\pm 1.6}$
4	$20.1\pm 0.5$	$22.5\pm 3.4$	$\mathbf{30.0\pm 2.4}$
6	$20.1\pm 0.5$	$22.5\pm 4.4$	$\mathbf{32.8\pm 2.4}$
8	$20.2\pm 0.5$	$22.9\pm 4.0$	$\mathbf{35.3\pm 2.9}$

Appendix

Table 9: Test accuracy (%) on SVHN at

B=2048

across privacy budgets. Mean

\pm

std over three seeds.

Method	$B\!=\!256$	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$
DP-SGD	$\mathbf{38.6\pm 12.4}$	$26.5\pm 3.0$	$21.3\pm 1.8$	$20.1\pm 0.5$	$20.3\pm 0.4$
DP-Adam	$29.0\pm 3.4$	$27.1\pm 1.4$	$24.4\pm 0.7$	$22.5\pm 3.4$	$21.8\pm 1.9$
DP-Muon	$27.8\pm 0.2$	$\mathbf{27.5\pm 0.5}$	$\mathbf{28.6\pm 0.8}$	$\mathbf{30.0\pm 2.4}$	$\mathbf{35.2\pm 3.1}$

Appendix

Table 10: Test accuracy (%) on SVHN at

\varepsilon=4

. Mean

\pm

std over three seeds.

Layer	$(m,n)$	$\Delta\sigma$	$B\!=\!256$	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$	$B\!=\!8192$
conv1	(64, 147)	0.418	✓	✓	✓	✓	✓	✓
layer1.0.c1	(64, 576)	0.156	✓	✓	✓	✓	✓	✓
layer1.0.c2	(64, 576)	0.216	✓	✓	✓	✓	✓	✓
layer1.1.c1	(64, 576)	0.236	✓	✓	✓	✓	✓	✓
layer1.1.c2	(64, 576)	0.247	✓	✓	✓	✓	✓	✓
layer2.0.c1	(128, 576)	0.318	✓	✓	✓	✓	✓	✓

Appendix

Table 11: Per-layer recovery threshold

B^{*}

for ResNet-18 on CIFAR-10 at

\varepsilon=4

\checkmark

B\geq B^{*}

(leading singular vector recoverable). Spectral gaps measured at epoch 1.

Layer	$(m,n)$	$\Delta\sigma$	$B\!=\!256$	$B\!=\!512$	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$	$B\!=\!8192$
conv1	(64, 147)	0.592	✓	✓	✓	✓	✓	✓
layer1.0.c1	(64, 576)	0.094	–	–	–	✓	✓	✓
layer1.0.c2	(64, 576)	0.056	–	–	–	–	–	✓
layer2.0.c1	(128, 576)	0.318	✓	✓	✓	✓	✓	✓
layer2.0.c2	(128, 1152)	0.204	✓	✓	✓	✓	✓	✓
layer3.0.c2	(256, 2304)	0.474	✓	✓	✓	✓	✓	✓

Appendix

Table 12: Per-layer recovery threshold

B^{*}

for ResNet-18 on ImageNet-1k at

\varepsilon=8

. Representative layers shown.

\checkmark

B\geq B^{*}

$B$	Steps	SNR	DP-SGD	DP-Muon
256	9750	0.06	39.4	39.0
512	4850	0.10	39.4	40.5
1024	2400	0.25	39.7	43.3
2048	1200	0.68	37.0	47.2
4096	600	1.75	39.3	48.7
Pearson $r$			$r_{\text{steps}}\!=\!+0.35$	$r_{\text{SNR}}\!=\!+0.87$

Appendix

Table 13: DP-SGD accuracy tracks step count while DP-Muon tracks gradient SNR. CIFAR-10, WRN-16-4,

\varepsilon=4

. Pearson

r

in last row.

Figure 7: Test accuracy vs. epoch for DP-SGD and DP-Muon on WRN-28-10 (CIFAR-10,

B=4096

\varepsilon=4

). Mean

\pm

std over three seeds. The gap emerges by epoch 10 and widens steadily.

Figure 8: Standard deviation of test accuracy (over three seeds) vs. batch size across three datasets (

\varepsilon=4

). Spectral methods (DP-Muon, DP-Muon-S) maintain consistently lower variance, especially at extreme batch sizes where DP-SGD variance is highest.

$k$	conv1	layer2.0.c1	layer4.0.c2	fc
	$(64\times 147)$	$(128\times 576)$	$(512\times 4608)$	$(10\times 512)$
0	0.550	0.631	0.123	0.460
1	0.653	0.731	0.295	0.791
2	0.806	0.866	0.367	0.910
3	0.939	0.964	0.484	0.933
4	0.983	0.982	0.585	0.938

Appendix

Table 14: Newton-Schulz convergence on ResNet-18 gradient matrices (CIFAR-10, epoch 1).

\cos(X_{k},UV^{\top})

: cosine similarity between the

k

-th iterate and the exact polar factor.

Method	Optimizer	LR	$\beta$	Nesterov	Post-processing	NS
DP-SGD	SGD	0.3	0.9	–	None	–
DP-Adam	Adam	0.001	–	–	None	–
DP-Muon	SGD	0.02	0.95	✓	$UV^{\top}$	5
DP-Muon-S	SGD	0.3	0.9	–	$\sigma_{1}UV^{\top}$	5
DiSK-SGD	SGD	0.01	–	–	Kalman ( $\kappa\!=\!1.5$ )	–
DOPPLER-SGD	SGD	3.0	–	–	EMA ( $\beta\!=\!0.9$ )	–

Appendix

Table 15: Method hyperparameters. NS = Newton-Schulz iterations.

$B$	$\sigma$	Steps	$\sigma$	Steps	$\sigma$	Steps	$\sigma$	Steps
	CIFAR-10		CIFAR-100		SVHN		Tiny-ImageNet
256	0.85	9750	0.85	9750	0.77	14300	0.72	19500
512	1.04	4850	1.04	4850	0.92	7150	0.85	9750
1024	1.32	2400	1.32	2400	1.15	3550	1.04	4850
2048	1.74	1200	1.74	1200	1.48	1750	1.32	2400
4096	2.35	600	2.35	600	1.99	850	1.74	1200

Appendix

Table 16: Noise multiplier

\sigma

and total training steps per batch size at

\varepsilon=4

\delta=10^{-5}

, 50 epochs. Computed via PRV accountant.

Figure 9: Batch size sweep on Tiny-ImageNet (

\varepsilon=4

, WRN-16-4). Mean

\pm

std over three seeds. DP-SGD and DP-Adam collapse at large

B

while spectral methods remain stable.

Method	$B\!=\!1024$	$B\!=\!2048$	$B\!=\!4096$	$B\!=\!8192$
DP-SGD	$\mathbf{21.1\pm 0.1}$	$20.7\pm 0.3$	$17.6\pm 0.4$	$12.0\pm 0.7$
DP-Adam	$16.1\pm 0.5$	$15.4\pm 0.4$	$13.3\pm 0.3$	$10.6\pm 0.4$
DP-Muon	$19.2\pm 0.2$	$20.0\pm 0.2$	$20.4\pm 0.2$	$18.9\pm 0.1$
DP-Muon-S	$20.0\pm 0.1$	$\mathbf{20.9\pm 0.3}$	$\mathbf{20.9\pm 0.3}$	$\mathbf{18.9\pm 0.2}$

Appendix

Table 17: Test accuracy (%) on Tiny-ImageNet at

\varepsilon=4

(WRN-16-4). Mean

\pm

std over three seeds. Best per column in bold .

Method	$\varepsilon=4$	$\varepsilon=8$
DP-SGD	$18.47\pm 0.41$	$24.60\pm 1.18$
DP-AdamW	$10.37\pm 0.21$	$18.64\pm 0.10$
DP-Muon	$\mathbf{20.97\pm 0.13}$	$\mathbf{27.32\pm 0.16}$

Appendix

Table 18: Top-1 accuracy (%) for from-scratch NF-ResNet-50 on ImageNet-1k.

B=32{,}768

, 120 epochs,

K=4

, three seeds. Best per column in bold .

Configuration	DP-SGD	DP-Muon	$\Delta$
$B=2048,\ K=1$	$488.1\pm 1.1$	$502.4\pm 0.8$	$+2.92\%$
$B=4096,\ K=4$	$3894.6\pm 1.0$	$3924.4\pm 11.1$	$+0.77\%$

Appendix

Table 19: Wall-clock overhead of Newton-Schulz orthogonalization. WRN-16-4, fp32, single A100. Milliseconds per logical step, mean

\pm

std over three runs.

\Delta

is the relative overhead of DP-Muon over DP-SGD.

Explore similar work

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning
May 19, 2026Mohammad Partohaghighi, Roummel MarciaDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Department of Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA · Department of Applied Mathematics University of California, Merced Merced, CA, USA
Differentially Private Natural Gradient Descent
Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Gradient Descent Natural Gradient Methods
Institute of Information Engineering, Chinese Academy of Sciences Beijing, China · Department of Computer Science, Metropolitan College, Boston University Boston, United States · Nanyang Technological University Singapore, Singapore
Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective
Mar 3, 2026Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov +2Differentially Private Stochastic Gradient Descent Differential Privacy
University of Basel, Switzerland · University of Zürich, Switzerland

May 19, 2026J/K move · Enter open · S save

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

Mohammad Partohaghighi, Roummel Marcia

Department of Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA · Department of Applied Mathematics University of California, Merced Merced, CA, USA

Deep Learning Optimization Differentially Private Stochastic Gradient Descent Differential Privacy

Open paper AskPDF arXiv

Differentially private stochastic gradient descent (DP-SGD) enables private deep learning through per-example clipping and calibrated Gaussian noise, but its high-variance updates can reduce utility on challenging datasets. We propose SMA-DP-SGD, a Spectral Memory-Aware Differentially Private Stochastic Gradient Descent method that augments DP-SGD with a fractional memory branch built only from previously privatized noisy releases. WeightWatcher-inspired power-law spectral exponents provide group-wise reliability signals, instantiated layer-wise in our experiments, to adapt the decay and effective memory depth. Private-history alignment, norm matching, and warm-up activation stabilize the memory contribution. Privacy remains transparent: conditioned on the private release history, the memory branch is fixed, and the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient

β

. Hence, SMA-DP-SGD preserves a clean conditional sensitivity structure and exactly recovers group-wise DP-SGD when

β=1

. Experiments on CIFAR-100, CIFAR-10, and MNIST show competitive or superior accuracy over several DP optimization baselines, with the largest gains on CIFAR-100 and CIFAR-10. CIFAR-10 ablations show that

β

controls the privacy--utility trajectory, while spectral and memory diagnostics confirm a controlled short-to-moderate effective memory depth and a small memory-branch ratio. Runtime analysis shows that the mechanism incurs additional overhead, about

2.94\times

DP-SGD in our CIFAR-10 implementation, revealing a practical trade-off between adaptive private memory and computational cost.

Abstract

Explore similar work

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

Differentially Private Natural Gradient Descent

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective