MOMENT-DP: Momentum Aware Evaluation of Correlated Noise for Differentially Private Training
Organizations: Google Cloud
Abstract
Temporally correlated noise can substantially improve the utility of differentially private optimization, yet candidate mechanisms are commonly designed or evaluated using only partial information about the optimization process, such as workload structure, learning-rate schedules, or local curvature. We study whether persistent optimizer state provides additional information that can change the relative utility of privacy-valid correlated-noise mechanisms. We introduce MOMENT-DP, an optimizer-state-aware framework that models how temporally correlated privacy noise propagates through momentum-based optimization. For locally quadratic objectives, MOMENT-DP expresses privacy-induced final loss through an optimizer transfer operator that jointly captures curvature, learning-rate scheduling, momentum, and temporal noise covariance. NoiseCurve's curvature-aware quadratic criterion is recovered exactly as the zero-momentum, constant-learning-rate special case. Changing only momentum while holding curvature, schedule, privacy calibration, and candidate mechanisms fixed induces genuine mechanism-ranking reversals. The effect persists outside exact quadratic dynamics in per-example-clipped private logistic regression, while remaining unresolved in the neural regime we study. These results establish optimizer state as an under-modeled, decision-relevant axis in evaluating temporally correlated privacy noise.
Figures & tables
| Oracle best | NC score | Moment-DP | |
|---|---|---|---|
| 0.00 | NoiseCurve | correct | correct |
| 0.50 | Dense MF | wrong | correct |
| 0.80 | Dense MF | wrong | correct |
| 0.90 | Dense MF | wrong | correct |
| 0.95 | Dense MF | wrong | correct |
| Condition | NC rank | NC excess loss | Best excess loss |
|---|---|---|---|
| , constant | 2/9 | ||
| , constant | 9/9 | ||
| , cosine | 9/9 | ||
| , warmup-cos. | 9/9 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Score | Pairwise | Kendall | Rel. regret | Top-1 |
|---|---|---|---|---|
| Static MeanSE | 58.0% | 0.143 | 12.7% | 33.3% |
| LR-aware MF | 60.8% | 0.200 | 11.7% | 33.3% |
| Curv.+sched., | 75.8% | 0.501 | 20.1% | 36.7% |
| Moment-DP |
| Condition | Best | Worst | Spread | Combined SE |
|---|---|---|---|---|
| SGD + constant | BandMF 12.3% | LR-aware 12.0% | 0.34 pp | 0.32 pp |
| Momentum + constant | IID/Toeplitz 22.7% | LR-aware 22.2% | 0.48 pp | 0.34 pp |
| Momentum + cosine | IID 17.6% | Dense 17.3% | 0.36 pp | 0.27 pp |
| Momentum + warmup-cosine | BandMF 17.1% | Toeplitz 16.8% | 0.29 pp | 0.15 pp |
Differential Privacy
Differential privacy limits how much an algorithm’s outputs can reveal about any one person’s data, with formal guarantees.
See all 309 papers