Deep Learning under Fractional-Order Differential Privacy

Mohammad Partohaghighi; Roummel Marcia

cs.CRMay 11, 2026

Deep Learning under Fractional-Order Differential Privacy

Authors: Mohammad Partohaghighi, Roummel Marcia

Abstract

Differentially private stochastic gradient descent (DP-SGD) is a standard approach to privacy-preserving learning based on per-example clipping, subsampling, Gaussian perturbation, and privacy accounting. Classical DP-SGD releases a noisy version of the current clipped subsampled gradient sum. We propose Fractional-Order Differentially Private Stochastic Gradient Descent (\textbf{FO-DP-SGD}), a mechanism-level extension that replaces this current-only query, before Gaussian noise is added, with a fractional recursive query combining the current clipped sum with a finite-window, power-law-weighted aggregation of previously released private sum-level outputs. This injects fractional memory into the release mechanism while preserving the standard \emph{sum-then-noise-then-divide} structure. Under add/remove adjacency with Poisson subsampling, the current-step sensitivity analysis shows that the only newly data-dependent term is the scaled current clipped sum. Hence, conditioned on the private history, the effective $\ell_2$ -sensitivity is at most $βC$ , where $C$ is the clipping threshold and $β\in(0,1]$ controls the current-step contribution. Thus, FO-DP-SGD admits standard per-step Rényi differential privacy accounting via a Poisson-subsampled Gaussian mechanism with effective noise-to-sensitivity ratio $σ/β$ , and composes to yield overall $(\varepsilon,δ)$ -differential privacy guarantees. FO-DP-SGD provides a framework for studying long-memory effects in private optimization. The fractional order, memory window, and mixing coefficient govern the trade-off among current-step sensitivity, signal retention, and private-history influence. Experiments on SVHN, CIFAR-10, and CIFAR-100 show improved test accuracy and privacy--utility performance over DP-SGD and private baselines including DP-Adam, DP-IS, SA-DP-SGD, ADP-AdamW, DP-SAT, and DP-Adam-AC.

Explore similar work

May 19, 2026cs.LG

SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

Differentially private stochastic gradient descent (DP-SGD) enables private deep learning through per-example clipping and calibrated Gaussian noise, but its high-variance updates can reduce utility on challenging datasets. We propose \textbf{SMA-DP-SGD}, a \textbf{Spectral Memory-Aware Differentially Private Stochastic Gradient Descent} method that augments DP-SGD with a fractional memory branch built only from previously privatized noisy releases. WeightWatcher-inspired power-law spectral exponents provide group-wise reliability signals, instantiated layer-wise in our experiments, to adapt the decay and effective memory depth. Private-history alignment, norm matching, and warm-up activation stabilize the memory contribution. Privacy remains transparent: conditioned on the private release history, the memory branch is fixed, and the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient

β

. Hence, SMA-DP-SGD preserves a clean conditional sensitivity structure and exactly recovers group-wise DP-SGD when

β=1

. Experiments on CIFAR-100, CIFAR-10, and MNIST show competitive or superior accuracy over several DP optimization baselines, with the largest gains on CIFAR-100 and CIFAR-10. CIFAR-10 ablations show that

β

controls the privacy--utility trajectory, while spectral and memory diagnostics confirm a controlled short-to-moderate effective memory depth and a small memory-branch ratio. Runtime analysis shows that the mechanism incurs additional overhead, about

2.94\times

DP-SGD in our CIFAR-10 implementation, revealing a practical trade-off between adaptive private memory and computational cost.

Mohammad Partohaghighi, Roummel Marcia

Differential Privacy Stochastic Gradient Descent AI Privacy Risks and Protection Deep Learning Long-Term Memory

Ask PDF

May 7, 2026cs.LG

Trade-off Functions for DP-SGD with Subsampling based on Random Shuffling: Tight Upper and Lower Bounds

We derive a tight analysis of the trade-off function for Differentially Private Stochastic Gradient Descent (DP-SGD) with subsampling based on random shuffling within the

f

-DP framework. Our analysis covers the regime

σ\geq \sqrt{3/\ln M}

, where

σ

is the noise multiplier and

M

is the number of rounds within a single epoch. Unlike

f

-DP analyses for Poisson subsampling, which yield non-closed implicit formulas that can be machine computed but are non-transparent, random shuffling admits a tight analysis yielding transparent and interpretable closed-form bounds. Our concrete bounds, derived via the Berry-Esseen theorem, are tight up to constant factors within the proof framework. We demonstrate worked parameter settings for a single epoch (

E=1

) with a corresponding trade-off function

\geq 1-a-δ

, that is, only

δ

below the ideal random guessing diagonal

1-a

: For

δ= 1/100

and

σ= 1

, roughly

M \approx 1.14\times 10^6

rounds and

N \approx 1.14\times 10^7

training samples suffice to achieve meaningful differential privacy. This is in contrast to recent negative results for the regime

σ\leq 1/\sqrt{2 \ln M}

. Our concrete bounds can be composed over multiple epochs leading to

δ

having a linear in

E

dependency, which restricts

E=O(\sqrt{M})

. To go beyond Berry--Esseen, we introduce a new proof technique based on a generalization of the law of large numbers that yields an asymptotic random guessing diagonal-limit result: if

E=c_M^2M

with

c_M\to 0

, then the

E

-fold composed trade-off function satisfies

f^{\otimes E}(a)\to 1-a

uniformly in

a\in[0,1]

with

δ

having only an

O(\sqrt{E})

dependency. We compare this asymptotic regime with the corresponding Poisson subsampling asymptotic, and highlight the characterization of explicit convergence rates as an open question.

Marten van Dijk, Murat Bilgehan Ertan

Differential Privacy Stochastic Gradient Descent Optimal Sample Complexity Subsampling Lower Bounds Single-Message Shuffle Model+1

Ask PDF

Jun 3, 2026cs.LG

When Do Fewer Coordinates Suffice in DP-SGD?

Differentially private stochastic gradient descent (DP-SGD) injects noise into every updated coordinate, making the injected noise energy scale with the ambient parameter dimension

d

. We ask when private training can update fewer coordinates without losing the signal needed for optimization. We propose \textsc{TP-TopK} (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training without public data, in which a private warm-up phase identifies a coordinate support used to guide the main training phase. We give a criterion characterizing when coordinate restriction can be beneficial, show via a nonconvex stationarity bound that under this condition the relevant noise term scales with the active dimension

k

rather than the full parameter dimension

d

, and provide a lower bound on the reliability of warm-up-based coordinate ranking. Experiments on MNIST, FMNIST, and CIFAR-10 show that learned coordinate supports can retain more gradient energy than size-matched random supports, with the largest gains when the active dimension is small and warm-up scores are informative.

Huiqi Zhang, Fang Xie

Differential Privacy Stochastic Gradient Descent Stationarity

Ask PDF