DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion

Tao Huang; Jiayang Meng; Xu Yang; Chen Hou; Hong Chen

cs.LGFeb 26, 2026

DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion

Authors: Tao Huang, Jiayang Meng, Xu Yang, Chen Hou, Hong Chen

Organizations: School of Computer Science and Big Data, Minjiang University, Fuzhou, Fujian, China. · School of Information, Renmin University of China, Beijing, China.

Abstract

Condition injection enables diffusion models to generate context-aware outputs, which is essential for many time-series tasks. However, heterogeneous conditional contexts (e.g., observed history, missingness patterns or outlier covariates) can induce heavy-tailed per-example gradients. Under Differentially Private Stochastic Gradient Descent (DP-SGD), these rare conditioning-driven heavy-tailed gradients disproportionately trigger global clipping, resulting in outlier-dominated updates, larger clipping bias, and degraded utility under a fixed privacy budget. In this paper, we propose DP-aware AdaLN-Zero, a drop-in sensitivity-aware conditioning mechanism for conditional diffusion transformers that limits conditioning-induced gain without modifying the DP-SGD mechanism. DP-aware AdaLN-Zero jointly constrains conditioning representation magnitude and AdaLN modulation parameters via bounded re-parameterization, suppressing extreme gradient tail events before gradient clipping and noise injection. Empirically, DP-SGD equipped with DP-aware AdaLN-Zero improves interpolation/imputation and forecasting under matched privacy settings. We observe consistent gains on a real-world power dataset and two public ETT benchmarks over vanilla DP-SGD. Moreover, gradient diagnostics attribute these improvements to conditioning-specific tail reshaping and reduced clipping distortion, while preserving expressiveness in non-private training. Overall, these results show that sensitivity-aware conditioning can substantially improve private conditional diffusion training without sacrificing standard performance.

Figures & tables

Panel of Figure 1 — Figure 1: Condition-amplified extremes exist even without DP. We compare normal training with training under DP-aware constraints without DP. Under normal training, $\|g_{\mathrm{cond}}\|_{2}$ is comparable to $\|g_{\mathrm{other}}\|_{2}$ at typical quantiles (see the blue curve in Figure 1(a) ), but $\|g_{\mathrm{cond}}\|_{2}$ exhibits rarer and heavier high-end tail events (see the blue curve in Figure 1(b) ). In contrast, DP-aware constraints selectively suppress the high-end tail of $\|g_{\mathrm{cond}}\|_{2}$ (and consequently that of $\|g\|_{2}$ ) far more than $\|g_{\mathrm{other}}\|_{2}$ : $p99$ drops by $\sim 3.5\times$ for $\|g_{\mathrm{cond}}\|_{2}$ vs. $\sim 1.2\times$ for $\|g_{\mathrm{other}}\|_{2}$ . This indicates targeted suppression of conditioning-induced amplification rather than uniform shrinkage, and suggests that (in DP-SGD) clipping events would be disproportionately governed by rare conditioning-path extremes.

		Interpolation/Imputation			Forecasting
Model	$\sigma$	point_RMSE $\downarrow$	point_MAPE $\downarrow$	point_MAE $\downarrow$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spec_dist $\downarrow$
Non-DP	-	0.584	1.174%	0.476	0.208	0.767	1.484e-4
DP-vanilla	0.03	3.034	5.673%	2.391	0.556	0.736	7.848e-4
DP-aware	0.03	1.804	3.541%	1.411	0.296	0.684	2.900e-4
DP-vanilla	0.05	3.498	9.339%	3.110	0.567	0.643	1.650e-3
DP-aware	0.05	2.019	4.970%	1.987	0.423	0.636	8.558e-4

Table 1: Results on PrivatePower under matched DP training. We evaluate mask-conditioned interpolation/imputation and forecasting. All metrics are lower-is-better. Bold marks the better of DP-vanilla and DP-aware at each

\sigma

Panel of Figure 2 — Figure 2: Gradient-norm distributions under DP training . Compared to DP-vanilla, DP-aware primarily suppresses the extreme tail of the conditioning pathway with minimal impact on the bulk of the distribution, indicating fewer condition-amplified outliers rather than uniform gradient shrinkage.

$\sigma$	Model	p95 Statistics					p99 Statistics
$\sigma$	Model	$S_{\text{other}}$	$S_{\text{cond}}$	$S_{\text{total}}$	$\rho_{\text{emp}}$	$\rho_{\text{cond}}$	$S_{\text{other}}$	$S_{\text{cond}}$	$S_{\text{total}}$	$\rho_{\text{emp}}$	$\rho_{\text{cond}}$
0.03	DP-vanilla	25.0	17.8	31.1	–	–	58.4	49.5	76.5	–	–
0.03	DP-aware	24.6	16.4	29.7	0.955	0.921	57.5	43.7	72.0	0.949	0.883
0.05	DP-vanilla	24.1	19.6	31.4	–	–	62.4	62.4	89.2	–	–
0.05	DP-aware	24.2	18.9	30.9	0.984	0.964	55.6	55.5	77.3	0.867	0.889
0.10	DP-vanilla	23.8	21.4	32.1	–	–	62.8	68.3	92.7	–	–

Table 2: Gradient-norm diagnostics on PrivatePower across noise multipliers

\sigma

. We report robust high-percentile summaries (

p95

p99

) of per-example gradient norms for conditioning-path parameters

\|g_{\mathrm{cond}}\|_{2}

and remaining parameters

\|g_{\mathrm{other}}\|_{2}

. We also report a compact proxy

S_{\text{total}}

, empirical diagnostics

\rho_{\mathrm{emp}}

and

\rho_{\mathrm{cond}}

; values below

1

indicate reductions relative to DP-vanilla under matched

(C,\sigma)

$\sigma$	Model	$p_{\mathrm{clip}}$	$\mathbb{E}[\eta]$	$\eta_{10}$	$\eta_{50}$	$\eta_{90}$
0.03	DP-vanilla	0.561	0.73	0.18	0.86	1.00
0.03	DP-aware	0.556	0.75	0.22	0.88	1.00
0.20	DP-vanilla	0.589	0.70	0.12	0.82	1.00
0.20	DP-aware	0.585	0.72	0.15	0.84	1.00

Table 3: Clipping statistics under matched

C

. We report the clipping rate

p_{\mathrm{clip}}

and quantiles of the clipping factor

\eta

. Across noise multipliers,

p_{\mathrm{clip}}

remains comparable, while DP-aware slightly increases low-quantile and median values of

\eta

, suggesting reduced clipping severity.

Model Variant	$\mathbf{c}$ bounded	$(\gamma,\beta,\alpha)$ bounded	$\rho_{\text{emp}}$	Forecast point_RMSE $\downarrow$	Forecast dist_JS $\downarrow$
DP-vanilla	$\times$	$\times$	1.00	0.567	0.723
DP-aware (only $\mathbf{c}$ -bounding)	$\checkmark$	$\times$	0.95	0.510	0.691
DP-aware (only AdaLN bounding)	$\times$	$\checkmark$	0.93	0.495	0.677
DP-aware (full, ours)	$\checkmark$	$\checkmark$	0.87	0.423	0.636

Table 4: Effects of forward-pass control components. We report

\rho_{\text{emp}}(p99)

and forecasting metrics (lower is better).

$\mathcal{B}_{M}(\cdot)$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spec_dist $\downarrow$
$\tanh$ (default)	0.423	0.636	8.558e-4
soft_clamp_band	0.425	0.639	8.70e-4
hard_clamp	0.452	0.662	9.45e-4
clamp_ste	0.467	0.674	9.90e-4

Table 5: Effect of the bounding operator

\mathcal{B}_{M}(\cdot)

\sigma{=}0.05

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\sigma$	MAE $\downarrow$	RMSE $\downarrow$	MAPE $\downarrow$	R2 $\uparrow$	dist_KL $\downarrow$	dist_JS $\downarrow$	dist_WS $\downarrow$	dist_KS $\downarrow$	MMD $\downarrow$
Non-DP (no DP)	–	0.155	0.208	1.90%	0.861	0.980	0.767	0.605	0.215	1.48e-2
DP-vanilla	0.03	0.430	0.556	4.85%	0.422	1.122	0.736	0.820	0.260	3.90e-2
DP-aware	0.03	0.235	0.296	3.10%	0.741	1.025	0.684	0.700	0.230	2.70e-2
DP-vanilla	0.05	0.448	0.567	5.10%	0.383	1.081	0.643	0.840	0.270	4.80e-2
DP-aware	0.05	0.325	0.423	4.00%	0.603	1.052	0.636	0.810	0.258	4.10e-2
DP-vanilla	0.1	1.280	1.637	12.9%	-0.105	1.555	0.833	1.458	0.390	1.90e-1

Appendix

Table 6: Forecasting metrics on PrivatePower across noise multipliers

\sigma

. Lower is better for all metrics except R2.

Model	$\sigma$	MAE $\downarrow$	RMSE $\downarrow$	MAPE $\downarrow$	R2 $\uparrow$	dist_KL $\downarrow$	dist_JS $\downarrow$	dist_WS $\downarrow$	dist_KS $\downarrow$	MMD $\downarrow$
Non-DP (no DP)	–	0.476	0.584	1.17%	0.775	0.900	0.660	0.560	0.190	1.20e-2
DP-vanilla	0.03	2.391	3.034	5.67%	-0.950	1.210	0.820	0.760	0.240	3.20e-2
DP-aware	0.03	1.411	1.804	3.54%	-0.250	1.030	0.740	0.670	0.220	2.40e-2
DP-vanilla	0.05	3.110	3.498	9.34%	-1.400	1.260	0.835	0.790	0.250	3.80e-2
DP-aware	0.05	1.987	2.019	4.97%	-0.600	1.090	0.760	0.700	0.230	2.90e-2
DP-vanilla	0.1	4.702	5.787	11.67%	-4.100	1.520	0.910	1.080	0.310	9.50e-2

Appendix

Table 7: Interpolation/imputation metrics on PrivatePower across noise multipliers

\sigma

. Lower is better for all metrics except R2.

Panel of Figure 3 — Figure 3: Gradient distributions under DP training (all $\sigma$ settings).

Model	Interpolation/Imputation			Forecasting
	point_RMSE $\downarrow$	point_MAPE $\downarrow$	point_MAE $\downarrow$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spectral_dist $\downarrow$
Non-DP (no DP)	0.310	2.35%	0.210	0.245	0.610	2.10e-4
DP-vanilla	0.780	5.90%	0.520	0.655	0.880	1.85e-3
DP-aware	0.560	4.10%	0.380	0.465	0.780	9.60e-4

Appendix

Table 8: Representative metrics on ETTh1 (matched DP mechanism). DP-vanilla and DP-aware are trained under matched DP-SGD settings (

C=1.0

, noise multiplier

\sigma=0.1

, batch size

B=96

, and steps

T=20,000

Model	Interpolation/Imputation			Forecasting
	point_RMSE $\downarrow$	point_MAPE $\downarrow$	point_MAE $\downarrow$	point_RMSE $\downarrow$	dist_WS $\downarrow$	temp_spectral_dist $\downarrow$
Non-DP (no DP)	0.355	2.80%	0.240	0.295	0.640	2.60e-4
DP-vanilla	0.945	6.80%	0.625	0.805	0.920	2.15e-3
DP-aware	0.685	4.90%	0.455	0.565	0.820	1.20e-3

Appendix

Table 9: Representative metrics on ETTm1 (matched DP mechanism). DP-vanilla and DP-aware are trained under matched DP-SGD settings (

C=1.0

, noise multiplier

\sigma=0.1

, batch size

B=96

, and steps

T=20,000

	$\sigma=0.05$			$\sigma=0.20$
$\mathcal{B}_{M}(\cdot)$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spec_dist $\downarrow$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spec_dist $\downarrow$
$\tanh$ (default)	0.423	0.636	8.558e-4	1.262	0.732	1.052e-2
soft_clamp_band	0.425	0.639	8.70e-4	1.270	0.737	1.070e-2
hard_clamp	0.452	0.662	9.45e-4	1.310	0.752	1.120e-2
clamp_ste	0.467	0.674	9.90e-4	1.330	0.760	1.150e-2

Appendix

Table 10: Bounding-operator comparison on PrivatePower for forecasting under matched DP-SGD. Smooth operators (

\tanh

and soft_clamp_band ) perform similarly and outperform hard truncation variants ( hard_clamp and clamp_ste ).

	$\sigma=0.05$			$\sigma=0.20$
$\mathcal{B}_{M}(\cdot)$	point_RMSE $\downarrow$	point_MAPE $\downarrow$	point_MAE $\downarrow$	point_RMSE $\downarrow$	point_MAPE $\downarrow$	point_MAE $\downarrow$
$\tanh$ (default)	2.019	4.97%	1.987	4.689	8.41%	3.442
soft_clamp_band	2.025	4.99%	1.995	4.710	8.44%	3.460
hard_clamp	2.120	5.22%	2.070	4.860	8.70%	3.560
clamp_ste	2.180	5.35%	2.120	4.950	8.85%	3.620

Appendix

Table 11: Bounding-operator comparison on PrivatePower for interpolation/imputation under matched DP-SGD. Trends match forecasting:

\tanh

and soft_clamp_band are consistently better than hard truncation variants.

Model	$\sigma$	$p_{\mathrm{clip}}\downarrow$	mean( $\eta$ ) $\uparrow$	p10( $\eta$ ) $\uparrow$	p50( $\eta$ ) $\uparrow$	p90( $\eta$ ) $\uparrow$	p99( $\eta$ ) $\uparrow$
DP-vanilla	0.03	0.561	0.73	0.18	0.86	1.00	1.00
DP-aware	0.03	0.554	0.75	0.22	0.88	1.00	1.00
DP-vanilla	0.05	0.570	0.72	0.16	0.84	1.00	1.00
DP-aware	0.05	0.568	0.73	0.18	0.85	1.00	1.00
DP-vanilla	0.10	0.582	0.71	0.14	0.83	1.00	1.00
DP-aware	0.10	0.578	0.72	0.16	0.84	1.00	1.00

Appendix

Table 12: Clipping behaviour under DP-SGD on PrivatePower (matched DP mechanism). We report the clipping activation rate

p_{\mathrm{clip}}

and summary statistics of the clipping factor

\eta=\min(1,\frac{C}{\|g\|_{2}})

. Lower

p_{\mathrm{clip}}

and larger typical

\eta

indicate less clipping-induced attenuation.

Panel of Figure 4 — Figure 4: Clipping behavior under threshold $C$ (all $\sigma$ settings). We compare DP-vanilla and DP-aware in terms of clipping factor $\eta=\min(1,\frac{C}{\|g_{\mathrm{total}}\|})$ and clipping activation rate $p_{\mathrm{clip}}=\mathbb{P}(\|g_{\mathrm{total}}\|>C)$ .

Setting	$c_{\max}$	$\gamma_{\max}$	$\beta_{\max}$	$\alpha_{\max}$	$\rho_{\text{emp}}$	point_RMSE $\downarrow$	dist_JS $\downarrow$
Loose	$1.00\,C_{0}$	$1.00\,\Gamma_{0}$	$1.00\,B_{0}$	$1.00\,A_{0}$	0.92	0.452	0.661
Medium	$0.90\,C_{0}$	$0.90\,\Gamma_{0}$	$0.90\,B_{0}$	$0.90\,A_{0}$	0.87	0.423	0.636
Tight	$0.75\,C_{0}$	$0.75\,\Gamma_{0}$	$0.75\,B_{0}$	$0.75\,A_{0}$	0.80	0.441	0.651
Too tight	$0.50\,C_{0}$	$0.50\,\Gamma_{0}$	$0.50\,B_{0}$	$0.50\,A_{0}$	0.73	0.562	0.750

Appendix

Table 13: Effect of DP-aware bound tightness on PrivatePower for forecasting. DP-vanilla and all DP-aware variants use a matched DP-SGD mechanism (global clipping norm

C=1.0

, noise multiplier

\sigma=0.05

, batch size

B=96

, and training steps

T=20{,}000

). We scale a base reference setting

(C_{0},\Gamma_{0},B_{0},A_{0})

for DP-aware bounds. We report

\rho_{\text{emp}}(p99)

(average-case diagnostic; smaller indicates lighter tails/smaller typical gradients) and forecasting metrics (lower is better). The best performance is highlighted in bold.

	Interpolation/Imputation			Forecasting
Model	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spectral_dist $\downarrow$	point_RMSE $\downarrow$	dist_JS $\downarrow$	temp_spectral_dist $\downarrow$
Non-DP	0.584	0.66	1.48e-4	0.208	0.77	1.48e-4
+ DP-aware ( Loose )	0.590	0.67	1.52e-4	0.212	0.62	1.55e-4
+ DP-aware ( Medium )	0.583	0.66	1.48e-4	0.207	0.77	1.48e-4
+ DP-aware ( Tight )	0.600	0.68	1.60e-4	0.218	0.63	1.62e-4
+ DP-aware ( Too tight )	0.720	0.78	2.30e-4	0.290	0.75	2.40e-4

Appendix

Table 14: Expressiveness check on PrivatePower without DP noise. We compare a non-private baseline to non-private DP-aware variants under different tightness settings. Lower is better for all metrics shown. The best performance is highlighted in bold.

$C$	Model	$\rho_{\text{emp}}$	RMSE $\downarrow$	dist_JS $\downarrow$
0.5	DP-vanilla	1.00	2.726	0.978
0.5	DP-aware	0.87	1.145	0.826
1.0	DP-vanilla	1.00	1.673	0.833
1.0	DP-aware	0.89	0.671	0.757
2.0	DP-vanilla	1.00	1.344	0.787
2.0	DP-aware	0.89	0.446	0.664

Appendix

Table 15: Effect of clipping threshold

C

on PrivatePower for forecasting at

\sigma=0.1

. All rows use a matched DP mechanism for each

C

(same

\sigma

B

, and

T

); only

C

changes. DP-aware consistently outperforms DP-vanilla across a range of

C

values.

Setting	$\rho_{\text{emp}}$	RMSE $\downarrow$	MAE $\downarrow$
Loose	0.92	2.231	2.170
Medium	0.87	2.019	1.987
Tight	0.80	2.672	2.246
Too tight	0.73	3.626	3.179

Appendix

Table 16: Effect of DP-aware bound tightness on PrivatePower for interpolation/imputation at

\sigma=0.05

under a matched DP mechanism. We report

\rho_{\text{emp}}

and representative interpolation metrics (lower is better); boldface denotes the best performance.

Model variant	$\rho_{\text{emp}}$	RMSE $\downarrow$	MAE $\downarrow$
DP-vanilla	1.00	3.498	3.110
DP-aware (only $\mathbf{c}$ bounded)	0.95	3.998	3.070
DP-aware (only AdaLN bounded)	0.93	2.337	2.112
DP-aware (full, ours)	0.89	2.019	1.987

Appendix

Table 17: Effects of forward-pass control components on PrivatePower for interpolation/imputation at

\sigma=0.05

. All rows use a matched DP mechanism. We report the empirical diagnostic

\rho_{\text{emp}}

and representative interpolation metrics (lower is better). Boldface indicates the best performance.

Panel of Figure 5 — Figure 5: Training loss dynamics on PrivatePower. Training MSE loss curves for Non-DP, DP-vanilla, and DP-aware under several DP noise multipliers. Larger $\sigma$ yields a higher loss floor, while DP-aware closely follows DP-vanilla across noise levels.

Method	mean $\downarrow$	p50 $\downarrow$	p90 $\downarrow$	rel. to Non-DP
Non-DP	0.100	0.047	0.201	1.00 $\times$
DP-vanilla	0.168	0.105	0.350	1.68 $\times$
DP-aware	0.169	0.108	0.351	1.69 $\times$

Appendix

Table 18: Runtime summary. Robust statistics of per-step wall-clock time for Non-DP, DP-vanilla, and DP-aware training.

Dataset	Records	Channels	Seq Len	Train/Val/Test	Normalization	ratio_range	pred_len_range	num_blocks_range
PrivatePower	$\approx$ 23.1k	7	168	70%/15%/15%	per-channel standardization	[0.1, 0.5]	[24, 96]	[4, 8]
ETTh1	$\approx$ 17.5k	7	96	12m/4m/4m	per-feature z-score	[0.1, 0.5]	[24, 96]	[4, 8]
ETTm1	$\approx$ 70k	7	96	12m/4m/4m	per-feature z-score	[0.1, 0.5]	[24, 96]	[4, 8]

Appendix

Table 19: Configurations of the datasets and their corresponding mask-based tasks. Each dataset trains a single conditional diffusion model that supports interpolation/imputation (random & stride masks), forecasting (block masks), and full-sequence reconstruction. For PrivatePower, we follow the private-data pipeline in our codebase; for ETTh1 and ETTm1, we follow the common ETT setting.

Conditional Diffusion Models

Conditional diffusion models generate or infer data by steering iterative denoising with observations, labels, text, physical constraints, or other inputs.

See all 276 papers

Privacy Accounting for Empirical Denoisers
Oct 20, 2026William Cheng, Dongjun Kim, Mijung ParkDifferential Privacy Diffusion Sampling
University of British Columbia · Stanford University · Now at Adobe.
StraightDP: Geometry-Aware Differential Privacy for Rectified-Flow Transformers
Jul 31, 2026Xujun Che, Depeng Xu, Xintao WuDiffusion Transformer Differentially Private Stochastic Gradient Descent
Department of Cybersecurity, University of North Carolina at Charlotte, Charlotte, NC, USA · Department of Electrical Engineering and Computer Science, University of Arkansas, Fayetteville, AR, USA
Protecting Sensitive Data in Image Synthesis via PAC-Private Adaptation for Diffusion Models
Oct 2, 2026Boming Miao, Tao Zhang, Netanel Raviv +3Privacy-Preserving ML Diffusion-Based Image Generation
Department of Computer Science and Engineering, Washington University in St. Louis

Oct 20, 2026J/K move · Enter open · S save

Privacy Accounting for Empirical Denoisers

William Cheng, Dongjun Kim, Mijung Park

University of British Columbia · Stanford University · Now at Adobe.

Differential Privacy Diffusion Sampling Diffusion-Based Denoising Gaussian Differential Privacy

Open paper AskPDF arXiv

Diffusion models have emerged as the foundation of modern generative systems, yet their high memorization capacity raises privacy concerns. While differentially private (DP) training provides formal guarantees, it comes at a cost in sample quality. In this work, we study a complementary question: how much privacy loss is incurred during the sampling process itself? Directly analyzing modern neural denoisers is challenging because their sensitivity to a single training example is generally intractable. To overcome this challenge, we construct and analyze a clipped privacy-accountable empirical denoiser, a training-free estimator of the Bayes-optimal denoiser, whose per-step sensitivity can be computed explicitly, allowing each denoising step to be a Gaussian mechanism. This perspective enables privacy accounting under Gaussian DP when each training example contributes to each denoising step, and under

f

-DP trade-off functions when each generated sample uses a single randomly subsampled block of the training data. We report every

(\varepsilon, δ)

guarantee as a certified upper bound from a privacy loss distribution accountant, and use a cumulant-based Edgeworth approximation as a fast diagnostic, characterizing where it is reliable and where it becomes anti-conservative. We also analyze the privacy loss along the denoising trajectory and show that it grows steeply near the critical window in which semantic content is formed. This observation motivates hybrid samplers that use the private empirical denoiser through the formative steps where it remains a faithful proxy for the neural denoiser, then switch to a public neural denoiser for the remaining refinement. Our results on CIFAR-10 and CelebA-64 show that the privacy of diffusion models depends not only on the training procedure, but also on sampling-time design choices such as subsampling, stopping time, and hybrid denoising.

Abstract

Conditional Diffusion Models

Explore similar work

Privacy Accounting for Empirical Denoisers

StraightDP: Geometry-Aware Differential Privacy for Rectified-Flow Transformers

Protecting Sensitive Data in Image Synthesis via PAC-Private Adaptation for Diffusion Models