Towards Hyperparameter Transfer for Differentially Private Optimization

Tianze Wang; Zhiqi Bu; Linjun Zhang

cs.LGOct 16, 2026

Towards Hyperparameter Transfer for Differentially Private Optimization

Authors: Tianze Wang, Zhiqi Bu, Linjun Zhang

Organizations: Department of Statistics, Rutgers University · Independent Researcher

Abstract

Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization ( $μ$ P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP- $μ$ P, a theoretically grounded framework that extends $μ$ P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP- $μ$ P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.

Figures & tables

Figure 1: DP- $\mu$ P Training Workflow. We employ $\mu$ P initialization to stabilize the initial state and align the Signal-to-Noise Ratio (SNR) to derive the appropriate noise parameters for the small proxy model. After identifying the optimal base learning rate on the proxy, we apply layer-wise scaling to the large model to ensure $\mathcal{O}(1)$ loss updates, resulting in invariant training dynamics across model scales. A numerical example is provided in Appendix C.6 .

Optimizer	NonDP lr scaling Yang et al. (2022a)	Clipping-only lr scaling	DP lr scaling
SGD	$\frac{n_{l}}{n_{l-1}}$	$\sqrt{\frac{n_{l}}{n_{l-1}}}$	$\frac{\sqrt{n_{l}/n_{l-1}}}{\frac{\sigma_{\text{DP}}}{B}(\sqrt{n_{l}}+\sqrt{n_{l-1}})}$
Adam	$\frac{1}{n_{l-1}}$	$\frac{1}{n_{l-1}}$	$\frac{\sqrt{n_{l}/n_{l-1}}}{\sqrt{n_{l}}+\sqrt{n_{l-1}}}$
Muon	$\sqrt{\frac{n_{l}}{n_{l-1}}}$	$\sqrt{\frac{n_{l}}{n_{l-1}}}$	$\sqrt{\frac{n_{l}}{n_{l-1}}}$

Table 1: Summary of layer-wise learning rate (lr) scaling rules for

\mu

P under different privacy settings. Note that for DP-SGD, the scaling explicitly accounts for the noise-to-batch ratio.

Panel of Figure 2 — Figure 2: Comparison of hyperparameter transfer scaling rules ( $\mu\text{P}$ ) on a 5-layer MLP (CIFAR-10). (a) Under the clipping-only regime (no noise), loss curves align across widths, validating $\mu\text{P}$ . (b) Scaling by the spectral norm of the noisy update fails to align curves, demonstrating the failure of standard $\mu\text{P}$ under differential privacy.

Figure 3: Comparison between SP and

\mu

P (SGD). We adjust the clipping coefficient and noise to satisfy the SNR condition while keeping the input dimension fixed.

Figure 4: Comparison between Muon and SGD-Muon. We evaluate different update rules (Muon vs. SGD) for the Input and Output layers under the

\mu

P scaling conditions. The left two panels show the non-private setting (test loss), and the right two panels show the DP setting (training loss).

Figure 5: DP-

\mu

P ViT on CIFAR-100. For Muon, the input and output layers are updated using SGD.

Figure 6: Coordinate check for MLP (DP-

\mu

P). The evolution of activation

l_{1}

norms across different widths shows consistent scaling, confirming correct implementation.

Figure 7: Coordinate check for ViT (DP-

\mu

P). Similar to the MLP case, the Vision Transformer maintains stable activation scales across varying widths.

Figure 8: GPT-2 Pretraining with DP-

\mu

P on Shakespeare Character-Level Task. Loss curves across varying widths (controlled by head counts) exhibit consistent ranking and dynamics. This consistency indicates that the optimal learning rate remains invariant across model sizes, demonstrating successful hyperparameter transfer.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	Non-Private	Clipping Only	Differentially Private (DP)
SGD	$\dfrac{n_{l}}{n_{l-1}}$	$\sqrt{\dfrac{n_{l}}{n_{l-1}}}$	$\dfrac{\sqrt{n_{l}/n_{l-1}}}{\frac{\sigma_{\text{DP}}}{B}(\sqrt{n_{l}}+\sqrt{n_{l-1}})}$
Adam	$\dfrac{1}{n_{l-1}}$	$\dfrac{1}{n_{l-1}}$	$\dfrac{\sqrt{n_{l}/n_{l-1}}}{\sqrt{n_{l}}+\sqrt{n_{l-1}}}$
Muon	$\sqrt{\dfrac{n_{l}}{n_{l-1}}}$	$\sqrt{\dfrac{n_{l}}{n_{l-1}}}$	$\sqrt{\dfrac{n_{l}}{n_{l-1}}}$

Appendix

Table 2: Summary of layer-wise learning rate scaling rules for

\mu

P under different privacy settings. This table restates Table 1 so that the derivation in this section can be read on its own.

	width = 256				width = 512				width = 1024
$B=$	125	250	500	1000	125	250	500	1000	125	250	500	1000
Input	1.13	0.56	0.28	0.14	1.25	0.62	0.31	0.16	1.40	0.70	0.35	0.17
First Hidden	0.51	0.25	0.13	0.06	0.72	0.36	0.18	0.09	1.03	0.51	0.26	0.13
Second Hidden	0.50	0.25	0.13	0.06	0.72	0.36	0.18	0.09	1.01	0.51	0.26	0.13
Third Hidden	0.51	0.25	0.13	0.07	0.72	0.36	0.18	0.09	1.02	0.51	0.25	0.13
Output	0.30	0.16	0.08	0.04	0.39	0.20	0.10	0.06	0.56	0.28	0.14	0.07

Appendix

Table 3: Empirical spectral norm of the mini-batch average gradients for DP-SGD across varying batch sizes and network widths.

	width = 256				width = 512				width = 1024
$B=$	125	250	500	1000	125	250	500	1000	125	250	500	1000
Input	71.05	71.37	71.22	70.86	77.51	77.70	77.85	77.50	87.65	87.47	87.14	86.90
First Hidden	31.52	31.46	31.57	31.52	44.86	44.57	45.15	44.94	63.45	63.48	63.44	63.71
Second Hidden	30.77	31.80	32.05	31.43	45.21	44.79	44.64	44.81	64.07	63.81	63.86	63.59
Third Hidden	31.96	31.82	31.41	31.55	44.66	44.72	44.82	44.93	63.52	63.70	63.55	63.93
Output	18.64	18.52	18.09	19.00	24.62	24.60	24.79	25.72	34.20	35.04	34.22	34.20

Appendix

Table 4: Empirical spectral norm of the mini-batch average gradients for DP-Adam across varying batch sizes and network widths.

	width = 320			width = 640			width = 1280
$B=$	64	128	256	64	128	256	64	128	256
Block 5 QKV	48.43	48.38	48.40	68.58	68.70	68.96	97.29	97.69	97.67
Block 5 FC1	53.06	52.87	53.71	75.45	75.86	76.01	107.15	106.98	107.16
Block 5 FC2	53.05	53.73	53.10	75.65	75.36	75.57	107.05	107.52	107.52
Block 10 QKV	48.39	48.36	47.99	69.27	68.85	68.88	97.46	97.68	97.71
Block 10 FC1	53.17	52.91	53.08	75.44	75.97	75.36	107.36	106.92	106.91

Appendix

Table 5: Empirical spectral norm of the mini-batch average gradients for DP-Adam on GPT-2 across varying batch sizes and network widths.

Figure 9: Comparison between SP and

\mu

P (Adam). Similar to SGD, we adjust the clipping coefficient and noise to satisfy the SNR condition.

Figure 10: Batch size transfer of 5-layer MLP on CIFAR-10. We compare training loss across different batch sizes using SGD, Adam, and Muon optimizers. Curves align when plotted against iterations, confirming effective transfer.

Figure 11: Batch size transfer of ViT on CIFAR-10. Similar to the MLP results, the ViT loss curves align across different batch sizes when plotted against iterations, confirming effective transfer for the Transformer architecture.

Figure 12: Failure of Standard Parametrization (SP) on ViT. As model width increases, the training dynamics under SP become unstable or diverge, failing to exhibit the hyperparameter transfer seen in

\mu

Figure 13:

\mu

P ViT on CIFAR-10. The optimizer configuration is SGD, Adam and Muon (with SGD boundary updates).

width	SP SGD	$\mu$ P SGD	SP Adam	$\mu$ P Adam	$\mu$ P Muon
256	$-1.50\pm 0.25$	$-3.55\pm 0.21$	$-10.50\pm 0.18$	$-9.75\pm 0.18$	$-8.25\pm 0.18$
512	$-1.70\pm 0.27$	$-3.60\pm 0.14$	$-10.90\pm 0.29$	$-9.65\pm 0.22$	$-8.75\pm 0.31$
1024	$-2.25\pm 0.25$	$-3.25\pm 0.00$	$-11.10\pm 0.22$	$-9.75\pm 0.18$	$-8.55\pm 0.11$
2048	$-2.85\pm 0.14$	$-3.15\pm 0.14$	$-11.55\pm 0.21$	$-9.15\pm 0.14$	$-8.75\pm 0.25$
4096	$-3.45\pm 0.21$	$-2.90\pm 0.14$	$-12.30\pm 0.21$	$-9.10\pm 0.14$	$-8.65\pm 0.22$
8192	$-4.00\pm 0.18$	$-2.90\pm 0.14$	$-12.85\pm 0.14$	$-8.90\pm 0.14$	$-8.65\pm 0.14$

Appendix

Table 6: Optimal

\log_{2}

learning rate (by test accuracy) at each width for a 5-layer MLP on CIFAR-10, reported as mean

\pm

std over 5 seeds. Slope

\beta

is the regression slope of the optimal

\log_{2}

learning rate against

\log_{2}

width, and Acc is the test accuracy (%) at the largest width.

width	$\mu$ P SGD	$\mu$ P Adam	$\mu$ P Muon
192	$-1.35\pm 0.34$	$-8.50\pm 0.25$	$-5.85\pm 0.14$
384	$-1.40\pm 0.22$	$-8.05\pm 0.11$	$-5.70\pm 0.11$
576	$-1.00\pm 0.35$	$-8.10\pm 0.29$	$-5.65\pm 0.14$
768	$-0.95\pm 0.45$	$-8.10\pm 0.22$	$-5.80\pm 0.21$
960	$-1.15\pm 0.22$	$-8.30\pm 0.21$	$-5.65\pm 0.14$
Slope $\beta$	$+0.16$	$+0.10$	$+0.06$

Appendix

Table 7: Optimal

\log_{2}

learning rate (by test accuracy) at each width for a ViT on CIFAR-10, reported as mean

\pm

std over 5 seeds. Slope

\beta

and Acc are defined as in Table 6 .

Figure 14: Coordinate check for MLP (SP) with SNR condition. Even with the SNR condition applied, standard parametrization often fails to preserve feature scale across widths.

Figure 15: Coordinate check for MLP (SP) without SNR condition. Without privacy-aware scaling, feature activations diverge significantly as width increases.

Figure 16: Coordinate check for ViT (SP). Similar to the MLP case, SP leads to inconsistent activation scales in Vision Transformers.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization
Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon Optimizer Deep Learning Optimization
King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia
Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers
Feb 24, 2026Akshita Gupta, Marieme Ngom, Sam Foreman +1Hyperparameter Transfer Adaptive Gradient Methods
Purdue University · Argonne National Laboratory
Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective
Mar 3, 2026Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov +2Differentially Private Stochastic Gradient Descent Differential Privacy
University of Basel, Switzerland · University of Zürich, Switzerland

Oct 17, 2026J/K move · Enter open · S save

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Alexander Gaponov, Grigory Malinovsky, Egor Shulgin, Peter Richtarik

King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia

Muon Optimizer Deep Learning Optimization Differentially Private Stochastic Gradient Descent

Open paper AskPDF arXiv

Training deep neural networks requires optimizers that balance computational efficiency with formal data privacy. While orthogonalized update rules, such as the recently introduced Muon optimizer, have shown strong empirical performance and scalability, their behavior under Differential Privacy (DP) is not yet understood. In this paper, we propose DP-Muon, a differentially private variant of the Muon optimizer. By analyzing Muon's orthogonalized updates through the lens of a Linear Minimization Oracle (LMO), we show that the leading effect of per-example clipping is a positive rescaling of the gradient, to which the LMO is blind, so that only a small directional residual controlled by the centered gradient variance enters the optimization error. This yields an

\mathcal O(K^{-1/4})

convergence rate, up to logarithmic factors, to a clipping-dependent neighborhood at a fixed noise scale and, with noise calibrated to a fixed

(\varepsilon,δ)

budget, the standard

(\sqrt d/(n\varepsilon))^{1/2}

privacy--utility bound of DP-SGD, without assuming bounded gradients. Furthermore, we investigate the hyperparameter landscape of DP-Muon and document how the optimal learning rate, momentum, and batch size change under privacy. Empirically, on image classification DP-Muon matches or improves on DP-SGD and DP-Adam, whereas on language modeling the best optimizer depends on the batch size, with DP-Muon best at small batches and DP-Adam best near the privacy-optimal batch size.

Abstract

Neural Network Optimization

Explore similar work

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Extending μμμP: Spectral Conditions for Feature Learning Across Optimizers

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers