How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Alexander Gaponov; Grigory Malinovsky; Egor Shulgin; Peter Richtarik

cs.LGOct 17, 2026

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

Authors: Alexander Gaponov, Grigory Malinovsky, Egor Shulgin, Peter Richtarik

Organizations: King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia

Abstract

Training deep neural networks requires optimizers that balance computational efficiency with formal data privacy. While orthogonalized update rules, such as the recently introduced Muon optimizer, have shown strong empirical performance and scalability, their behavior under Differential Privacy (DP) is not yet understood. In this paper, we propose DP-Muon, a differentially private variant of the Muon optimizer. By analyzing Muon's orthogonalized updates through the lens of a Linear Minimization Oracle (LMO), we show that the leading effect of per-example clipping is a positive rescaling of the gradient, to which the LMO is blind, so that only a small directional residual controlled by the centered gradient variance enters the optimization error. This yields an $\mathcal O(K^{-1/4})$ convergence rate, up to logarithmic factors, to a clipping-dependent neighborhood at a fixed noise scale and, with noise calibrated to a fixed $(\varepsilon,δ)$ budget, the standard $(\sqrt d/(n\varepsilon))^{1/2}$ privacy--utility bound of DP-SGD, without assuming bounded gradients. Furthermore, we investigate the hyperparameter landscape of DP-Muon and document how the optimal learning rate, momentum, and batch size change under privacy. Empirically, on image classification DP-Muon matches or improves on DP-SGD and DP-Adam, whereas on language modeling the best optimizer depends on the batch size, with DP-Muon best at small batches and DP-Adam best near the privacy-optimal batch size.

Figures & tables

Work	Gradient assumption	Noise in rate	Guarantee	Momentum–LMO
Koloskova et al. (2023)	centered variance	fixed	avg. $\mathbb{E}\\|\nabla f\\|\lesssim K^{-1/4}+\min\{\sigma,\sigma^{2}/c\}$	–
Yang et al. (2022)	a.s. relative envelope	fixed / calibrated	$\mathbb{E}\min_{k}\\|\nabla f\\|\lesssim K^{-1/4}$ ; calibrated: $(dc^{3}\log\frac{1}{\delta})^{1/4}(n\varepsilon)^{-1/2}$	–
Chen et al. (2020)	none (distance to symmetric)	calibrated	weighted clipped stationarity $\lesssim c\sqrt{d}/(n\varepsilon)$ $+$ Wasserstein term	–
Shulgin and Richtárik (2024)	centered $q$ -th moment	fixed	weighted avg. $\mathbb{E}\\|\nabla f\\|^{2}\lesssim K^{-1/2}+\mathcal{N}_{\rm QC}$	–
Vatan Khah et al. (2026)	centered $p$ -th moment	fixed / calibrated	$\min_{k}\\|\nabla f\\|^{2}\lesssim\widetilde{\mathcal{O}}(K^{-1/2})+\mathcal{N}_{\rm clip}$ (w.h.p.)	–
Kim and Fan (2026)	raw-gradient moment	fixed multiplier, $c$ grows with $K$	avg. $\mathbb{E}\\|\nabla f\\|_{\rm nuc}\lesssim K^{-1/4}\sqrt{\log K}$ or $K^{-1/6}$	matrix, finite NS

Table 1: Optimization guarantees for per-example clipping. “Fixed”: the rate in

K

holds at a constant noise scale, so the privacy budget grows with

K

; “calibrated”: noise calibrated to a fixed

(\varepsilon,\delta)

is substituted into the bound. Constants and logarithms are suppressed unless shown. Theorem numbers, exact assumptions, and conventions are listed in Appendix E .

Figure 1: Best test loss as a function of batch size for SGD, Adam, and Muon. All plotted values represent the optimal configuration found across the learning rate and momentum grid search. (Left) Standard non-private training. (Right) Differentially private (DP) training for 80 epochs. We perform three independent runs to compute the reported deviation intervals.

Figure 2: Language modeling (30M GPT): best loss versus batch size at the best learning rate and momentum for each batch size. Left: non-private. Right: DP (

\varepsilon=8

), including Scion. Note the different batch-size and loss ranges.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Norm $\\|\cdot\\|$	Method	$\operatorname{LMO}(g)$	$\rho_{1}$	$\rho_{2}$	$\mathbb{E}\\|Z\\|_{\star}$
Spectral	Muon	$-UV^{\top}$ for $g=U\Sigma V^{\top}$	$\sqrt{\min\{m,n^{\prime}\}}$	$1$	$\asymp\min\{m,n^{\prime}\}\sqrt{\max\{m,n^{\prime}\}}$
$\ell_{\infty}$	sign descent	$-\operatorname{sign}(g)$	$\sqrt{d}$	$1$	$d\sqrt{2/\pi}$
$\ell_{2}$	normalized GD	$-g/\\|g\\|_{2}$	$1$	$1$	$\approx\sqrt{d}$
$\ell_{1}$	coordinate descent	$-\operatorname{sign}(g_{j})e_{j}$ , $j\in\arg\max_{i}\|g_{i}\|$	$1$	$\sqrt{d}$	$\asymp\sqrt{\log d}$

Appendix

Table 2: Common LMO geometries on

\mathbb{R}^{d}

(or on

\mathbb{R}^{m\times n^{\prime}}

for the spectral norm).

CIFAR-10 ( $n=50$ k, 80 epochs, $d\approx 2$ M)					Language modeling ( $n\approx 586$ k, 1 epoch, $d\approx 30$ M)
$B$	$K$	$\sigma_{\rm mult}$	NSR	DP/samp.	$B$	$K$	$\sigma_{\rm mult}$	NSR	DP/samp.
64	62500	0.58	12.9	103	128	4578	0.40	17.0	192
128	31250	0.65	7.2	81	256	2289	0.42	8.9	142
256	15625	0.74	4.1	65	512	1144	0.43	4.7	105
512	7812	0.87	2.4	55	1024	572	0.46	2.4	78
1024	3906	1.07	1.5	47	2048	286	0.48	1.3	59

Appendix

Table 3: Noise multiplier for

(\varepsilon,\delta)=(8,1/(2n))

from the Rényi-DP accountant, ratio of the privacy-noise norm to the clipping radius (

\mathrm{NSR}=\sigma_{\rm mult}\sqrt{d}/B

), and ratio of the root-mean-square norm of the privacy noise to the root-mean-square bound

c/\sqrt{B}

on the sampling error of clipped gradients (DP/samp.

=\sigma_{\rm mult}\sqrt{d/B}

). Since

c/\sqrt{B}

is an upper bound, this ratio is a lower bound on the ratio of root-mean-square noise levels; it does not bound individual realizations. Bold: observed DP-optimal batch sizes (Figures 1 and 2 ). The row

B=2500

for CIFAR-10 is included for the accountant only; it is not part of the DP grid of Appendix G.1 .

Figure 3: Left: norm of the privacy noise added to the averaged gradient relative to the clipping radius, for the two experimental settings; shaded regions mark the observed DP-optimal batch sizes. Right: decay exponent of the privacy noise in the batch size; sampling noise has exponent

1/2

	non-private				private ( $\varepsilon=8$ )
$B$	SGD	Adam	Muon	spread	SGD	Adam	Muon	spread	Scion
128	5.55	3.99	3.86	1.69	6.84	6.87	6.78	0.09	6.84
256	5.98	4.13	3.91	2.07	6.75	6.75	6.67	0.08	6.79
512	6.45	4.43	3.97	2.48	6.62	6.49	6.55	0.13	6.67
1024	6.94	4.82	4.10	2.84	6.55	6.33	6.52	0.22	6.54
2048	–	–	–	–	6.68	6.36	6.51	0.32	6.52

Appendix

Table 4: Best loss per optimizer and batch size for language modeling (from the heatmaps; lower is better) and spread across optimizers. Non-private runs use

B\leq 1024

; Scion was run only with privacy and is excluded from the spread for comparability.

Panel of Figure 4 — Figure 4: CNN, non-private: peak test accuracy for each batch size and learning rate ( top ) or momentum ( bottom ).

Panel of Figure 5 — Figure 5: CNN, DP ( $\varepsilon=8$ ): peak test accuracy for each batch size and learning rate ( top ) or momentum ( bottom ).

Figure 6: CNN,

B=1024

: peak test accuracy and best test loss versus the privacy budget

\varepsilon

, with hyperparameters tuned at

\varepsilon=8

. Shaded regions show the standard deviation over three runs.

Panel of Figure 7 — Figure 7: GPT (30M), non-private: best loss for each batch size and learning rate ( top ) or momentum ( bottom ).

Panel of Figure 8 — Figure 8: GPT (30M), DP ( $\varepsilon=8$ ): best loss for each batch size and learning rate ( top ) or momentum ( bottom ).

Figure 9: GPT (30M), DP ( $\varepsilon=8$ ): best loss for every learning rate (columns) and momentum (rows), for $B\in\{128,256,512,1024\}$ . “Gluon Scale 1” denotes the Scion variant used in Figure 2 .

Figure 10: GPT (30M), DP ( $\varepsilon=8$ ): best loss for every learning rate (columns) and momentum (rows), for $B\in\{2048,4096,8192,16384\}$ . “Gluon Scale 1” denotes the Scion variant used in Figure 2 .

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

May 13, 2026

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order

2K+2

. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.

Jihwan Kim, Chenglin Fan

Seoul National University

Deep Learning Optimization Differentially Private Stochastic Gradient Descent Muon Optimizer

Ask PDF

Oct 2, 2026

Inner Momentum for Differentially Private Muon

Differentially private training clips each per-example gradient before adding noise. This clipping is radial for each example, yet unequal clipping factors can distort the relative singular-vector geometry of their average. Muon is particularly exposed to this effect, since its update is an approximate polar factor UV^T that depends only on the singular vectors that clipping can shift. To curb this degradation, we propose averaging each sampled example's Muon gradient over the current model and a short history of recent models before clipping. The clipped batch matrix then separates into a common rescaling and a covariance residual R between sampled gradients and clipping values, with ||R||_F <= sigma_lambda sigma_G, bounding the clipping-induced distortion directly. We further show that a finite Newton-Schulz iteration preserves the polar factor of its input under these spectral conditions, confirming that our correction survives orthogonalization. In private GPT-2 fine-tuning on E2E and DART at epsilon in {1, 2, 4, 8}, DP-Muon-IM improves BLEU and ROUGE-L over DP-Muon in every seed-matched comparison, and non-private diagnostics show 2-4% lower pre-noise polar error.

Bishnu Bhusal, Minh Vu, Ben Southworth +3

Los Alamos National Laboratory · University of Missouri · Georgia Institute of Technology

Muon Optimizer Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent

Ask PDF

Oct 16, 2026

Towards Hyperparameter Transfer for Differentially Private Optimization

Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization (

μ

P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-

μ

P, a theoretically grounded framework that extends

μ

P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-

μ

P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.

Tianze Wang, Zhiqi Bu, Linjun Zhang

Department of Statistics, Rutgers University · Independent Researcher

Neural Network Optimization Hyperparameter Transfer Layer-Wise Learning Rate Adaptation Differentially Private Stochastic Gradient Descent

Ask PDF

Abstract

Muon Optimizer

Explore similar work

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

Inner Momentum for Differentially Private Muon

Towards Hyperparameter Transfer for Differentially Private Optimization