Inner Momentum for Differentially Private Muon

Bishnu Bhusal; Minh Vu; Ben Southworth; Geigh Zollicoffer; Rohit Chadha; Manish Bhattarai

cs.LGOct 2, 2026

Inner Momentum for Differentially Private Muon

Authors: Bishnu Bhusal, Minh Vu, Ben Southworth, Geigh Zollicoffer, Rohit Chadha, Manish Bhattarai

Organizations: Los Alamos National Laboratory · University of Missouri · Georgia Institute of Technology

Abstract

Differentially private training clips each per-example gradient before adding noise. This clipping is radial for each example, yet unequal clipping factors can distort the relative singular-vector geometry of their average. Muon is particularly exposed to this effect, since its update is an approximate polar factor UV^T that depends only on the singular vectors that clipping can shift. To curb this degradation, we propose averaging each sampled example's Muon gradient over the current model and a short history of recent models before clipping. The clipped batch matrix then separates into a common rescaling and a covariance residual R between sampled gradients and clipping values, with ||R||_F <= sigma_lambda sigma_G, bounding the clipping-induced distortion directly. We further show that a finite Newton-Schulz iteration preserves the polar factor of its input under these spectral conditions, confirming that our correction survives orthogonalization. In private GPT-2 fine-tuning on E2E and DART at epsilon in {1, 2, 4, 8}, DP-Muon-IM improves BLEU and ROUGE-L over DP-Muon in every seed-matched comparison, and non-private diagnostics show 2-4% lower pre-noise polar error.

Figures & tables

Figure 1: Intuition for DP-Muon with inner momentum (DP-Muon-IM). (a) DP-Muon-IM averages each example’s Muon gradients at the current and recent private model states before flat clipping. (b-c) Unequal clipping factors introduce a residual $R$ relative to a common rescaling of the batch mean. IM averages per-example gradients before this reweighting. (d) The residual can change the polar factor. We measure residual size and polar error separately; the utility experiments evaluate the complete method.

Method	Source	BLEU $\uparrow$	ROUGE-L $\uparrow$
DP-Muon	Kim and Fan (2026)	$63.78$	$66.96$
DP-Muon-BC	Kim and Fan (2026)	$63.66$	$66.71$
KF-DP-Muon-IM	This work	$64.77$	$67.37$
KF-DP-Muon	This work	$64.95$	$67.36$
DP-Muon	This work	$64.79$	$67.58$
DP-Muon-IM	This work	$65.65$	$67.80$

Table 1: E2E test results for GPT-2 fine-tuning at target

\epsilon=8

		DP-Muon		DP-Muon-IM
Task	Target $\epsilon$	BLEU $\uparrow$	ROUGE-L $\uparrow$	BLEU $\uparrow$	ROUGE-L $\uparrow$
E2E	$1$	$57.64\pm 1.97$	$62.71\pm 0.21$	$60.21\pm 2.65$	$64.41\pm 1.02$
	$2$	$63.05\pm 0.34$	$65.89\pm 0.62$	$63.76\pm 0.26$	$66.43\pm 0.60$
	$4$	$64.10\pm 0.29$	$66.72\pm 0.18$	$64.83\pm 0.57$	$67.03\pm 0.39$
	$8$	$64.74\pm 0.05$	$67.31\pm 0.25$	$65.34\pm 0.39$	$67.60\pm 0.17$
DART	$1$	$32.34\pm 0.59$	$50.20\pm 0.23$	$33.19\pm 0.42$	$50.91\pm 0.16$

Table 2: Test generation after ten epochs: mean

\pm

standard deviation over seeds

37

–

39

Figure 2: Effect of history length in separate utility and geometry experiments. Left: validation BLEU versus training time after ten epochs. Right: pre-noise polar-error reduction before optimizer update step

10

, with Muon-only history.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Target $\epsilon$	DP-Muon (No-IM)	DP-Muon-IM
E2E	$1$	$(0.0016,0.0028)$	$(0.0016,0.0035)$
	$2$	$(0.0016,0.0035)$	$(0.0016,0.0035)$
	$4$	$(0.0016,0.0035)$	$(0.0020,0.0035)$
	$8$	$(0.0020,0.0035)$	$(0.0020,0.0035)$
DART	$1$	$(0.00105,0.00168)$	$(0.00105,0.00168)$
	$2$	$(0.0013125,0.00168)$	$(0.0013125,0.0021)$

Appendix

Table 3: Validation-selected base learning rates for DP-Muon (No-IM) and DP-Muon-IM; each pair is (Muon, AdamW). Both methods retain the same standard optimizer moments.

Task	Target $\epsilon$	Noise multiplier $\sigma_{\mathrm{DP}}$	Achieved $\epsilon$	$\delta$
E2E	$1$	$2.05078125$	$0.996290$	$1/84122$
	$2$	$1.2646484375$	$1.999911$	$1/84122$
	$4$	$0.889892578125$	$3.996703$	$1/84122$
	$8$	$0.67626953125$	$7.998976$	$1/84122$
DART	$1$	$1.767578125$	$0.993732$	$1/125318$
	$2$	$1.123046875$	$1.998849$	$1/125318$

Appendix

Table 4: Per-run PRV calibration for our utility runs, with

\delta=1/(2N)

at every budget. Achieved

\epsilon

is rounded.

History enabled	Test NLL $\downarrow$	BLEU $\uparrow$	ROUGE-L $\uparrow$
Neither	$1.2842$	$63.66$	$66.90$
Muon only	$1.2779$	$63.76$	$66.95$
AdamW only	$1.2686$	$64.37$	$67.12$
Both	$1.2586$	$65.30$	$67.47$

Appendix

Table 5: Optimizer-group ablation on E2E at target

\epsilon=4

and

\delta=1/84122

. NLL is measured in nats per target token.

Figure 3: Corrected three-seed polar diagnostic at

(K,\gamma)=(4,0.9)

. Relative reduction is computed within each seed as

100(1-E_{\mathrm{IM}}/E_{\mathrm{current}})

, then averaged. Faint lines show individual seeds; bars and shading are pointwise sample SD, not confidence intervals.

Figure 4: Normalized clipping residual

\eta_{R}

on E2E for historical averaging (

K=4

\gamma=0.9

) and a single current-state evaluation at the same model states. Points show means over training seeds

42

–

44

; bars show sample standard deviation, not confidence intervals. All seven pre-update checkpoints are shown. Clipping uses the complete raw query at

C=1

; the residual metric covers Muon coordinates.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum
May 13, 2026Jihwan Kim, Chenglin FanDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Seoul National University
How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization
Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon Optimizer Deep Learning Optimization
King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia
MuCon: Clipped Muon Updates for LLM Training
May 26, 2026Albert YiMuon Optimizer Spectral Methods

May 13, 2026J/K move · Enter open · S save

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

Jihwan Kim, Chenglin Fan

Seoul National University

Deep Learning Optimization Differentially Private Stochastic Gradient Descent Muon Optimizer

Open paper AskPDF arXiv

We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order

2K+2

. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.

Abstract

Muon Optimizer

Explore similar work

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

MuCon: Clipped Muon Updates for LLM Training