Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Xiaohui Xie

cs.LGSep 26, 2026

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Authors: Xiaohui Xie

Organizations: Department of Computer Science University of California, Irvine

Abstract

Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched learning rate reproduces Muon's first-order mean response. The stochastic update is a different matter: after the response is matched, Muon retains a nonlinear residual that is uncorrelated with the input noise and contributes additional covariance. A Hermite expansion shows how momentum acts on this residual. Its higher-order components decorrelate faster than the linear component, so momentum suppresses the residual's accumulated covariance relative to the linear part, but never eliminates it. In a local quadratic surrogate that evaluates the residual on the stationary noise buffer, the residual adds stationary covariance and raises the stationary loss floor at every stable step size, while leaving the contraction dynamics unchanged. Simulations of the full nonlinear recursion on quadratics and measurements on frozen transformer gradients support each step of this picture. Together, the results make a theoretical case for replacing orthogonalization by response-matched momentum SGD once optimization becomes noise-dominated.

Figures & tables

Figure 1: At small signal the mean polar update is the predicted scaled gradient step. Monte Carlo estimates of $\mathbb{E}P(Y+\sigma Z)$ at $\|Y\|_{F}=0.1\sigma$ for $N\times N$ matrices: (a) slope along $Y$ against the noise scale ( $N=16$ ), (b) slope against the matrix size ( $\sigma=1$ ), with the asymptote $8/(3\pi\sqrt{N})$ , and (c) estimated against predicted entries ( $N=12$ , $\sigma=1$ ). Dashed lines are the independent predictions $c_{N,N}/\sigma$ ; error bars are 95% intervals.

Figure 2: Momentum substantially reduces the covariance excess of orthogonalization but does not eliminate it, and the prediction carries over to the stationary loss of the full nonlinear recursion. (a) Long-window ratio $\mathcal{R}_{\beta}$ of Muon to matched momentum SGD for exact polar updates under EMA (solid), with the degree-three bound ( 22 ) (dotted). (b) Covariance ratio without momentum ( $\beta=0$ , gray, equal to $\mathcal{R}_{0}$ ) and with EMA or Nesterov momentum at $\beta=0.95$ (long-window limit), for exact polar (solid) and NS5 (hatched): momentum removes most, but not all, of the excess for both maps and both filters. (c) Stationary excess-loss ratio of nonlinear exact-polar EMA Muon to matched momentum SGD on quadratics, at three matched learning rates $\eta_{L}$ , against the prediction $\mathcal{R}_{\beta}$ (dashed).

	exact polar		NS5
$\eta_{L}\lambda_{\max}$	nonlinear Muon	surrogate	nonlinear Muon	surrogate
$0.02$	$1.081\pm 0.001$	$1.088$	$1.055\pm 0.001$	$1.060$
$0.05$	$1.078\pm 0.001$	$1.094$	$1.052\pm 0.001$	$1.065$
$0.1$	$1.074\pm 0.001$	$1.104$	$1.049\pm 0.001$	$1.072$
$0.2$	$1.070\pm 0.001$	$1.123$	$1.045\pm 0.001$	$1.085$
$0.4$	$1.064\pm 0.002$	$1.159$	$1.041\pm 0.002$	$1.111$

Table 1: The stationary-loss gap persists at every stable step size. Ratio of stationary losses of nonlinear EMA Muon (

\beta=0.9

) to matched momentum SGD on a

16\times 16

matrix quadratic, over a

320

-fold range of effective steps (95% intervals over 64 paired chains), with the frozen-residual prediction of Theorem 6.2 (NS5: the same formula with the NS5 Hermite weights). The measured ratio stays between

1.04

and

1.13

, above one and well below

\mathcal{R}_{0}

; the surrogate is accurate at small steps and conservative at large ones.

Figure 3: The residual persists under real transformer gradient noise and in the reference implementation, and momentum reduces it only partly. Accumulated covariance ratio $\mathcal{R}^{\rm lin}_{256}$ for six hidden matrices of a frozen transformer (effective batch 64). (a) Real noise (solid) against isotropic Gaussian controls (dotted), NS5 in float32. (b) Exact polar and NS5 in float32 and bfloat16 for three of the matrices at $\beta=0$ , EMA $0.95$ , and Nesterov $0.95$ ; black lines mark the isotropic controls. (c) The main checkpoint (step 400, solid) and an earlier one (step 100, dashed). Error bars are 95% intervals conditional on the stored gradient pool.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Shape	Map	$\mathcal{R}_{0}$	EMA $0.9$	EMA $0.95$	bound $0.95$	EMA $\beta\uparrow 1$	Nesterov $0.95$	$w_{3}$ share
$1\times 1$	polar	$1.571$	$1.097\pm 0.001$	$1.091\pm 0.001(1.092)$	$1.191$	$1.088\pm 0.001$	$1.062\pm 0.001$	$0.29$
$1\times 16$	polar	$1.032$	$1.010\pm 0.000$	$1.010\pm 0.000$	$1.011$	$1.010\pm 0.000$	$1.007\pm 0.000$	$0.88$
$1\times 16$	NS5	$1.031$	$1.010\pm 0.000$	$1.010\pm 0.000$	$1.011$	$1.010\pm 0.000$	$1.007\pm 0.000$	$0.88$
$4\times 4$	polar	$1.471$	$1.090\pm 0.000$	$1.086\pm 0.000$	$1.157$	$1.084\pm 0.000$	$1.059\pm 0.000$	$0.36$
$4\times 4$	NS5	$1.441$	$1.078\pm 0.000$	$1.074\pm 0.000$	$1.147$	$1.072\pm 0.000$	$1.051\pm 0.000$	$0.32$
$4\times 16$	polar	$1.086$	$1.026\pm 0.000$	$1.026\pm 0.000$	$1.029$	$1.026\pm 0.000$	$1.018\pm 0.000$	$0.80$

Appendix

Table 2: Accumulated-noise ratios from estimated Hermite weights (95% intervals over 16 batches). ‘bound’ is the degree-3 bound of Corollary 5.2 evaluated with the estimated

\mathcal{R}_{0}

; for NS5 it is the bound of Proposition B.1 . The scalar exact value is in parentheses. The

w_{3}

share is

w_{3}/\sum_{\ell\geq 3}w_{\ell}

Shape	Map	Filter	$\beta$	$\eta$	Predicted	Observed	RMS $\\|g\\|_{F}/(s\sqrt{v})$
$1\times 1$	polar	EMA	$0.0$	$0.001$	$1.571$	$1.594\pm 0.062$	$0.03$
$1\times 1$	polar	EMA	$0.0$	$0.00025$	$1.571$	$1.553\pm 0.052$	$0.01$
$1\times 1$	polar	EMA	$0.0$	$6.25e-05$	$1.571$	$1.557\pm 0.041$	$0.01$
$1\times 1$	polar	EMA	$0.5$	$0.001$	$1.208$	$1.206\pm 0.035$	$0.04$
$1\times 1$	polar	EMA	$0.5$	$0.00025$	$1.208$	$1.226\pm 0.028$	$0.02$
$1\times 1$	polar	EMA	$0.5$	$6.25e-05$	$1.208$	$1.217\pm 0.033$	$0.01$

Appendix

Table 3: All moving-chain results. Observed: ratio of mean excess risks of momentum Muon and momentum SGD with the same filter, with 95% delta-method intervals over 32 paired chains, conditional on the calibration constant. Predicted:

\mathcal{R}_{\beta}

from Table 2 (exact for the scalar case).

layer	noise	$\beta=0$	EMA .5	EMA .9	EMA .95	Nest. .9	Nest. .95	red. EMA	red. Nest.
L3 q	iso. Gaussian	1.506 $\pm$ 0.002	1.196 $\pm$ 0.001	1.119 $\pm$ 0.001	1.119 $\pm$ 0.001	1.079 $\pm$ 0.001	1.083 $\pm$ 0.002	0.76	0.84
L3 q	iso. Gaussian $+\bar{Y}$	1.511 $\pm$ 0.002	1.202 $\pm$ 0.001	1.126 $\pm$ 0.001	1.118 $\pm$ 0.001	1.089 $\pm$ 0.001	1.089 $\pm$ 0.001	0.77	0.83
L3 q	cov. Gaussian	1.537 $\pm$ 0.006	1.204 $\pm$ 0.004	1.118 $\pm$ 0.003	1.120 $\pm$ 0.003	1.078 $\pm$ 0.003	1.080 $\pm$ 0.003	0.78	0.85
L3 q	cov. Gaussian $+\bar{Y}$	1.542 $\pm$ 0.008	1.215 $\pm$ 0.006	1.168 $\pm$ 0.004	1.192 $\pm$ 0.004	1.111 $\pm$ 0.005	1.138 $\pm$ 0.003	0.65	0.75
L3 q	real	1.426 $\pm$ 0.005	1.167 $\pm$ 0.004	1.162 $\pm$ 0.004	1.190 $\pm$ 0.003	1.099 $\pm$ 0.004	1.134 $\pm$ 0.003	0.55	0.68
L3 v	iso. Gaussian	1.506 $\pm$ 0.002	1.196 $\pm$ 0.001	1.120 $\pm$ 0.001	1.120 $\pm$ 0.001	1.081 $\pm$ 0.001	1.084 $\pm$ 0.001	0.76	0.83

Appendix

Table 4:

\mathcal{R}^{\rm lin}_{256}

at the step-400 checkpoint, effective batch 64, NS5 in float32, for five noise sources and six filters. The last two columns give the reduction fraction

(\mathcal{R}_{0}-\mathcal{R}_{\beta})/(\mathcal{R}_{0}-1)

for EMA

0.95

and Nesterov

0.95

. Intervals are 95% jackknife intervals over 16 chain groups.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 149 papers

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum Methods Gradient Descent
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
Jun 7, 2026Ganzhao YuanMomentum Methods Nonconvex Stochastic Optimization
Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China

May 13, 2026J/K move · Enter open · S save

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen+2

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Abstract

Muon Optimizer

Explore similar work

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality