Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Xiaohui Xie

cs.LGSep 26, 2026

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Authors: Xiaohui Xie

Organizations: Department of Computer Science University of California, Irvine

Abstract

Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched learning rate reproduces Muon's first-order mean response. The stochastic update is a different matter: after the response is matched, Muon retains a nonlinear residual that is uncorrelated with the input noise and contributes additional covariance. A Hermite expansion shows how momentum acts on this residual. Its higher-order components decorrelate faster than the linear component, so momentum suppresses the residual's accumulated covariance relative to the linear part, but never eliminates it. In a local quadratic surrogate that evaluates the residual on the stationary noise buffer, the residual adds stationary covariance and raises the stationary loss floor at every stable step size, while leaving the contraction dynamics unchanged. Simulations of the full nonlinear recursion on quadratics and measurements on frozen transformer gradients support each step of this picture. Together, the results make a theoretical case for replacing orthogonalization by response-matched momentum SGD once optimization becomes noise-dominated.

Figures & tables

Figure 1: At small signal the mean polar update is the predicted scaled gradient step. Monte Carlo estimates of $\mathbb{E}P(Y+\sigma Z)$ at $\|Y\|_{F}=0.1\sigma$ for $N\times N$ matrices: (a) slope along $Y$ against the noise scale ( $N=16$ ), (b) slope against the matrix size ( $\sigma=1$ ), with the asymptote $8/(3\pi\sqrt{N})$ , and (c) estimated against predicted entries ( $N=12$ , $\sigma=1$ ). Dashed lines are the independent predictions $c_{N,N}/\sigma$ ; error bars are 95% intervals.

Figure 2: Momentum substantially reduces the covariance excess of orthogonalization but does not eliminate it, and the prediction carries over to the stationary loss of the full nonlinear recursion. (a) Long-window ratio $\mathcal{R}_{\beta}$ of Muon to matched momentum SGD for exact polar updates under EMA (solid), with the degree-three bound ( 22 ) (dotted). (b) Covariance ratio without momentum ( $\beta=0$ , gray, equal to $\mathcal{R}_{0}$ ) and with EMA or Nesterov momentum at $\beta=0.95$ (long-window limit), for exact polar (solid) and NS5 (hatched): momentum removes most, but not all, of the excess for both maps and both filters. (c) Stationary excess-loss ratio of nonlinear exact-polar EMA Muon to matched momentum SGD on quadratics, at three matched learning rates $\eta_{L}$ , against the prediction $\mathcal{R}_{\beta}$ (dashed).

	exact polar		NS5
$\eta_{L}\lambda_{\max}$	nonlinear Muon	surrogate	nonlinear Muon	surrogate
$0.02$	$1.081\pm 0.001$	$1.088$	$1.055\pm 0.001$	$1.060$
$0.05$	$1.078\pm 0.001$	$1.094$	$1.052\pm 0.001$	$1.065$
$0.1$	$1.074\pm 0.001$	$1.104$	$1.049\pm 0.001$	$1.072$
$0.2$	$1.070\pm 0.001$	$1.123$	$1.045\pm 0.001$	$1.085$
$0.4$	$1.064\pm 0.002$	$1.159$	$1.041\pm 0.002$	$1.111$

Table 1: The stationary-loss gap persists at every stable step size. Ratio of stationary losses of nonlinear EMA Muon (

\beta=0.9

) to matched momentum SGD on a

16\times 16

matrix quadratic, over a

320

-fold range of effective steps (95% intervals over 64 paired chains), with the frozen-residual prediction of Theorem 6.2 (NS5: the same formula with the NS5 Hermite weights). The measured ratio stays between

1.04

and

1.13

, above one and well below

\mathcal{R}_{0}

; the surrogate is accurate at small steps and conservative at large ones.

Figure 3: The residual persists under real transformer gradient noise and in the reference implementation, and momentum reduces it only partly. Accumulated covariance ratio $\mathcal{R}^{\rm lin}_{256}$ for six hidden matrices of a frozen transformer (effective batch 64). (a) Real noise (solid) against isotropic Gaussian controls (dotted), NS5 in float32. (b) Exact polar and NS5 in float32 and bfloat16 for three of the matrices at $\beta=0$ , EMA $0.95$ , and Nesterov $0.95$ ; black lines mark the isotropic controls. (c) The main checkpoint (step 400, solid) and an earlier one (step 100, dashed). Error bars are 95% intervals conditional on the stored gradient pool.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Shape	Map	$\mathcal{R}_{0}$	EMA $0.9$	EMA $0.95$	bound $0.95$	EMA $\beta\uparrow 1$	Nesterov $0.95$	$w_{3}$ share
$1\times 1$	polar	$1.571$	$1.097\pm 0.001$	$1.091\pm 0.001(1.092)$	$1.191$	$1.088\pm 0.001$	$1.062\pm 0.001$	$0.29$
$1\times 16$	polar	$1.032$	$1.010\pm 0.000$	$1.010\pm 0.000$	$1.011$	$1.010\pm 0.000$	$1.007\pm 0.000$	$0.88$
$1\times 16$	NS5	$1.031$	$1.010\pm 0.000$	$1.010\pm 0.000$	$1.011$	$1.010\pm 0.000$	$1.007\pm 0.000$	$0.88$
$4\times 4$	polar	$1.471$	$1.090\pm 0.000$	$1.086\pm 0.000$	$1.157$	$1.084\pm 0.000$	$1.059\pm 0.000$	$0.36$
$4\times 4$	NS5	$1.441$	$1.078\pm 0.000$	$1.074\pm 0.000$	$1.147$	$1.072\pm 0.000$	$1.051\pm 0.000$	$0.32$
$4\times 16$	polar	$1.086$	$1.026\pm 0.000$	$1.026\pm 0.000$	$1.029$	$1.026\pm 0.000$	$1.018\pm 0.000$	$0.80$

Appendix

Table 2: Accumulated-noise ratios from estimated Hermite weights (95% intervals over 16 batches). ‘bound’ is the degree-3 bound of Corollary 5.2 evaluated with the estimated

\mathcal{R}_{0}

; for NS5 it is the bound of Proposition B.1 . The scalar exact value is in parentheses. The

w_{3}

share is

w_{3}/\sum_{\ell\geq 3}w_{\ell}

Shape	Map	Filter	$\beta$	$\eta$	Predicted	Observed	RMS $\\|g\\|_{F}/(s\sqrt{v})$
$1\times 1$	polar	EMA	$0.0$	$0.001$	$1.571$	$1.594\pm 0.062$	$0.03$
$1\times 1$	polar	EMA	$0.0$	$0.00025$	$1.571$	$1.553\pm 0.052$	$0.01$
$1\times 1$	polar	EMA	$0.0$	$6.25e-05$	$1.571$	$1.557\pm 0.041$	$0.01$
$1\times 1$	polar	EMA	$0.5$	$0.001$	$1.208$	$1.206\pm 0.035$	$0.04$
$1\times 1$	polar	EMA	$0.5$	$0.00025$	$1.208$	$1.226\pm 0.028$	$0.02$
$1\times 1$	polar	EMA	$0.5$	$6.25e-05$	$1.208$	$1.217\pm 0.033$	$0.01$

Appendix

Table 3: All moving-chain results. Observed: ratio of mean excess risks of momentum Muon and momentum SGD with the same filter, with 95% delta-method intervals over 32 paired chains, conditional on the calibration constant. Predicted:

\mathcal{R}_{\beta}

from Table 2 (exact for the scalar case).

layer	noise	$\beta=0$	EMA .5	EMA .9	EMA .95	Nest. .9	Nest. .95	red. EMA	red. Nest.
L3 q	iso. Gaussian	1.506 $\pm$ 0.002	1.196 $\pm$ 0.001	1.119 $\pm$ 0.001	1.119 $\pm$ 0.001	1.079 $\pm$ 0.001	1.083 $\pm$ 0.002	0.76	0.84
L3 q	iso. Gaussian $+\bar{Y}$	1.511 $\pm$ 0.002	1.202 $\pm$ 0.001	1.126 $\pm$ 0.001	1.118 $\pm$ 0.001	1.089 $\pm$ 0.001	1.089 $\pm$ 0.001	0.77	0.83
L3 q	cov. Gaussian	1.537 $\pm$ 0.006	1.204 $\pm$ 0.004	1.118 $\pm$ 0.003	1.120 $\pm$ 0.003	1.078 $\pm$ 0.003	1.080 $\pm$ 0.003	0.78	0.85
L3 q	cov. Gaussian $+\bar{Y}$	1.542 $\pm$ 0.008	1.215 $\pm$ 0.006	1.168 $\pm$ 0.004	1.192 $\pm$ 0.004	1.111 $\pm$ 0.005	1.138 $\pm$ 0.003	0.65	0.75
L3 q	real	1.426 $\pm$ 0.005	1.167 $\pm$ 0.004	1.162 $\pm$ 0.004	1.190 $\pm$ 0.003	1.099 $\pm$ 0.004	1.134 $\pm$ 0.003	0.55	0.68
L3 v	iso. Gaussian	1.506 $\pm$ 0.002	1.196 $\pm$ 0.001	1.120 $\pm$ 0.001	1.120 $\pm$ 0.001	1.081 $\pm$ 0.001	1.084 $\pm$ 0.001	0.76	0.83

Appendix

Table 4:

\mathcal{R}^{\rm lin}_{256}

at the step-400 checkpoint, effective batch 64, NS5 in float32, for five noise sources and six filters. The last two columns give the reduction fraction

(\mathcal{R}_{0}-\mathcal{R}_{\beta})/(\mathcal{R}_{0}-1)

for EMA

0.95

and Nesterov

0.95

. Intervals are 95% jackknife intervals over 16 chain groups.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 149 papers

May 13, 2026

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Ask PDF

Jun 2, 2026

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers.

Xianliang Li, Zihan Zhang, Weiyang Liu +1

1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3

Deep Learning Optimization Momentum Methods Muon Optimizer Spectral Methods Gradient Descent Dynamics

Ask PDF

Jun 7, 2026

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning. However, most current orthogonalized methods are still paired with fixed, externally scheduled, or otherwise open-loop magnitude rules, so their scale is not directly calibrated from the realized optimization trajectory. Motivated by the closed-loop perspective behind Lipschitz-free and noise-adaptive methods, we propose OptMuon, a family of adaptive momentum orthogonalization methods for stochastic nonconvex optimization. OptMuon combines Muon-style polar-factor directions with a trajectory-dependent AdaGrad-Norm-type coefficient schedule, so that the update magnitude is determined by the observed gradient and momentum history rather than by a prescribed Lipschitz-dependent rule. The schedule does not use the smoothness constant, the variance level, or the bounded-gradient constant in parameter selection, and its running-maximum correction prevents isolated gradient spikes from causing excessive coefficient collapse. Under lower-boundedness, unbiased stochastic gradients with bounded variance, smoothness, and an almost-sure bounded stochastic-gradient condition, we prove two complementary expected-stationarity guarantees. OptMuon-A achieves the noise-adaptive rate

\tilde{\mathcal O}(T^{-1/2}+σ^{1/2}T^{-1/4})

under average smoothness, while OptMuon-I achieves

\tilde{\mathcal O}(T^{-1/2}+σ^{1/3}T^{-1/3})

under individual smoothness. In the zero-noise regime, both bounds automatically reduce to a nearly optimal deterministic first-order rate

\tilde{\mathcal O}(T^{-1/2})

without manual hyperparameter retuning. These results show that closed-loop scalar adaptation can be combined with Muon-style momentum orthogonalization while retaining noise adaptivity and zero-noise optimality up to logarithmic factors.

Ganzhao Yuan

Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China

Momentum Methods Nonconvex Stochastic Optimization Muon Optimizer Stochastic Optimization Convergence Adaptive Gradient Methods

Ask PDF

Abstract

Muon Optimizer

Explore similar work

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality