Convergence of Practical Muon

Haonan Wang; Yu Wu; Minghui Liwang; Xinlei Yi; Yiguang Hong

cs.LGSep 27, 2026

Convergence of Practical Muon

Authors: Haonan Wang, Yu Wu, Minghui Liwang, Xinlei Yi, Yiguang Hong

Organizations: Tongji University · Shanghai Research Institute for Intelligent Autonomous Systems

Abstract

Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$ ; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an $\mathcal{O}(T^{-1/4})$ convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of $\sqrt{d}$ , where $T$ is the iteration horizon and $d$ is the parameter dimension. Experiments further support the theoretical convergence results.

Figures & tables

Reference	WD	NS	Nonconvex	Convergence rate
Shen et al. (2026)	$\times$	Exact	$\checkmark$	$\mathcal{O}(\sqrt{r}/T^{1/4})$
Kim and Oh (2026)	$\times$	Standard	$\checkmark$	$\mathcal{O}(\sqrt{r}/T^{1/4})^{\dagger}$
Choudhury et al. (2026)	$\times$	Standard	$\checkmark$	$\mathcal{O}(r/T^{1/4})$
Do et al. (2026)	$\times$	Practical	Strongly convex	neighborhood (obj.)
Qian and Richtárik (2026)	$\times$	Std./Pract.	$\checkmark$	$\mathcal{O}(r/T^{1/4})^{\dagger}$
Chen et al. (2026)	$\checkmark$	Exact	$\checkmark$	neighborhood (KKT)

Table 1: Comparison of existing convergence results for Muon. The rate column uses

\mathbb{E}\|\nabla f\|_{\mathrm{F}}

unless a KKT stationarity measure or expected objective gap (obj.) is indicated.

\dagger

: The nuclear norm rates established in the original papers are converted to Frobenius norm rates under our assumptions. Std. and Pract. denote standard and practical NS, respectively.

(a) One-step contraction
$h\backslash\kappa$	$100$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$
$1$	$4.9749\times 10^{-5}$	$0.0762$	$0.8655$	$0.9865$	$0.9987$
$10$	$5.4271\times 10^{-7}$	$0.7462$	$0.9744$	$0.9974$	$0.9997$
$100$	$0.0839$	$0.8685$	$0.9868$	$0.9987$	$0.9999$
(b) Full trajectory (numerical simulations)
$h\backslash\kappa$	$100$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$

Table 2: One-step and full trajectory comparisons for the ill-conditioned quadratic. Panel (a) reports

f(W_{1}^{\mathrm{P}})/f(W_{1}^{\mathrm{S}})

; panel (b) reports

T_{\mathrm{P}}/T_{\mathrm{S}}

, where

T_{\mathrm{P}}

and

T_{\mathrm{S}}

are the numbers of iterations required by practical and standard NS, respectively, to reduce the objective to

10^{-6}

of its initial value.

Figure 1: Empirical $\mathcal{O}(T^{-1/4})$ convergence and $\sqrt{d}$ improvement over AdamW on ResNet-32.

Figure 2: Training loss and orthogonalization error factor $(\hskip-1.00006pt1\hskip-0.89996pt-\hskip-0.50003pt\epsilon_{t\hskip-0.40974pt,\hskip-0.40974pt5}\hskip-1.00006pt)^{\hskip-0.81949pt-\hskip-0.81949pt1}$ on the 124M GPT-2-style model.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Sharp bounds for five-step Newton–Schulz maps.

Dimension (M)	Muon (Practical NS + WD)	Muon (Standard NS + WD)	AdamW
0.47	$(0.1,0.0625)$	$(0.1,0.0625)$	$(0.01,0.0625)$
1.86	$(0.05,0.1)$	$(0.05,0.1)$	$(0.01,0.15)$
7.41	$(0.02,0.3)$	$(0.05,0.1)$	$(0.001,0.04)$
66.46	$(0.05,0.15)$	$(0.0075,0.35)$	$(0.001,0.0625)$
118.11	$(0.01,0.6)$	$(0.1,0.0625)$	$(0.0003,0.1)$
184.50	$(0.02,0.4)$	$(0.01,0.35)$	$(0.0003,0.2)$

Appendix

Table 3: Selected stepsize and WD pairs for ResNet-32.

Figure 4: Frobenius gradient norm convergence rates for ResNet-32 models of varying dimensions. The gradient norms exhibit an overall $T^{-1/4}$ trend, with Muon generally converging faster than AdamW.

Algorithm \ Dimension (M)	0.47	1.86	7.41	66.46	118.11	184.50
Muon (practical + WD)	$59.64\pm 0.21$	$66.35\pm 0.38$	$70.22\pm 0.25$	$73.66\pm 0.18$	$72.74\pm 0.62$	$74.57\pm 0.04$
Muon (standard + WD)	$58.71\pm 0.36$	$65.59\pm 0.31$	$68.97\pm 0.38$	$72.28\pm 0.20$	$70.88\pm 0.40$	$72.92\pm 0.15$
AdamW	$45.95\pm 1.15$	$52.29\pm 1.88$	$57.03\pm 0.55$	$59.79\pm 1.61$	$62.69\pm 0.37$	$63.17\pm 0.62$

Appendix

Table 4: CIFAR-100 test top-1 accuracy (over three seeds) for ResNet-32 models of varying dimensions after 2,000 iterations. Muon consistently achieves higher accuracy than AdamW.

Figure 5: Loss curves for ResNet-32 models of varying dimensions. Muon consistently achieves lower training and test losses than AdamW.

Figure 6: Joint sweep over stepsize and WD for ResNet-32 models of varying dimensions.

Model	Muon (Practical NS + WD)	Muon (Standard NS + WD)	AdamW
124M GPT-2-style	$(0.02,0.000625)$	$(0.02,0.0025)$	$(0.001,0.01)$
355M GPT-2-style	$(0.02,0.02)$	$(0.02,0.0025)$	$(0.0003,0.00125)$
0.6B Qwen3-style	$(0.04,0.0025)$	$(0.02,0.0025)$	$(0.0015,0.000625)$

Appendix

Table 5: Selected stepsize and WD pairs for language model training.

Figure 7: Loss curves across the three language models. Muon consistently achieves lower training and validation losses than AdamW.

Figure 8: Momentum singular-value distributions for the query projection in the middle transformer block of the three language models. The distributions span a broad range and include very small normalized singular values.

Figure 9: Orthogonalization errors $\epsilon_{t,5}$ and the corresponding factors $(1-\epsilon_{t,5})^{-1}$ for practical and standard NS across the three language models. Practical NS consistently exhibits smaller errors and factors, while the large factor values need not imply slow convergence.

Figure 10: Joint sweep over stepsize and WD for language model training.

Explore similar work

May 11, 2026

Muown: Row-Norm Control for Muon Optimization

Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the

\ell_\infty

geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.

Kai Lion, Florian Hübler, Bingcong Li +2

Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Language Model Pretraining Muon Optimizer

Ask PDF

Feb 19, 2026

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers' matrix structure via orthogonalized momentum, showing superior performance in large language model training. We propose a new optimizer and a diagonal extension, NAMO and NAMO-D, providing the first principled integration of orthogonalized momentum with norm-based Adam-type noise adaptation. NAMO scales orthogonalized momentum using a single adaptive stepsize, preserving orthogonality while improving upon Muon at negligible additional cost. NAMO-D instead right-multiplies orthogonalized momentum by a diagonal matrix with clamped entries. This design enables neuron-wise noise adaptation and aligns with the common near block-diagonal Hessian structure. Under standard assumptions, we establish optimal convergence rates for both algorithms in the deterministic setting and show that, in the stochastic setting, their convergence guarantees adapt to the noise level of stochastic gradients. Experiments on pretraining GPT-2 models demonstrate improved performance of both NAMO and NAMO-D compared to the AdamW and Muon baselines, with NAMO-D achieving further gains over NAMO via an additional clamping hyperparameter that balances the competing goals of maintaining a well-conditioned update direction and leveraging fine-grained noise adaptation.

Minxin Zhang, Yuxuan Liu, Hayden Schaeffer

Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA

Deep Learning Optimization Adaptive Gradient Methods Muon Optimizer Stochastic Optimization Convergence

Ask PDF

Sep 30, 2026

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one global orthogonalization. Under gradient-dependent

(\mathcal L_0,\mathcal L_1,q)

-smoothness and conditionally unbiased stochastic gradients with bounded layer-wise variance, we establish an

\mathcal O(T^{-1/4})

bound on the expected average Frobenius gradient norm. The analysis retains the Nesterov recursion and requires neither bounded stochastic gradients, symmetric noise, nor a uniform positive lower bound on the nonzero output singular values. Its constants contain no explicit matrix-dimension or rank factors when the number of blocks and problem constants are fixed. The proof follows a descent inequality and a decomposition of the momentum tracking error into initialization, noise, and drift. For the original five-step quintic, we verify the required scalar-map bounds analytically; the result also allows step-dependent coefficients satisfying the same bounds. A complementary nuclear-norm result quantifies rank dependence under a stronger spectral condition. The vanishing rate uses coupled learning-rate and momentum schedules, including the standard single-coefficient Nesterov rule.

Hanyng Peng, Hui Wang, Yue Yu

Pengcheng Laboratory, Shenzhen, China

Newton-Schulz Iteration Gradient Descent Nonconvex Stochastic Optimization Muon Optimizer Stochastic Optimization Convergence

Ask PDF