Convergence of Practical Muon

Haonan Wang; Yu Wu; Minghui Liwang; Xinlei Yi; Yiguang Hong

cs.LGSep 27, 2026

Convergence of Practical Muon

Authors: Haonan Wang, Yu Wu, Minghui Liwang, Xinlei Yi, Yiguang Hong

Organizations: Tongji University · Shanghai Research Institute for Intelligent Autonomous Systems

Abstract

Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$ ; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an $\mathcal{O}(T^{-1/4})$ convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of $\sqrt{d}$ , where $T$ is the iteration horizon and $d$ is the parameter dimension. Experiments further support the theoretical convergence results.

Figures & tables

Reference	WD	NS	Nonconvex	Convergence rate
Shen et al. (2026)	$\times$	Exact	$\checkmark$	$\mathcal{O}(\sqrt{r}/T^{1/4})$
Kim and Oh (2026)	$\times$	Standard	$\checkmark$	$\mathcal{O}(\sqrt{r}/T^{1/4})^{\dagger}$
Choudhury et al. (2026)	$\times$	Standard	$\checkmark$	$\mathcal{O}(r/T^{1/4})$
Do et al. (2026)	$\times$	Practical	Strongly convex	neighborhood (obj.)
Qian and Richtárik (2026)	$\times$	Std./Pract.	$\checkmark$	$\mathcal{O}(r/T^{1/4})^{\dagger}$
Chen et al. (2026)	$\checkmark$	Exact	$\checkmark$	neighborhood (KKT)

Table 1: Comparison of existing convergence results for Muon. The rate column uses

\mathbb{E}\|\nabla f\|_{\mathrm{F}}

unless a KKT stationarity measure or expected objective gap (obj.) is indicated.

\dagger

: The nuclear norm rates established in the original papers are converted to Frobenius norm rates under our assumptions. Std. and Pract. denote standard and practical NS, respectively.

(a) One-step contraction
$h\backslash\kappa$	$100$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$
$1$	$4.9749\times 10^{-5}$	$0.0762$	$0.8655$	$0.9865$	$0.9987$
$10$	$5.4271\times 10^{-7}$	$0.7462$	$0.9744$	$0.9974$	$0.9997$
$100$	$0.0839$	$0.8685$	$0.9868$	$0.9987$	$0.9999$
(b) Full trajectory (numerical simulations)
$h\backslash\kappa$	$100$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$

Table 2: One-step and full trajectory comparisons for the ill-conditioned quadratic. Panel (a) reports

f(W_{1}^{\mathrm{P}})/f(W_{1}^{\mathrm{S}})

; panel (b) reports

T_{\mathrm{P}}/T_{\mathrm{S}}

, where

T_{\mathrm{P}}

and

T_{\mathrm{S}}

are the numbers of iterations required by practical and standard NS, respectively, to reduce the objective to

10^{-6}

of its initial value.

Figure 1: Empirical $\mathcal{O}(T^{-1/4})$ convergence and $\sqrt{d}$ improvement over AdamW on ResNet-32.

Figure 2: Training loss and orthogonalization error factor $(\hskip-1.00006pt1\hskip-0.89996pt-\hskip-0.50003pt\epsilon_{t\hskip-0.40974pt,\hskip-0.40974pt5}\hskip-1.00006pt)^{\hskip-0.81949pt-\hskip-0.81949pt1}$ on the 124M GPT-2-style model.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Sharp bounds for five-step Newton–Schulz maps.

Dimension (M)	Muon (Practical NS + WD)	Muon (Standard NS + WD)	AdamW
0.47	$(0.1,0.0625)$	$(0.1,0.0625)$	$(0.01,0.0625)$
1.86	$(0.05,0.1)$	$(0.05,0.1)$	$(0.01,0.15)$
7.41	$(0.02,0.3)$	$(0.05,0.1)$	$(0.001,0.04)$
66.46	$(0.05,0.15)$	$(0.0075,0.35)$	$(0.001,0.0625)$
118.11	$(0.01,0.6)$	$(0.1,0.0625)$	$(0.0003,0.1)$
184.50	$(0.02,0.4)$	$(0.01,0.35)$	$(0.0003,0.2)$

Appendix

Table 3: Selected stepsize and WD pairs for ResNet-32.

Figure 4: Frobenius gradient norm convergence rates for ResNet-32 models of varying dimensions. The gradient norms exhibit an overall $T^{-1/4}$ trend, with Muon generally converging faster than AdamW.

Algorithm \ Dimension (M)	0.47	1.86	7.41	66.46	118.11	184.50
Muon (practical + WD)	$59.64\pm 0.21$	$66.35\pm 0.38$	$70.22\pm 0.25$	$73.66\pm 0.18$	$72.74\pm 0.62$	$74.57\pm 0.04$
Muon (standard + WD)	$58.71\pm 0.36$	$65.59\pm 0.31$	$68.97\pm 0.38$	$72.28\pm 0.20$	$70.88\pm 0.40$	$72.92\pm 0.15$
AdamW	$45.95\pm 1.15$	$52.29\pm 1.88$	$57.03\pm 0.55$	$59.79\pm 1.61$	$62.69\pm 0.37$	$63.17\pm 0.62$

Appendix

Table 4: CIFAR-100 test top-1 accuracy (over three seeds) for ResNet-32 models of varying dimensions after 2,000 iterations. Muon consistently achieves higher accuracy than AdamW.

Figure 5: Loss curves for ResNet-32 models of varying dimensions. Muon consistently achieves lower training and test losses than AdamW.

Figure 6: Joint sweep over stepsize and WD for ResNet-32 models of varying dimensions.

Model	Muon (Practical NS + WD)	Muon (Standard NS + WD)	AdamW
124M GPT-2-style	$(0.02,0.000625)$	$(0.02,0.0025)$	$(0.001,0.01)$
355M GPT-2-style	$(0.02,0.02)$	$(0.02,0.0025)$	$(0.0003,0.00125)$
0.6B Qwen3-style	$(0.04,0.0025)$	$(0.02,0.0025)$	$(0.0015,0.000625)$

Appendix

Table 5: Selected stepsize and WD pairs for language model training.

Figure 7: Loss curves across the three language models. Muon consistently achieves lower training and validation losses than AdamW.

Figure 8: Momentum singular-value distributions for the query projection in the middle transformer block of the three language models. The distributions span a broad range and include very small normalized singular values.

Figure 9: Orthogonalization errors $\epsilon_{t,5}$ and the corresponding factors $(1-\epsilon_{t,5})^{-1}$ for practical and standard NS across the three language models. Practical NS consistently exhibits smaller errors and factors, while the large factor values need not imply slow convergence.

Figure 10: Joint sweep over stepsize and WD for language model training.

Explore similar work

Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany
Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum
Feb 19, 2026Minxin Zhang, Yuxuan Liu, Hayden SchaefferDeep Learning Optimization Adaptive Gradient Methods
Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuNewton-Schulz Iteration Gradient Descent
Pengcheng Laboratory, Shenzhen, China

May 11, 2026J/K move · Enter open · S save

Muown: Row-Norm Control for Muon Optimization

Kai Lion, Florian Hübler, Bingcong Li, Antonio Orvieto+1

Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Language Model Pretraining Muon Optimizer

Open paper AskPDF arXiv

Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the

\ell_\infty

geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.

Abstract

Explore similar work

Muown: Row-Norm Control for Muon Optimization

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum