The Row Normalization Puzzle in Muon

Jiayu Zhang; Tianyi Lin

cs.LGSep 30, 2026

The Row Normalization Puzzle in Muon

Authors: Jiayu Zhang, Tianyi Lin

Organizations: Department of Industrial Engineering and Operations Research Columbia University

Abstract

This paper examines how row-wise renormalization affects Muon, focusing on the gap between NorMuon's worst-case guarantees and its practical performance (Li et al.). Despite its growing adoption and promising performance in large language model (LLM) pretraining, NorMuon's worst-case guarantees remain poorly understood. One fundamental question is: Does row normalization yield provable convergence gains, potentially through its interaction with approximate polar computation and exponential moving-average momentum? Our results show that row normalization introduces a dimension-dependent factor in the worst-case iteration complexity under the operator-norm geometry, which persists even with exact polar computation and any fixed momentum parameters. Indeed, we establish an algorithm-dependent lower bound and a matching upper bound in deterministic settings, and extend our upper bound analysis to stochastic settings. Both upper-bound analyses allow approximate polar computation. Experiments show that NorMuon is slower than Muon on synthetic problems inspired by our worst-case construction, yet outperforms Muon in LLM pretraining. These findings sharpen the puzzle of why row normalization helps in practice and complement the recent findings of Dewulf et al.

Figures & tables

$m$	$8$	$32$	$128$	$512$	$2048$
Muon	$5.70_{\pm 0.03}$	$5.70_{\pm 0.02}$	$5.70_{\pm 0.02}$	$5.72_{\pm 0.03}$	$5.72_{\pm 0.03}$
NorMuon (Algorithm 1 )	$5.70_{\pm 0.03}$	$5.46_{\pm 0.02}$	$5.00_{\pm 0.02}$	$4.12_{\pm 0.01}$	$3.15_{\pm 0.01}$
NorMuon with Eq. ( 4.1 )	$5.70_{\pm 0.03}$	$5.46_{\pm 0.02}$	$5.00_{\pm 0.01}$	$4.13_{\pm 0.01}$	$3.15_{\pm 0.01}$

Table 1: Mean

\pm

twice the standard error of the loss reduction

(\log_{10}{F(X_{0})}-\log_{10}{\min_{0\leq t\leq T}F(X_{t})})

over 100 random seeds. A larger value means better performance.

	AdamW	Muon	NorMuon (column)	NorMuon (row)
Test loss	$0.4298_{\pm 0.0005}$	$\mathbf{0.3884}_{\pm 0.0004}$	$0.3889_{\pm 0.0005}$	$0.3900_{\pm 0.0004}$
Test accuracy	$92.33_{\pm 0.02}\%$	$93.79_{\pm 0.03}\%$	$\mathbf{93.87}_{\pm 0.02}\%$	$93.83_{\pm 0.02}\%$

Table 2: Mean

\pm

standard error of the test loss and accuracy over 50 seeds for CIFARNET.

		MMLU	HSwag	PIQA	Wino.	ARC-C	ARC-E	BoolQ	CSQA	SIQA	OBQA	Avg	Loss
$1.38$ B	AdamW	$31.9$	$54.9$	$74.6$	$54.2$	$45.1$	$72.3$	$57.9$	$56.6$	$46.2$	$48.2$	$54.2$	$2.3553$
	Muon	$\mathbf{33.3}$	$60.3$	$75.9$	$55.5$	$43.8$	$73.6$	$\mathbf{64.4}$	$\mathbf{60.0}$	$49.1$	$47.1$	$56.3$	$2.2914$
	NorMuon	$\mathbf{33.3}$	$\mathbf{60.9}$	$\mathbf{76.1}$	$\mathbf{58.4}$	$\mathbf{45.4}$	$\mathbf{75.0}$	$62.0$	$59.6$	$\mathbf{50.2}$	$\mathbf{49.8}$	$\mathbf{57.1}$	$\mathbf{2.2810}$
$6.44$ B	AdamW	$34.2$	$60.8$	$76.4$	$57.2$	$43.5$	$73.1$	$66.9$	$60.2$	$48.8$	$46.4$	$56.8$	$2.3748$
	Muon	$36.2$	$67.3$	$\mathbf{78.5}$	$\mathbf{61.4}$	$\mathbf{48.7}$	$\mathbf{79.6}$	$65.7$	$63.1$	$\mathbf{53.0}$	$50.4$	$60.4$	$2.2803$
	NorMuon	$\mathbf{36.4}$	$\mathbf{68.2}$	$78.0$	$60.5$	$47.9$	$76.5$	$\mathbf{72.8}$	$\mathbf{64.5}$	$52.5$	$\mathbf{51.6}$	$\mathbf{60.9}$	$\mathbf{2.2674}$

Table 3: LLM pretraining downstream accuracies and test losses.

Figure 1: Validation loss in LLM pretraining for the (a) $286$ M, (b) $1.38$ B, and (c) $6.44$ B models.

Figure 2: Ablation studies. (a) Final validation loss versus step size. (b) NorMuon applied to different groups of matrices. (c) Comparison with other Muon variants that use normalization.

NorMuon parameter groups	$286$ M	$1.38$ B	Other optimizers	$286$ M	$1.38$ B
Muon	$2.7311$	$2.3587$	Muon	$2.7311$	$2.3587$
NorMuon	$\mathbf{2.7248}$	$\mathbf{2.3442}$	NorMuon	$2.7248$	$2.3442$
QKVO only	$2.7319$	$2.3572$	Aurora	$\mathbf{2.7237}$	$\mathbf{2.3435}$
MLP input only	$2.7258$	$2.3448$	Muon+	$2.7265$	$2.3483$
MLP output only	$2.7260$	$2.3463$	MuonEq-R	$2.7305$	$2.3587$
NorMuon (row-wise MLP)	$2.7260$	$\mathbf{2.3442}$

Table 4: Test loss for parameter group and optimizer ablations.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$m$	Muon	NorMuon (Algorithm 1 )	NorMuon with Eq. ( 4.1 )
$8$	$(10^{-3},0,\text{--})$	$(10^{-3},0,0.95)$	$(10^{-3},0,0)$
$32$	$(10^{-3},0,\text{--})$	$(10^{-2.5},0.5,0.99)$	$(10^{-2.5},0.5,0.99)$
$128$	$(10^{-3},0,\text{--})$	$(10^{-2.5},0.5,0.9)$	$(10^{-2.5},0.5,0.9)$
$512$	$(10^{-3},0,\text{--})$	$(10^{-2},0.5,0.9)$	$(10^{-2},0.5,0.9)$
$2048$	$(10^{-3},0,\text{--})$	$(10^{-1.5},0.5,0.9)$	$(10^{-1.5},0.5,0.9)$

Appendix

Table 5: Selected hyperparameters

(\gamma,\beta_{1},\beta_{2})

for the synthetic experiment.

$m$	Muon	NorMuon (Algorithm 1 )	NorMuon with Eq. ( 4.1 )
$8$	$(10^{-1},0,\text{--})$	$(10^{-1},0,0.95)$	$(10^{-1},0,0)$
$32$	$(10^{-1},0,\text{--})$	$(10^{-0.5},0,0.9)$	$(10^{-0.5},0,0.9)$
$128$	$(10^{-1},0,\text{--})$	$(10^{-0.5},0,0.9)$	$(10^{-0.5},0,0.9)$
$512$	$(10^{-1},0,\text{--})$	$(10^{0},0,0.99)$	$(10^{0},0,0.99)$
$2048$	$(10^{-1},0,\text{--})$	$(10^{0},0,0)$	$(10^{0},0,0)$

Appendix

Table 6: Selected hyperparameters

(\gamma,\beta_{1},\beta_{2})

for the synthetic experiment with a cosine step-size schedule.

$m$	$8$	$32$	$128$	$512$	$2048$
Muon	$5.51_{\pm 0.05}$	$5.57_{\pm 0.04}$	$5.55_{\pm 0.05}$	$5.57_{\pm 0.04}$	$5.57_{\pm 0.05}$
NorMuon (Algorithm 1 )	$5.51_{\pm 0.05}$	$4.64_{\pm 0.04}$	$4.75_{\pm 0.05}$	$3.63_{\pm 0.05}$	$3.58_{\pm 0.05}$
NorMuon with Eq. ( 4.1 )	$5.51_{\pm 0.05}$	$4.64_{\pm 0.04}$	$4.75_{\pm 0.05}$	$3.63_{\pm 0.05}$	$3.58_{\pm 0.05}$

Appendix

Table 7: Mean

\pm

2\times

standard error of the loss reduction

\log_{10}F(X_{0})-\log_{10}\min_{0\leq t\leq T}F(X_{t})

over 100 evaluation seeds under a cosine schedule with

T=25

. A larger value means better performance.

Figure 4: Loss curves for the cosine schedule synthetic experiment with $T=25$ .

Model	Depth	Hidden dim.	MLP dim.	Vocab. size	Seq. length	Matrix params	Batch size (tokens)	Training tokens
$286$ M	$12$	$768$	$3072$	$32{,}768$	$2048$	$84.9$ M	$524{,}288$	$2.20$ B
$1.38$ B	$24$	$1536$	$6144$	$32{,}768$	$2048$	$679$ M	$1{,}048{,}576$	$14.6$ B
$6.44$ B	$32$	$2048$	$8192$	$131{,}072$	$4096$	$1.61$ B	$1{,}048{,}576$	$19.7$ B

Appendix

Table 8: Model configurations and training budgets for LLM pretraining.

Benchmark	Muon	NorMuon
MMLU	33.3 ±0.3	33.3 ±0.06
HellaSwag	60.3 ±0.2	60.9 ±0.4
PIQA	75.9 ±0.3	76.1 ±0.4
WinoGrande	55.5 ±0.4	58.4 ±1.5
ARC-C	43.8 ±0.8	45.4 ±0.3
ARC-E	73.6 ±0.3	75.0 ±0.8

Appendix

Table 9: Mean

\pm

standard deviation of LLM pretraining downstream accuracies and test losses.

Figure 5: Ablation studies on the $1.38$ B model trained on $7.66$ B tokens. (a) Final validation loss versus step size. (b) Muon and NorMuon with 3, 5, and 7 PolarExpress (PE) iterations. (c) NorMuon with different second-moment factors $\beta_{2}$ . (d) Muon and NorMuon with different weight decay (WD) factors.

Step size	$0.01$	$0.02$	$0.03$	$0.04$
Muon	$2.3587$	$2.3587$	$\mathbf{2.3571}$	$2.3684$
NorMuon	$\mathbf{2.3436}$	$2.3442$	$2.3533$	$2.3637$

Appendix

Table 10: Test loss versus base step size for the

1.38

B model trained on

7.66

B tokens.

PE steps	$3$	$5$	$7$
Muon	$2.3882$	$2.3587$	$2.3562$
NorMuon	$2.3620$	$2.3442$	$2.3440$

Appendix

Table 11: Test loss versus the number of PolarExpress (PE) steps.

NorMuon $\beta_{2}$	$0$	$0.9$	$0.95$	$0.99$	Muon
Test loss	$2.3440$	$2.3435$	$2.3442$	$2.3439$	$2.3587$

Appendix

Table 12: Test loss versus the second-moment factor

\beta_{2}

Weight decay	$0.01$	$0.02$	$0.05$
Muon	$2.3557$	$2.3587$	$2.3880$
NorMuon	$2.3424$	$2.3442$	$2.3685$

Appendix

Table 13: Test loss versus weight decay.

Figure 6: Relative differences between Muon and NorMuon in the alignment and in the norms of the momentum and update direction, for different parameter groups. Error bars show the 10th–90th percentile range across the matrices in each group.

Explore similar work

Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Muon Optimizer Design
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLarge Language Model Pretraining Muon
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Muon Batch Normalization
Tilde Research

May 11, 2026, cs.LGJ/K move · Enter open · S save

Muown: Row-Norm Control for Muon Optimization

Kai Lion, Florian Hübler, Bingcong Li, Antonio Orvieto+1

Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Muon Optimizer Design

Open paper AskPDF arXiv

Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the

\ell_\infty

geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.

Abstract

Explore similar work

Muown: Row-Norm Control for Muon Optimization

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Aurora: A Leverage-Aware Spectral Optimizer