MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

Ruijie Zhang; Yequan Zhao; Ziyue Liu; Zhengyang Wang; Yupeng Su; Liyan Tan; Zheng Zhang

cs.LGFeb 25, 2026

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

Authors: Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Yupeng Su, Liyan Tan, Zheng Zhang

Organizations: University of California at Santa Barbara

Abstract

Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition is that polar iterations, by flattening the singular spectrum to all ones, should also eliminate column- and row-wise norm imbalance in the update. We show that this is not true in practice: practical polar steps can substantially amplify the imbalance. We term this the post-polar imbalanced update problem, and prove that such imbalance tightens the second-order term in a blockwise descent analysis, weakening Muon's per-step descent guarantee. Motivated by this analysis, we propose Muon+, a one-line fix that inserts a single normalization step after polar orthogonalization. Muon+ adds no optimizer state. Across pre-training experiments on GPT and LLaMA models from 60M to 7B parameters, spanning both compute-optimal budgets and extended token-to-parameter ratios up to approximately 200, Muon+ consistently outperforms Muon in terms of training and validation perplexity, leading to significant overall pre-training speedup.

Figures & tables

Figure 1: Pre-training GPT and LLaMA models at scales ranging from 130M to 1B parameters under compute-optimal settings. Quantitative results are provided in Section 3 . Muon+ consistently outperforms Muon across all runs. We also conduct larger-scale and overtraining experiments for both GPT and LLaMA; the results are presented in Section 3 .

Panel of Figure 2 — Figure 2: Comparison of variance behavior under different settings during the realistic training of a 60M LLaMA model on 1.1B FineWeb tokens.

Figure 3: Layer-6 Q Projection Update matrix of Muon. The update matrix is highly unbalanced. The trends over iteration/training are shown in Figure 2 .

Figure 4: Regions of imbalance amplification and reduction. When $\xi_{ij}$ lies in the red (blue) region, the imbalance is amplified (reduced). We also provide figures for other commonly used coefficients in Appendix D .

Panel of Figure 5 — Figure 5: Stable step-size comparison between Muon and Muon+ . Muon+ remains stable under larger learning rates, while Muon exhibits unstable training and degraded validation loss.

Model	Param (M)	Train Tokens (B)	Muon	Muon+
GPT-Small	124	3.0	29.66	27.64 (-2.02)
GPT-Base	362	7.2	21.70	19.98 (-1.72)
GPT-Large	774	15.5	17.82	16.91 (-0.91)
GPT-Huge (bf16)	6,654	20.0	15.90	14.69 (-1.21)

Table 1: Validation PPL of Muon vs. Muon+ on GPT models.

Model	Param (M)	Train Tokens (B)	AdamW	Muon	Muon+
LLaMA-60M	58	1.1	33.10	25.75	25.25 (-0.50)
LLaMA-130M	134	2.2	23.64	19.06	18.65 (-0.41)
LLaMA-350M	368	6.4	16.18	14.02	13.41 (-0.61)
LLaMA-1B	1339	13.1	14.38	10.68	10.31 (-0.37)
LLaMA-7B (bf16)	6,738	19.8	–	10.54	10.12 (-0.42)

Table 2: Validation PPL of Muon vs. Muon+ on LLaMA models.

Model	Param (M)	Train Tokens (B)	Muon	Muon+
GPT-Base	362	72	16.97	15.84 (-1.13)
LLaMA-350M	368	72	11.48	11.03 (-0.45)

Table 3: Overtraining GPT-Base/LLaMA-350M. Models are trained on 72 billion FineWeb tokens.

	Ave.	OBQA	HellaSwag	ARC-E	WSC	Winogrande	BoolQ	PIQA
Muon	0.481	0.306	0.446	0.444	0.375	0.522	0.574	0.700
Muon+	0.494	0.320	0.480	0.471	0.365	0.534	0.577	0.712

Table 4: Downstream evaluation results on GPT-Base overtraining checkpoint.

Figure 6: Training loss curves under overtraining for GPT-Base.

Model	Target loss	Muon steps	Muon+ steps	Speed-up
LLaMA-130M	3.060	4228	3448	$\uparrow$ 22.6%
LLaMA-350M	2.788	3064	2374	$\uparrow$ 29.1%
GPT-Base	3.144	3447	2515	$\uparrow$ 37.1%
GPT-Large	2.927	3710	3032	$\uparrow$ 22.4%

Table 5: Training cost required to reach a target loss.

Model	None	Col	Row	Col-Row	Row-Col
LLaMA-60M	25.75	25.34	25.29	25.25	25.25
LLaMA-130M	19.34	19.16	18.98	18.65	18.68
LLaMA-350M	14.02	13.73	13.46	13.41	13.44

Table 6: Best validation perplexity under different normalization directions. For each model, we report the best result across learning rates. Lower is better.

Table 13

Method	GPT-Small	GPT-Base
Muon+	27.64	19.98
Turbo-Muon	29.69	21.91
NorMuon	28.44	21.31
AdaMuon	29.27	22.38

Table 9: Validation PPL (

\downarrow

) across different Muon methods.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Spearman rank correlation between pre-polar and post-polar (using 5 NS iterations) row norms over training (mean $\pm$ standard deviation across layers). All correlations remain positive throughout training, indicating that rows that are large before the polar step tend to remain large afterward.

Figure 8: Regions of imbalance amplification and reduction for Jordan Coefficients.

Figure 9: Regions of imbalance amplification and reduction for You Coefficients.

Figure 10: Regions of imbalance amplification and reduction for PolarExpress Coefficients.

Model	$n_{\text{embd}}$	$n_{\text{layer}}$	$n_{\text{head}}$	Param(M)
GPT-Small	768	12	12	124
GPT-Base	1024	24	16	362
GPT-Large	1280	36	20	774
GPT-Huge	4096	32	32	6654

Appendix

Table 10: Architecture configurations of GPT models.

Model	$n_{\text{embd}}$	$n_{\text{layer}}$	$n_{\text{head}}$	FFN dim	Param(M)
60M	512	8	8	1376	58
130M	768	12	12	2048	134
350M	1024	24	16	2736	368
1B	2048	24	32	5461	1339
7B	4096	32	32	11008	6738

Appendix

Table 11: Architecture configurations of LLaMA-style models.

Model	Hyperparameter	Muon	Muon+
GPT-Small	Sequence length	2048	2048
	Learning rate	$0.005$	$0.01$
	Weight decay	$0.1$	$0.1$
	Norm d	$-$	Col-Row
GPT-Base	Sequence length	4096	4096
	Learning rate	$0.005$	$0.005$

Appendix

Table 12: Best training hyperparameters for GPT-Small/Base/Large/Huge on FineWeb. Sequence lengths are set to 2048/4096/8192 for Small/Base/Large, respectively. We use Jordan orthogonalization for all runs. We keep the same learning rate scheduler as in NanoGPT: a constant learning rate for the first 40% of training steps followed by a linear decay to zero. Sweeping results are provided in Appendix F .

Model	Hyperparameter	Muon	Muon+
GPT-Base	Learning rate	$0.005$	$0.005$
	Weight decay	$0.1$	$0.1$
	Norm d	$-$	Row-Col

Appendix

Table 13: Training hyperparameters for overtraining GPT-Base. We train 72 billion FineWeb tokens for each setting. Sequence length is 4096 for both runs.

Model	Hyperparameter	Muon	Muon+
LLaMA-60M	Sequence length	1024	1024
	Learning rate	$0.06$	$0.06$
	LR scheduler	Cosine	Cosine
	Weight decay	$0.1$	$0.1$
	Warmup ratio	$0.1$	$0.1$
	Norm d	$-$	Row-Col

Appendix

Table 14: Best training hyperparameters for LLaMA on FineWeb. Sequence lengths are set to 1024 for 60M/130M and 4096 for 350M/1B. All runs use Jordan orthogonalization.

\mathrm{Norm}_{(\mathrm{col\_row})}

and

\mathrm{Norm}_{(\mathrm{row\_col})}

yield nearly identical performance. Additional sweeping results are provided in Appendix F .

Model	Hyperparameter	Muon	Muon+
LLaMA-350M	Learning rate	$0.04$	$0.04$
	Weight decay	$0.1$	$0.1$
	Norm d	$-$	Col-Row

Appendix

Table 15: Training hyperparameters for overtraining LLaMA-350M. We train 72 billion FineWeb tokens for each setting. Sequence length is 4096 for both runs.

LLaMA-60M
LR	None	Col	Row	Col-Row	Row-Col
0.005	28.98	28.17	28.14	28.10	28.03
0.01	27.03	26.59	26.42	26.46	26.41
0.02	26.22	25.83	25.69	25.65	25.68
0.04	25.90	25.49	25.30	25.28	25.28
0.06	25.75	25.34	25.29	25.25	25.25

Appendix

Table 16: Best validation perplexity per norm setting and learning rate for all LLaMA models. Bold marks the best entry in each row. None denotes the Muon baseline.

GPT-Small
LR	None	Col	Row	Col-Row	Row-Col
0.003	31.10	29.75	28.37	28.39	29.01
0.005	29.66	28.81	27.97	27.91	28.09
0.01	29.87	28.08	27.69	27.64	27.76
0.02	30.13	29.03	29.27	28.56	27.72
0.04	30.14	29.71	30.25	30.58	29.37

Appendix

Table 17: Best validation perplexity per norm setting and learning rate for GPT Models. Bold marks the best entry in each row. None is the Muon baseline.

Figure 11: Visualization of sweep result for LLaMA-350M

Model	Muon	MUON+
GPT-Base	21.68 $\pm$ 0.050	19.98 $\pm$ 0.031
LLaMA-350M	14.06 $\pm$ 0.046	13.42 $\pm$ 0.020

Appendix

Table 18: Mean validation perplexity over 5 random seeds. We report mean

\pm

standard deviation for GPT-Base and LLaMA-350M. Lower is better. Hyperparameters are consistent with Table 17 and 16 .

Model	Muon	MUON+
GPT-Base	21.70	20.16
LLaMA-350M	14.02	13.55

Appendix

Table 19: Matching Frobenius norm. Hyperparameters are consistent with Table 17 and 16 .

Method	Time / step	Ratio
Muon	892.8	1.000
Muon+ (col)	896.6	1.004
Muon+ (row)	900.6	1.008
Muon+ (col_row)	909.9	1.019
Muon+ (row_col)	910.3	1.019

Appendix

Table 20: Per-step runtime comparison between Muon and MUON+.

Panel of Figure 12 — Figure 12: Detailed Imbalance Trend

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

Sep 28, 2026

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called matrix-equilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.

Chang-Wei Shi, Xu Wang, Wu-Jun Li

National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Language Model Pretraining Efficient Language Model Training LLM Training

Ask PDF

Mar 30, 2026

MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions typically either rescale updates after orthogonalization or use heavier whitening-based preconditioners before it. We introduce {\method}, a lightweight family of pre-orthogonalization equilibration schemes for Muon with three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C). By rebalancing the momentum matrix before finite-step Newton--Schulz orthogonalization, {\method} improves the geometry seen by orthogonalization. We show that finite-step orthogonalization is governed by the input spectrum, especially stable rank and condition number, and that row/column normalization acts as a zeroth-order surrogate for whitening. For hidden matrix weights, R is the default variant. Theoretically, {\method} (R) retains the standard

\widetilde{\mathcal O}(T^{-1/4})

Muon-type nonconvex stationarity guarantee with decoupled weight decay and a horizon-free diminishing learning-rate schedule, and extends it to finite-step NS5 up to an explicit inexactness constant. In LLaMA2 pretraining on C4, {\method} (R) consistently outperforms Muon on 130M, 350M, and 1B models, with faster convergence and lower validation perplexity. The code is available at the MuonEq codebase.

Da Chang, Qiankun Shi, Lvgang Zhang +5

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences +4

Neural Network Optimization Muon Optimizer Orthogonal Optimization

Ask PDF

Apr 11, 2026

Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning

Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, the orthogonalization quality of Muon hinges on the number of Newton--Schulz (NS) iterations performed, which poses efficiency challenges due to its non-trivial computation and communication cost. We propose Muon

^2

, an extension of Muon, to improve both quality and efficiency by applying Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon

^2

, leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon

^2

demonstrates dramatic improvement over Muon at each polar step. Across GPT, LLaMA, and Mixture-of-Experts pre-training experiments up to 13B parameters, Muon

^2

(and its memory-efficient variant Muon

^2

-F that preserves most of its benefits) consistently outperforms Muon and its variants while reducing NS iterations by 40%, and saves up to 1/4 training time over Muon when achieving the same loss.

Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4

University of California at Santa Barbara · University at Albany, SUNY

Neural Network Optimization Muon Optimizer Adaptive Gradient Methods

Ask PDF

Abstract

Muon Optimizer

Explore similar work

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

Muon2^22: Boosting Muon via Adaptive Second-Moment Preconditioning

Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning