Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning

Ziyue Liu; Ruijie Zhang; Zhengyang Wang; Yequan Zhao; Yupeng Su; Zi Yang; Zheng Zhang

cs.LGApr 11, 2026

Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning

Authors: Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang

Organizations: University of California at Santa Barbara · University at Albany, SUNY

Abstract

Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, the orthogonalization quality of Muon hinges on the number of Newton--Schulz (NS) iterations performed, which poses efficiency challenges due to its non-trivial computation and communication cost. We propose Muon $^2$ , an extension of Muon, to improve both quality and efficiency by applying Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon $^2$ , leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon $^2$ demonstrates dramatic improvement over Muon at each polar step. Across GPT, LLaMA, and Mixture-of-Experts pre-training experiments up to 13B parameters, Muon $^2$ (and its memory-efficient variant Muon $^2$ -F that preserves most of its benefits) consistently outperforms Muon and its variants while reducing NS iterations by 40%, and saves up to 1/4 training time over Muon when achieving the same loss.

Figures & tables

Panel of Figure 1 — Figure 1: Spectral effect of Muon 2 on the input matrix of the Newton–Schultz iteration.

Panel of Figure 2 — Figure 2: Spectral effect of Muon 2 on the Newton–Schultz (NS) output (i.e., ${\bf f_{N_{s}}}(\sigma)$ ) at each step (i.e., $N_{s}$ ).

Figure 3: Cosine similarity of the output matrix vs. true orthogonalized update for Muon , Muon 2 and Muon 2 -F at different Newton-Schultz steps.

Model	NS Steps	Muon	Muon 2	Muon 2 -F
GPT-Small	$N_{s}=3$	32.70	28.12 (-4.58)	28.30 (-4.40)
GPT-Small	$N_{s}=5$	29.51	27.95 (-1.56)	27.93 (-1.58)
GPT-Base	$N_{s}=3$	24.69	20.39 (-4.30)	21.17 (-3.52)
GPT-Base	$N_{s}=5$	21.47	19.96 (-1.51)	20.58 (-0.89)
GPT-Large	$N_{s}=3$	21.13	16.99 (-4.14)	17.69 (-3.44)
GPT-Large	$N_{s}=5$	17.56	16.52 (-1.04)	16.55 (-1.01)

Table 1: Validation perplexity (

\downarrow

) of Muon and Muon 2 on GPT models across Newton–Schulz iterations.

Figure 4: Learning rate sweep on GPT-Large comparing Muon 2 and Muon across Newton–Schulz iterations.

Model	NS Steps	Muon	Muon 2	Muon 2 -F
LLaMA-60M	$N_{s}=3$	26.37	24.59 (-1.78)	24.68 (-1.69)
LLaMA-60M	$N_{s}=5$	24.98	24.60 (-0.38)	24.66 (-0.32)
LLaMA-350M	$N_{s}=3$	14.91	13.44 (-1.47)	13.55 (-1.36)
LLaMA-350M	$N_{s}=5$	14.03	13.46 (-0.57)	13.44 (-0.59)
LLaMA-1B	$N_{s}=3$	11.63	10.42 (-1.21)	10.49 (-1.14)
LLaMA-1B	$N_{s}=5$	10.62	10.21 (-0.41)	10.21 (-0.41)

Table 2: Validation perplexity (

\downarrow

) of Muon and Muon 2 on LLaMA models across Newton–Schulz iterations.

Model	NS Steps	Muon	Muon 2
LLaMA-13B	$N_{s}=3$	10.54	9.11 (-1.43)
LLaMA-13B	$N_{s}=5$	9.24	8.71 (-0.53)
MoE-7B-A1B	$N_{s}=3$	11.93	9.85 (-2.08)
MoE-7B-A1B	$N_{s}=5$	10.61	9.66 (-0.95)

Table 3: Validation perplexity (

\downarrow

) of Muon and Muon 2 on LLaMA-13B and MoE models across NS iterations.

	NS Steps	Avg	ARC-c	ARC-e	OBQA	HellaS.	PIQA	WinoG.	LAMBADA	MMLU
Muon	$N_{s}=3$	40.94	24.23	42.80	32.00	44.37	71.22	53.43	36.52	22.95
Muon	$N_{s}=5$	43.43	26.79	46.09	33.20	49.75	72.25	55.41	40.99	22.96
Muon 2	$N_{s}=3$	43.61	26.37	47.81	31.80	50.23	72.20	55.01	42.50	22.97
Muon 2	$N_{s}=5$	45.24	27.13	49.03	35.20	52.61	74.05	55.88	45.08	22.95

Table 4: Zero-shot evaluation on LLaMA-1B trained by Muon and Muon 2 across different NS iterations.

	GPT-Small		GPT-Base
NS Steps	$N_{s}=3$	$N_{s}=5$	$N_{s}=3$	$N_{s}=5$
Muon 2	28.12	27.95	20.39	19.96
Muon Jordan et al. (2024)	32.70 (+4.58)	29.51 (+1.56)	24.69 (+4.30)	21.47 (+1.51)
PolarExpress Amsel et al. (2025)	30.01 (+1.89)	29.42 (+1.47)	22.74 (+2.35)	21.16 (+1.20)
Turbo-Muon Boissin et al. (2025)	29.70 (+1.58)	29.66 (+1.71)	23.46 (+3.07)	21.93 (+1.97)
NorMuon Li et al. (2025)	30.35 (+2.23)	28.40 (+0.45)	23.33 (+2.94)	21.27 (+1.31)

Table 5: Validation perplexity (

\downarrow

) of Muon 2 comparing against Muon variants on GPT-Small and Base.

Figure 5: On LLaMA-1B, Muon 2 achieves same loss with 25% fewer training steps than Muon .

Model	Memory (GB)
Model	AdamW	Muon	Muon 2	Muon 2 -F
LLaMA-1B	20.39	18.56	20.36	18.57
LLaMA-7B	59.31	47.29	60.66	47.53

Table 6: Memory comparison across Adam, Muon , Muon 2 and Muon 2 -F .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: How the NS iteration maps singular values using different coefficients.

Panel of Figure 7 — Figure 7: The comparison of convergence zones between Keller ( Muon ’s Jordan et al. (2024) polar method) and PolarExpress Amsel et al. (2025) .

Model	$n_{\text{embd}}$	$n_{\text{layer}}$	$n_{\text{head}}$	Param(M)
GPT-Small	768	12	12	124
GPT-Base	1024	24	16	362
GPT-Large	1280	36	20	774

Appendix

Table 7: Architecture configurations of GPT models.

LR	Baseline		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.003	34.90	31.18	31.00	30.04
0.005	32.70	29.51	28.83	28.41
0.010	33.49	29.85	28.36	28.01
0.020	33.36	29.86	28.12	27.95
0.040	33.40	30.13	29.30	28.99

Appendix

Table 8: Learning rate sweep on GPT-Small . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 8: Learning Rates Sweep on GPT-Small

LR	Baseline		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.003	25.43	22.11	21.73	21.01
0.005	24.69	21.75	20.64	20.33
0.010	25.42	21.99	20.39	19.96
0.020	24.78	21.47	20.50	20.40
0.040	25.75	22.22	21.97	21.24

Appendix

Table 9: Learning rate sweep on GPT-Base . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 9: Learning Rates Sweep on GPT-Base

LR	Baseline		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.003	21.88	18.44	18.14	17.38
0.005	21.13	18.32	17.28	16.82
0.010	21.50	18.32	16.99	16.60
0.020	21.57	17.57	17.28	16.52
0.040	23.77	18.80	18.41	17.55

Appendix

Table 10: Learning rate sweep on GPT-Large . The best validation perplexity (

\downarrow

) of each method is bolded.

Model	$n_{\text{embd}}$	$n_{\text{inter}}$	$n_{\text{layer}}$	$n_{\text{head}}$	Param(M)
LLaMA-60M	512	1344	8	8	58
LLaMA-350M	1024	2736	24	16	368
LLaMA-1B	2048	5472	24	32	1280
LLaMA-13B	5120	13827	40	40	13018

Appendix

Table 11: Architecture configurations of LLaMA models.

LR	Muon		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.02	26.44	25.91	25.90	25.87
0.04	26.37	24.91	24.77	24.68
0.05	27.06	24.94	24.78	24.68
0.06	27.26	24.98	24.59	24.60
0.08	27.35	25.15	24.71	24.71

Appendix

Table 12: Learning rate sweep on LLaMA-60M . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 10: Learning Rates Sweep on LLaMA-60M

LR	Muon		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.04	14.91	14.03	13.44	13.46
0.05	15.18	14.12	13.50	13.51
0.06	15.33	14.18	13.58	13.59
0.07	15.43	14.30	13.67	13.63
0.08	15.76	14.49	13.71	13.73

Appendix

Table 13: Learning rate sweep on LLaMA-350M . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 11: Learning Rates Sweep on LLaMA-350M

LR	Muon		Muon 2 (ours)
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.04	11.63	10.62	10.43	10.21
0.05	11.86	10.70	10.50	10.26
0.06	11.98	10.77	10.55	10.31

Appendix

Table 14: Learning rate sweep on LLaMA-1B . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 12: Learning Rates Sweep on LLaMA-1B

Model	$n_{\text{embd}}$	$n_{\text{inter}}$	$n_{\text{layer}}$	$n_{\text{head}}$	$n_{\text{exp}}$	$n_{\text{act}}$	Param(M)
MoE-7B-A1B	2048	4096	24	32	16	2	6980

Appendix

Table 15: Architecture configurations of Mixture-of-Experts model.

	NS Steps	Avg	ARC-c	ARC-e	OBQA	HellaS.	PIQA	WinoG.	LAMBADA	MMLU
Muon	$N_{s}=3$	39.42	24.32	44.23	30.80	39.92	68.82	52.09	32.19	22.95
Muon	$N_{s}=5$	40.34	24.91	44.07	30.40	42.50	69.80	51.07	37.07	22.92
Muon 2	$N_{s}=3$	40.86	25.77	44.91	29.40	43.72	69.70	50.51	40.03	22.85
Muon 2	$N_{s}=5$	41.57	24.49	46.93	31.60	45.01	69.80	51.62	40.07	23.02

Appendix

Table 16: Zero-shot evaluation on GPT-Large trained by Muon and Muon 2 across different NS iterations.

LR	PolarExpress		Turbo-Muon		NorMuon
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.003	31.51	30.45	31.36	30.31	32.50	29.90
0.005	30.01	29.42	29.74	29.66	30.70	28.40
0.010	31.32	29.73	29.70	29.79	31.43	28.72
0.020	30.31	29.66	30.10	29.88	31.05	28.48
0.040	31.25	30.03	30.07	30.82	30.35	29.09

Appendix

Table 17: Learning rate sweep on GPT-Small . The best validation perplexity (

\downarrow

) of each method is bolded.

LR	PolarExpress		Turbo-Muon		NorMuon
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.003	23.10	21.80	24.48	22.79	24.70	21.89
0.005	22.75	21.39	23.95	22.51	23.90	21.54
0.010	23.38	21.44	24.14	22.42	24.18	21.42
0.020	22.74	21.16	23.46	21.93	23.33	21.27
0.040	23.70	22.43	26.23	22.72	24.25	21.64

Appendix

Table 18: Learning rate sweep on GPT-Base . The best validation perplexity (

\downarrow

) of each method is bolded.

LR	GPT-Small		GPT-Base
LR	$N_{s}{=}3$	$N_{s}{=}5$	$N_{s}{=}3$	$N_{s}{=}5$
0.001	36.63	31.49	26.07	22.73
0.003	31.20	29.30	27.91	22.94
0.005	33.45	30.48	26.08	22.42

Appendix

Table 19: Learning rate sweep of AdaMuon on GPT-Small and GPT-Base . The best validation perplexity (

\downarrow

) of each method is bolded.

Figure 13: On LLaMA-1B, Muon 2 achieves same loss with 1/4 fewer GPU-hours than Muon .

Model	NS Steps	Step Time (ms)
Muon	$N_{s}=3$	2958
Muon	$N_{s}=5$	3053
Muon 2	$N_{s}=3$	2971
Muon 2	$N_{s}=5$	3085

Appendix

Table 20: Per-training-step time for LLaMA-1B on a supercomputer with 64 nodes (4 GPUs each).

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model Pretraining Newton-Schulz Iteration
The Chinese University of Hong Kong · Shopee · MBZUAI +3
Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz Iteration Orthogonal Optimization
IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2
MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
Feb 25, 2026Ruijie Zhang, Yequan Zhao, Ziyue Liu +4Muon Optimizer Deep Learning Optimization
University of California at Santa Barbara

May 18, 2026J/K move · Enter open · S save

AMO: Operator-level Adaptive Muon Orthogonalization

Xinlin Zhuang, Panyi Ouyang, Yichen Li, Jiangming Shi+6

The Chinese University of Hong Kong · Shopee · MBZUAI+3

Language Model Pretraining Newton-Schulz Iteration Muon Optimizer Second-Order Optimization Adaptive Gradient Methods

Open paper AskPDF arXiv

Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton-Schulz (NS) iteration as its core operation. Standard Muon applies a uniform NS schedule to all parameter matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-matrix heterogeneity is pervasive and strongly associated with matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.

Abstract

Neural Network Optimization

Explore similar work

AMO: Operator-level Adaptive Muon Orthogonalization

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training