HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

Tianyu Pang; Yujie Fang; Zihang Liu; Shenyang Deng; Lei Hsiung; Shuhua Yu; Yaoqing Yang

cs.LGMar 10, 2026

HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

Authors: Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, Yaoqing Yang

Organizations: Dartmouth College · Microsoft · International Computer Science Institute · University of California, Berkeley ·

Abstract

Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by the Heavy-Tailed Self-Regularization (HT-SR) theory, we propose HTMuon. HTMuon preserves Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tailed weight spectra. Experiments on LLM pretraining and image classification show that HTMuon consistently improves performance over state-of-the-art baselines and can also serve as a plug-in on top of existing Muon variants. For example, on LLaMA pretraining on the C4 dataset, HTMuon reduces perplexity by up to $0.98$ compared to Muon. We further theoretically show that HTMuon corresponds to steepest descent under the Schatten- $q$ norm constraint and provide convergence analysis in smooth non-convex settings. The implementation of HTMuon is available at https://github.com/TDCSZ327/HTmuon.

Figures & tables

Panel of Figure 1 — Figure 1: Muon_NS vs. Muon_SVD on C4 dataset. (a) Validation perplexity for LLaMA-60M/135M trained: Muon_NS consistently achieves lower perplexity than Muon_SVD . Both Learning rates for 60M is 0.03 and for 135M is 0.02. (b)(c) Spectra of update matrices at steps $1/9000/19000$ shown in different colors: Muon_SVD enforces an exactly ”all-ones” spectrum in the update matrices; Muon_NS stays close to one but retains noticeable deviations, implicitly down-weighting noise-dominated singular-vector directions and correlating with improved performance.

Panel of Figure 2 — Figure 2: (a)Average PL $\bar{\alpha}$ of weight ESDs for LLaMA-60M and LLaMA-135M trained on C4 with Muon and COSMOS . Muon yields a higher mean $\bar{\alpha}$ , indicating less heavy-tailed spectra than COSMOS ; (b) COSMOS outperforms Muon for LLaMA-60M and LLaMA-135M models on C4 datatset.

	LLaMa-60M	LLaMa-135M	LLaMa-350M
Adam	32.21	23.01	17.11
AdamW	31.85	23.33	16.96
Muon	28.80	22.23	16.81
HTMuon	27.88	21.25	16.79

Table 1: Comparison with dominant Pretraining Optimizers on LLaMA Models of Varying Sizes on the C4 dataset. Lower perplexity indicates better performance. Detailed hyperparameters are provides in Table 18 , 19 and 20 in Appendix C .

	AdamW	Muon	HTMuon
GPT-2 small	25.19	22.46	22.20

Table 2: Comparison with dominant Pretraining Optimizers on GPT-2 small on the OpenWebText dataset. Lower perplexity indicates better performance. Hyperparameter settings are provided in Appendix C .

Panel of Figure 3 — Figure 3: Comparison with Muon variant optimizers on LLaMA-60M and 135M on C4 dataset. All optimizers are carefully tuned via grid search; detailed results and hyperparameter settings are provided in Table 7 and 19 in Appendix B and C .

Panel of Figure 4 — Figure 4: Comparison with state-of-the-art pretraining optimizers on LLaMA-60M and 135M on C4 dataset. All optimizers are carefully tuned via grid search; detailed results and hyperparameter settings are provided in Table 7 and 19 in Appendix B and C .

	CIFAR-100		CIFAR-10
Optimizer	ResNet18	ResNet50	ResNet18	ResNet50
SGDM	77.82	78.57	95.15	95.16
Muon	77.95	79.85	95.39	95.97
NorMuon	77.82	79.78	95.57	96.03
HTMuon	78.24	80.16	95.63	96.13
HTMuon + NorMuon	78.51	80.22	95.40	96.04

Table 3: Results on CIFAR-100 and CIFAR-10 dataset. Higher accuracy indicates better performance.

	Adam	Muon	HTMuon
ViT-Tiny	67.67	71.02	71.16

Table 4: Results on ImageNet-1K. Hyperparameter settings are provided in Appendix C .

	Adam	AdamW	Muon	HTMuon
LLaMA-1B	15.22	15.11	14.33	14.17

Table 5: We train LLaMA-1B with interval

=5

on C4. Hyperparameter settings are provided in Table 20 in Appendix C .

Panel of Figure 5 — Figure 5: we evaluate applying HTMuon and HTMuon_NS on LLaMA-60M and LLaMA-135M every 1, 5, 10, and 25 steps. We report the average per-step runtime overhead for all methods. Detailed results and hyperparameter settings are provided in Table 8 and 9 in Appendix B .

Panel of Figure 6 — Figure 6: Layer-wise PL $\alpha$ for LLaMA and ResNet model weights trained with Muon and HTMuon . All models used for visualization are trained using each optimizer’s best-performing hyperparameter configuration. For hyperparameter configurations, please refer to Appendix C .

Panel of Figure 7 — Figure 7: Layer-wise spectral norm and frobenius norm for LLaMA model weights trained with Muon and HTMuon . All models used for visualization are trained using each optimizer’s best-performing hyperparameter configuration. For hyperparameter configurations, please refer to Appendix C .

Optimizer	ARC-c	ARC-e	PIQA	Hellaswag	OBQA	Winogrande	BOOLQ	Avg.
Adam	21.93	31.27	65.18	27.49	17.40	53.83	62.17	39.90
AdamW	21.16	32.37	64.25	27.29	17.80	52.49	62.23	39.66
Muon	20.56	32.66	65.13	31.53	17.00	51.22	62.14	40.03
HTMuon	22.18	33.16	66.59	33.76	17.20	52.72	61.93	41.08

Table 6: Zero-shot evaluation results (

\uparrow

) on seven commonsense reasoning benchmarks for the LLaMA-1B model pretrained with different methods.

Panel of Figure 8 — Figure 8: We conduct grid search on $p$ . We find $p=0.125$ is a strong choice. Note that $p=0$ reduces to Muon . Detailed results provided in Table 14 in Appendix C .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Training loss curves for LLaMA-60M and LLaMA-135M. Learning rate for both models and optimizers is 0.03. Both curves are smoothed via a simple moving average (uniform weights, window = 50).

	Muon	HTMuon	NorMuon	HTMuon+NorMuon	AdaMuon	MARS	SOAP	Cautious	COSMOS	GaLore	Sophia
LLaMa-60M	28.80	27.88	28.17	27.55	28.67	29.13	28.96	29.71	28.62	34.26	34.02
LLaMa-135M	22.23	21.25	21.99	21.11	22.43	22.42	22.43	23.15	22.15	25.11	25.63

Appendix

Table 7: We evaluate HTMuon on LLaMA-60M and LLaMA-135M, comparing it against several Muon variant optimizers and other state-of-the-art optimizers. Red indicates the best value, and blue denotes the second-best value. We find that HTMuon consistently outperforms these baselines. More hyperparameter details are provided in Appendix C .

Panel of Figure 10 — Figure 10: we evaluate applying HTMuon $+$ NorMuon on LLaMA-60M and LLaMA-135M every 1, 5, 10, and 25 steps (while other steps applying NorMuon ) . We report the average per-step runtime overhead for all methods. Detailed results and hyperparameter settings are provided in Table 8 and 9 in Appendix C .

	LLaMA-60M				LLaMA-135M
Optimizer	1	5	10	25	1	5	10	25
Muon		28.80				22.23
HTMuon	27.88	28.36	28.58	28.64	21.25	22.01	22.13	22.20
HTMuon_NS	28.03	28.42	28.67	28.74	21.35	22.05	22.14	22.20
NorMuon		28.17				21.99
HTMuon $+$ NorMuon	27.55	27.85	28.01	28.12	21.11	21.61	21.81	21.95

Appendix

Table 8: We report the exact PPL obtained when training LLaMA-60M and LLaMA-135M using HTMuon , HTMuon_NS , and HTMuon+NorMuon under different update intervals. For Muon and NorMuon , interval updates are not applicable; therefore, we only report their PPL after training.

	LLaMA-60M				LLaMA-135M
Optimizer	1	5	10	25	1	5	10	25
Muon		0.51				0.59
HTMuon	1.26	0.67	0.61	0.54	1.70	0.82	0.70	0.63
HTMuon_NS	0.66	0.59	0.56	0.54	0.92	0.68	0.64	0.63
NorMuon		0.54				0.60
HTMuon $+$ NorMuon	1.27	0.70	0.64	0.59	1.78	0.86	0.81	0.72

Appendix

Table 9: We report the average per-step runtime overhead obtained when training LLaMA-60M and LLaMA-135M using HTMuon , HTMuon_NS , and HTMuon+NorMuon under different update intervals. We measure per-step runtime on 4 NVIDIA A6000 GPUs. For Muon and NorMuon , interval updates are not applicable; therefore, we only report average per-step runtime overhead.

	LLaMA-60M		LLaMA-135M
Optimizer	PPL	Time	PPL	Time(hour)
Muon	28.84	0.67	22.27	1.40
HTMuon_NS (Interval=5)	28.50	0.71	22.02	1.55
HTMuon (Interval=5)	28.36	1.02	22.04	2.40
HTMuon_NS	28.06	0.87	21.36	2.17
HTMuon	27.87	2.46	21.24	6.57

Appendix

Table 10: We report the perplexity and wall-clock time of when training LLaMA-60M and LLaMA-135M using Muon , HTMuon , HTMuon_NS , HTMuon (Interval=5), HTMuon_NS (Interval=5). We measure wall-clock time on 4 NVIDIA RTX PRO 6000 GPUs.

Panel of Figure 11 — Figure 11: (a)(b): Layer-wise PL $\alpha$ for ResNet-18 model weight ESDs on CIFAR-100 and CIFAR-10 trained with Muon and HTMuon . (c)(d): Layer-wise PL $\alpha$ for LLaMA model weight ESDs trained with Muon and HTMuon_HT . All models used for visualization are trained using each optimizer’s best-performing hyperparameter configuration. For hyperparameter configurations, please refer to Appendix C .

Panel of Figure 12 — Figure 12: We conduct learning-rate grid searches for Muon , NorMuon , HTMuon , and HTMuon+NorMuon across various datasets and architectures, including LLaMA models on C4 and ResNet models on CIFAR-100 and CIFAR-10. We find that HTMuon outperforms Muon and NorMuon across most learning rates, demonstrating the effectiveness and robustness of our method.

	LLaMA-60M				LLaMA-135M
Optimizer	0.01	0.02	0.03	0.04	0.01	0.02	0.02	0.04
Muon	28.80	28.99	28.88	29.93	22.23	22.51	22.59	23.73
NorMuon	28.63	28.59	28.17	29.07	21.99	22.17	22.06	22.76
HTMuon	28.48	28.75	27.88	30.22	22.95	21.96	21.25	21.92
HTMuon $+$ NorMuon	30.14	28.63	27.55	28.03	22.71	21.82	21.11	21.76

Appendix

Table 11: Detailed results for learning rate grid search on C4. Red indicates the best value, and blue denotes the second-best value.

	ResNet18				ResNet50
Optimizer	0.015	0.02	0.025	0.03	0.015	0.02	0.025	0.03
Muon	77.10	77.16	77.95	77.53	79.15	79.41	79.67	79.85
NorMuon	76.75	77.82	77.64	77.63	79.13	79.78	79.74	79.61
HTMuon	77.18	77.65	78.24	78.21	79.55	80.03	80.16	80.07
HTMuon $+$ NorMuon	77.51	77.78	78.51	78.02	79.60	80.03	79.93	80.22

Appendix

Table 12: Detailed results for learning rate grid search on CIFAR-100. Red indicates the best value, and blue denotes the second-best value.

	ResNet18				ResNet50
Optimizer	0.01	0.015	0.02	0.025	0.01	0.015	0.02	0.025
Muon	95.10	95.39	95.25	95.36	95.8	95.90	95.97	95.90
NorMuon	95.30	95.35	95.57	95.24	95.74	96.03	95.91	95.75
HTMuon	95.27	95.23	95.63	95.39	96.13	95.84	96.07	95.86
HTMuon $+$ NorMuon	95.36	95.25	95.40	95.17	96.02	95.85	96.04	95.97

Appendix

Table 13: Detailed results for learning rate grid search on CIFAR-10. Red indicates the best value, and blue denotes the second-best value.

	LLaMA-60M					LLaMA-135M
Optimizer	0	0.1	0.125	0.2	0.25	0	0.1	0.125	0.2	0.25
HTMuon	28.80	27.91	27.88	28.56	29.97	22.23	21.33	21.25	21.66	22.43
HTMuon $+$ NorMuon	28.17	27.48	27.55	28.15	29.54	21.99	21.21	21.11	21.33	22.1

Appendix

Table 14: Detailed results for ablation study on

p

on C4 dataset. Red indicates the best value, and blue denotes the second-best value.

	ResNet18					ResNet50
Dataset	0	0.1	0.125	0.2	0.25	0	0.1	0.125	0.2	0.25
CIFAR-10	95.39	95.14	95.63	95.46	95.49	95.97	95.88	95.95	95.95	95.95
CIFAR-100	77.95	77.65	78.24	78.58	78.32	79.67	79.40	80.16	80.30	80.27

Appendix

Table 15: Detailed results for ablation study on

p

for HTMuon on CIFAR datasets. Red indicates the best value.

Optimizer	0	0.1	0.125	0.2	0.25
HTMuon	22.46	22.40	22.20	22.49	22.83

Appendix

Table 16: Detailed results for ablation study on

p

for HTMuon on OpenWebText dataset. Red indicates the best value.

	Muon	HTMuon	NorMuon	HTMuon+NorMuon	HTMuon_HT	HTMuon_HT+NorMuon
LLaMa-60M	28.8	27.88	28.17	27.55	28.52	28.02
LLaMa-135M	22.23	21.25	21.99	21.11	21.65	21.26

Appendix

Table 17: Detailed results for HTMuon_HT . Red indicates the best value, and blue denotes the second-best value. We use

\alpha=0.25

and learning rate

=0.03

in Algorithm 6 for experiments.

Params	Hidden	Intermediate	Heads	Blocks	Steps	Data amount	Batch Size
60M	512	1376	8	8	10K	1B	512
135M	768	2048	12	12	20K	2B	512
350M	1024	2736	16	24	60K	6B	512
1B	2048	5461	32	24	90K	9B	512

Appendix

Table 18: Hyperparameters of LLaMA models.

	LLaMA-60M		LLaMA-135M
Optimizer	LR	WD	LR	WD
Adam	1e-3	1e-5	1e-3	1e-5
AdamW	1e-3	0.1	1e-3	0.1
Muon	{ 0.01 ,0.02,0.03,0.04}	0.1	{ 0.01 ,0.02,0.03,0.04}	0.1
NorMuon	{0.01,0.02, 0.03 ,0.04}	0.1	{ 0.01 ,0.02,0.03,0.04}	0.1
AdaMuon	{6e-4, 1e-3, 5e-3 , 1e-2}	0.1	{ 1e-3, 5e-3, 1e-2 , 2e-2}	0.1

Appendix

Table 19: Hyperparameters for LLaMA-60M and LLaMA-135M On C4 Dataset. The boldfaced values denote the optimal hyperparameters.

	LLaMA-350M		LLaMA-1B
Optimizer	LR	WD	LR	WD
Adam	1e-3	1e-5	6e-4	1e-6
AdamW	1e-3	0.1	6e-4	0.1
Muon	{0.0025, 0.005 ,0.01,0.015}	0.1	{ 0.005 ,0.01}	0.1
HTMuon	{0.0025, 0.005 ,0.01,0.015}	0.1	{ 0.005 ,0.01}	0.1

Appendix

Table 20: Hyperparameters for LLaMA-350M and LLaMA-1B On C4 Dataset. The boldfaced values denote the optimal hyperparameters.

	ResNet18		ResNet50
Optimizer	LR	WD	LR	WD
SGDM	{0.1,0.2, 0.3 ,0.4}	5e-4	{0.1,0.2, 0.3 ,0.4}	5e-4
Muon	{0.015,0.02, 0.025 ,0.03}	0.1	{0.015,0.02,0.025, 0.03 }	0.1
NorMuon	{0.015, 0.02 ,0.025,0.03}	0.1	{0.015, 0.02 ,0.025,0.03}	0.1
HTMuon	{0.015,0.02, 0.025 ,0.03}	0.1	{0.015,0.02, 0.025 ,0.03}	0.1
HTMuon + NorMuon	{0.015,0.02, 0.025 ,0.03}	0.1	{0.015,0.02, 0.025, 0.03 }	0.1

Appendix

Table 21: Hyperparameters for ResNet18 and ResNet50 On CIFAR-100 Dataset. The boldfaced values denote the optimal hyperparameters.

	ResNet18		ResNet50
Optimizer	LR	WD	LR	WD
SGDM	{0.1, 0.2 ,0.3,0.4}	5e-4	{0.1,0.2, 0.3 ,0.4}	5e-4
Muon	{0.01, 0.015 ,0.02,0.025}	0.1	{0.01,0.015, 0.02 ,0.025}	0.1
NorMuon	{0.01,0.015, 0.02 ,0.025}	0.1	{0.01, 0.015 ,0.02,0.025}	0.1
HTMuon	{0.01,0.015, 0.02 ,0.025}	0.1	{ 0.01 ,0.015,0.02,0.025}	0.1
HTMuon + NorMuon	{0.01,0.015, 0.02 ,0.025}	0.1	{0.01,0.015, 0.02 ,0.025}	0.1

Appendix

Table 22: Hyperparameters for ResNet18 and ResNet50 On CIFAR-10 Dataset. The boldfaced values denote the optimal hyperparameters.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 149 papers

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
Feb 25, 2026Ruijie Zhang, Yequan Zhao, Ziyue Liu +4Muon Optimizer Deep Learning Optimization
University of California at Santa Barbara
DynMuon: A Dynamic Spectral Shaping View of Muon
May 16, 2026Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1Neural Network Optimization Muon Optimizer
University of Wisconsin–Madison · MIT · Elorian AI
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model Pretraining Efficient Language Model Training
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Feb 25, 2026J/K move · Enter open · S save

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang+3

University of California at Santa Barbara

Muon Optimizer Deep Learning Optimization Language Model Pretraining

Open paper AskPDF arXiv

Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition is that polar iterations, by flattening the singular spectrum to all ones, should also eliminate column- and row-wise norm imbalance in the update. We show that this is not true in practice: practical polar steps can substantially amplify the imbalance. We term this the post-polar imbalanced update problem, and prove that such imbalance tightens the second-order term in a blockwise descent analysis, weakening Muon's per-step descent guarantee. Motivated by this analysis, we propose Muon+, a one-line fix that inserts a single normalization step after polar orthogonalization. Muon+ adds no optimizer state. Across pre-training experiments on GPT and LLaMA models from 60M to 7B parameters, spanning both compute-optimal budgets and extended token-to-parameter ratios up to approximately 200, Muon+ consistently outperforms Muon in terms of training and validation perplexity, leading to significant overall pre-training speedup.

Abstract

Muon Optimizer

Explore similar work

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

DynMuon: A Dynamic Spectral Shaping View of Muon

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining