NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

Hadi Mohaghegh Dolatabadi; Thalaiyasingam Ajanthan; Sameera Ramasinghe; Chamin P Hewa Koneputugodage; Shamane Siriwardhana; Violetta Shevchenko; Karol Pajak; James Snewin

cs.LGMar 4, 2026

NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

Authors: Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Chamin P Hewa Koneputugodage, Shamane Siriwardhana, Violetta Shevchenko, Karol Pajak, James Snewin,

Organizations: Pluralis Research

Abstract

The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-of-the-art compression pipelines leverage the low-rank structure of trained weight matrices, a phenomenon often associated with the properties of popular optimizers such as Adam. In this context, Muon is a recently proposed optimizer that improves LLM pretraining via full-rank update steps, but its induced weight-space structure has not been characterized yet. In this work, we report a surprising empirical finding: despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines. Motivated by this insight, we propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure. Across billion-parameter-scale models, we show that NuMuon increases weight compressibility and improves post-compression model quality under state-of-the-art LLM compression pipelines while retaining Muon's favorable convergence behavior.

Figures & tables

Panel of Figure 1 — Figure 1: Normalized stable rank evolution for Qwen3-0.6B across training steps for feedforward projection matrices. Each subplot shows the mean stable rank (normalized by the maximum rank), with shaded regions indicating standard deviation across all layers. All other weight matrices exhibit a similar low-rank behavior throughout training (see Figure 13 in the Appendix).

Figure 2: Validation perplexity on WikiText2 against generation inference throughput for Llama3-1.8B models compressed via SVD-LLM ( Wang et al., 2025b ) . As seen, for a given perplexity, NuMuon-trained models provide the fastest inference for moderate to extreme compression rates (40-80%). Our results for other models can be found in Figure 20 .

Panel of Figure 3 — Figure 3: $\delta_{1}^{(\mathrm{F})}$ as a proxy for the tail bound in Assumption 3.5 for the feedforward projection matrices. As we see, this quantity is bounded and close to zero for NuMuon, supporting this assumption. For other parameters, please see Figure 10 .

Optim.	Train PPL $\downarrow$	Val. PPL $\downarrow$	Time/Step (s) $\downarrow$	GPU Mem. (GB) $\downarrow$
AdamW	10.83	18.75	25.12	37.62
Muon	10.01	15.39	25.39	32.51
NuMuon	10.59	17.56	30.03	33.88

Table 1: Training/validation perplexity (PPL) and efficiency metrics across optimizers for training Llama3-1.8B models. Validation PPL is computed on WikiText2 dataset.

Panel of Figure 4 — Figure 4: Training loss convergence for language models of size 0.6B-1.8B parameter count. For each model family, we use AdamW, Muon, and NuMuon to train the model. For more details, please see Section C.1 .

Method	Optimizer	Val. PPL $\downarrow$	Downstream Tasks							Avg $\uparrow$
Method	Optimizer	Val. PPL $\downarrow$	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg $\uparrow$
ASVD	AdamW	718.71	22.61	37.25	30.15	6.00	26.40	54.52	49.88	32.40
	Muon	1443.55	24.66	31.90	29.28	7.08	27.80	55.93	50.12	32.40
	NuMuon	20.56 (-98.6%)	32.76	62.88	50.27	42.64	37.20	71.38	56.20	50.48 (+55.8%)
SVD-LLM	AdamW	27.94	28.41	49.37	40.69	29.52	30.00	65.13	51.22	42.05
	Muon	26.39	26.19	43.14	36.74	28.86	28.20	59.52	53.83	39.50

Table 2: WikiText2 validation perplexity and downstream task performance across compressed Llama3-1.8B models using various LLM compression methods 40% compression. Full results for all models and methods can be found in Section C.2 .

Figure 5: NuMuon’s relative performance against Muon under SoTA LLM compression methods. The scatter plot displays the performance improvement on downstream tasks as well as validation perplexity improvements for all compression methods at all rates. The positive quadrant shows performance improvement.

Scheduler	Time/Step (s)	Val. PPL $\downarrow$		Task Acc $\uparrow$
Scheduler	Time/Step (s)	0%	80%	0%	80%
Piecewise ( $0.25$ )	9.94	20.01	33.39	45.87	37.92
Cosine ( $0.25$ )	10.49	20.20	36.10	45.69	35.89
Fixed ( $0.05$ )	8.96	29.13	31.93	39.64	38.74
Fixed ( $0.25$ )	9.81	21.85	44.69	44.30	34.36
Fixed ( $0.50$ )	11.33	20.27	46.66	45.17	34.38

Table 3: 80% Dobi-SVD LLM compression performance for Qwen3-0.6B models compared to non-compressed base models under various rank schedulers for NuMuon.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Model Architecture
Model Configuration	Qwen3-0.6B	Olmo2-1.4B	Llama3-1.8B
Hidden Dimensions	1024	2048	2048
Number of Layers	28	16	30
Attention Heads	16	32	32
Key-Value Heads	8	8	8
Head Dimension	128	64	64

Appendix

Table 4: Model architectures and training configuration.

Optimizer
Configuration	AdamW	Muon	NuMuon
Learning Rate	3e-4	1e-2	1e-2
Weight Decay	–	0.1	0.1
Beta1	–	0.9	0.9
Beta2	–	0.95	0.95
Epsilon	1e-8	1e-8	1e-8

Appendix

Table 5: Optimizer configuration and learning rate scheduler settings.

Figure 11: Different rank schedulers used for training Qwen3-0.6B models with NuMuon. Please see Section 3.4 for more details.

Panel of Figure 12 — Figure 12: Normalized learning rate scheduler for training language models of size 0.6B-1.8B parameter count. For each model family, we use AdamW, Muon, and NuMuon to train the model. Following Semenov et al. (2025) extensive benchmarks, we use a WSD ( Hu et al., 2024 ) schedule for training Muon and NuMuon, while we use a cosine decay for AdamW. For more details, please see Section C.1 .

Panel of Figure 13 — Figure 13: Normalized stable rank evolution for Qwen3-0.6B across training steps for different weight matrices. Each subplot shows the mean stable rank (normalized by the maximum rank), with shaded regions indicating standard deviation across all layers.

Panel of Figure 14 — Figure 14: Normalized stable rank for weight matrices of Qwen3-0.6B models. Each subplot shows the stable rank (normalized by the maximum rank) of the converged model across all layers.

Panel of Figure 15 — Figure 15: Normalized stable rank for weight matrices of Olmo2-1.4B models. Each subplot shows the stable rank (normalized by the maximum rank) of the converged model across all layers.

Panel of Figure 16 — Figure 16: Normalized stable rank for weight matrices of Llama3-1.8B models. Each subplot shows the stable rank (normalized by the maximum rank) of the converged model across all layers.

Panel of Figure 17 — Figure 17: Normalized stable rank of well-known, large-scale models trained via Muon. The plot shows Moonlight-16B-A3B ( Liu et al., 2025 ) (top) as well as Kimi-K2-1T ( Bai et al., 2025 ) (bottom). For each model, we show the distribution of the normalized stable rank (left) as well as its evolution by the layer index (right).

Panel of Figure 18 — Figure 18: Grassmann distance between the top- $64$ left singular vectors of the weight space $\boldsymbol{W}$ and the optimizer update $\boldsymbol{\Delta W}$ . For NuMuon, we use the cosine rank scheduler. As demonstrated, while Muon ignores the structure of the weight space in its updates (high, fixed Grassmann distance), NuMuon gradually concentrates the update’s energy around the top subspaces of the weight space (lower Grassmann distance).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	25.39	27.30	49.66	36.04	27.77	31.20	65.02	49.64	40.95
	Muon	19.20	30.72	57.95	45.56	37.69	33.60	67.74	54.38	46.81
	NuMuon	20.20 (+5.2%)	29.95	55.68	43.51	35.40	34.40	67.41	53.51	45.69 (-2.4%)
20%	AdamW	61.91	22.53	37.21	30.83	12.13	28.20	54.68	51.70	33.90
	Muon	29.87	28.67	52.06	40.80	30.49	33.80	65.94	52.96	43.53

Appendix

Table 6: WikiText2 validation perplexity and downstream task performance across compressed Qwen3-0.6B models using ASVD.

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	20.58	30.20	55.47	44.96	34.76	35.80	68.93	50.67	45.83
	Muon	17.80	34.04	60.10	50.94	39.28	37.80	70.46	54.85	49.64
	NuMuon	19.06 (+7.1%)	31.06	59.55	48.40	37.80	37.00	68.99	55.41	48.32 (-2.7%)
20%	AdamW	35.89	28.07	49.12	40.12	23.09	30.20	64.47	50.67	40.82
	Muon	738.86	22.78	40.19	28.98	2.87	27.40	58.43	50.51	33.02

Appendix

Table 7: WikiText2 validation perplexity and downstream task performance across compressed Olmo2-1.4B models using ASVD.

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	18.75	33.79	62.67	51.28	40.46	37.00	71.82	55.33	50.34
	Muon	15.39	38.82	67.80	58.35	47.47	41.80	74.37	59.27	55.41
	NuMuon	17.56 (+14.1%)	35.49	65.19	54.96	44.38	38.20	72.91	58.56	52.81 (-4.7%)
20%	AdamW	27.19	31.57	57.53	46.29	32.25	36.20	68.06	52.01	46.27
	Muon	19.13	34.47	62.71	52.62	42.60	37.60	71.38	56.27	51.09

Appendix

Table 8: WikiText2 validation perplexity and downstream task performance across compressed Llama3-1.8B models using ASVD.

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	25.39	27.30	49.66	36.04	27.77	31.20	65.02	49.64	40.95
	Muon	19.20	30.72	57.95	45.56	37.69	33.60	67.74	54.38	46.81
	NuMuon	20.20 (+5.2%)	29.95	55.68	43.51	35.40	34.40	67.41	53.51	45.69 (-2.4%)
20%	AdamW	39.36	25.17	42.30	31.10	14.30	27.40	56.86	49.96	35.30
	Muon	27.89	26.62	45.33	37.70	28.04	31.40	62.46	53.83	40.77

Appendix

Table 9: WikiText2 validation perplexity and downstream task performance across compressed Qwen3-0.6B models using SVD-LLM (Whitening).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	20.58	30.20	55.47	44.96	34.76	35.80	68.93	50.67	45.83
	Muon	17.80	34.04	60.10	50.94	39.28	37.80	70.46	54.85	49.64
	NuMuon	19.06 (+7.1%)	31.06	59.55	48.40	37.80	37.00	68.99	55.41	48.32 (-2.7%)
20%	AdamW	26.65	26.11	50.38	39.95	27.60	32.00	65.23	51.85	41.87
	Muon	20.13	30.72	56.31	43.98	37.94	33.80	65.72	56.67	46.45

Appendix

Table 10: WikiText2 validation perplexity and downstream task performance across compressed Olmo2-1.4B models using SVD-LLM (Whitening).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	18.75	33.79	62.67	51.28	40.46	37.00	71.82	55.33	50.34
	Muon	15.39	38.82	67.80	58.35	47.47	41.80	74.37	59.27	55.41
	NuMuon	17.56 (+14.1%)	35.49	65.19	54.96	44.38	38.20	72.91	58.56	52.81 (-4.7%)
20%	AdamW	24.27	29.78	56.19	43.82	35.77	33.40	67.03	53.67	45.67
	Muon	18.04	30.20	58.38	47.86	40.69	34.40	66.54	58.33	48.06

Appendix

Table 11: WikiText2 validation perplexity and downstream task performance across compressed Llama3-1.8B models using SVD-LLM (Whitening).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	25.39	27.30	49.66	36.04	27.77	31.20	65.02	49.64	40.95
	Muon	19.20	30.72	57.95	45.56	37.69	33.60	67.74	54.38	46.81
	NuMuon	20.20 (+5.2%)	29.95	55.68	43.51	35.40	34.40	67.41	53.51	45.69 (-2.4%)
20%	AdamW	38.49	25.51	46.21	32.10	19.66	30.60	61.21	50.59	37.98
	Muon	25.84	29.52	51.09	40.22	30.43	32.40	66.87	53.04	43.37

Appendix

Table 12: WikiText2 validation perplexity and downstream task performance across compressed Qwen3-0.6B models using SVD-LLM (Whitening + LoRA).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	20.58	30.20	55.47	44.96	34.76	35.80	68.93	50.67	45.83
	Muon	17.80	34.04	60.10	50.94	39.28	37.80	70.46	54.85	49.64
	NuMuon	19.06 (+7.1%)	31.06	59.55	48.40	37.80	37.00	68.99	55.41	48.32 (-2.7%)
20%	AdamW	28.69	26.96	52.10	41.15	30.70	32.80	66.21	52.09	43.14
	Muon	21.48	32.76	57.74	46.80	38.19	35.60	68.93	54.70	47.82

Appendix

Table 13: WikiText2 validation perplexity and downstream task performance across compressed Olmo2-1.4B models using SVD-LLM (Whitening + LoRA).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	18.75	33.79	62.67	51.28	40.46	37.00	71.82	55.33	50.34
	Muon	15.39	38.82	67.80	58.35	47.47	41.80	74.37	59.27	55.41
	NuMuon	17.56 (+14.1%)	35.49	65.19	54.96	44.38	38.20	72.91	58.56	52.81 (-4.7%)
20%	AdamW	21.69	30.03	55.01	46.22	37.69	34.80	68.55	53.83	46.59
	Muon	18.04	30.20	58.38	47.86	40.69	34.40	66.54	58.33	48.06

Appendix

Table 14: WikiText2 validation perplexity and downstream task performance across compressed Llama3-1.8B models using SVD-LLM (Whitening + LoRA).

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	25.39	27.30	49.66	36.04	27.77	31.20	65.02	49.64	40.95
	Muon	19.20	30.72	57.95	45.56	37.69	33.60	67.74	54.38	46.81
	NuMuon	20.20 (+5.2%)	29.95	55.68	43.51	35.40	34.40	67.41	53.51	45.69 (-2.4%)
20%	AdamW	26.94	26.19	48.95	35.48	28.47	31.00	65.02	49.01	40.59
	Muon	19.90	30.03	57.15	45.58	37.51	33.60	68.93	54.22	46.72

Appendix

Table 15: WikiText2 validation perplexity and downstream task performance across compressed Qwen3-0.6B models using Dobi-SVD.

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	20.58	30.20	55.47	44.96	34.76	35.80	68.93	50.67	45.83
	Muon	17.80	34.04	60.10	50.94	39.28	37.80	70.46	54.85	49.64
	NuMuon	19.06 (+7.1%)	31.06	59.55	48.40	37.80	37.00	68.99	55.41	48.32 (-2.7%)
20%	AdamW	23.12	29.18	53.45	43.09	28.45	33.80	67.36	51.14	43.78
	Muon	18.81	31.91	58.59	48.39	35.57	35.20	68.88	56.67	47.89

Appendix

Table 16: WikiText2 validation perplexity and downstream task performance across compressed Olmo2-1.4B models using Dobi-SVD.

Comp.	Optimizer	Val. PPL	Downstream Tasks							Avg
Comp.	Optimizer	Val. PPL	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg
0%	AdamW	18.75	33.79	62.67	51.28	40.46	37.00	71.82	55.33	50.34
	Muon	15.39	38.82	67.80	58.35	47.47	41.80	74.37	59.27	55.41
	NuMuon	17.56 (+14.1%)	35.49	65.19	54.96	44.38	38.20	72.91	58.56	52.81 (-4.7%)
20%	AdamW	21.42	34.04	56.52	49.41	31.85	35.00	70.13	55.09	47.43
	Muon	16.37	34.56	63.59	54.99	45.97	38.80	71.33	58.01	52.46

Appendix

Table 17: WikiText2 validation perplexity and downstream task performance across compressed Llama3-1.8B models using Dobi-SVD.

Panel of Figure 19 — Figure 19: Validation perplexity on WikiText2 dataset for compressed LLMs (Qwen3-0.6B, Olmo2-1.4B, and Llama3-1.8B) using different compression methods. As seen, models pretrained via NuMuon consistently achieve the lowest validation perplexity, yielding better performance at extreme compression ratios.

Panel of Figure 20 — Figure 20: Validation perplexity on WikiText2 against generation inference throughput for Qwen3-0.6B, Olmo2-1.4B, and Llama3-1.8B models compressed via SVD-LLM. NuMuon trained models deliver the lowest perplexity given a throughput, and as such, are far more efficienct that AdamW- or Muon-trained models at moderate to extreme compression rates. Also, we observe that as the model size grows, NuMuon-trained models become more amenable to compression.

Comp.	Optim.	Rank Schedule	Val. PPL $\downarrow$	Downstream Tasks							Avg $\uparrow$
Comp.	Optim.	Rank Schedule	Val. PPL $\downarrow$	ARC-C	ARC-E	H’SWag	LAM’DA	OpenBkQA	PIQA	W’Grande	Avg $\uparrow$
0%	AdamW	-	25.39	27.30	49.66	36.04	27.77	31.20	65.02	49.64	40.95
	Muon	-	19.20	30.72	57.95	45.56	37.69	33.60	67.74	54.38	46.81
	NuMuon	Piecewise ( $25d/100$ )	20.01	29.69	56.44	43.78	35.51	35.00	67.79	52.88	45.87
		Cosine ( $25d/100$ )	20.20	29.95	55.68	43.51	35.40	34.40	67.41	53.51	45.69
		Fixed ( $5d/100$ )	29.13	25.60	48.06	33.68	25.13	31.20	63.98	49.80	39.64

Appendix

Table 18: WikiText2 validation perplexity and downstream task performance for various rank schedules.

Panel of Figure 21 — Figure 21: Normalized stable rank for weight matrices of Qwen3-0.6B models trained with NuMuon under different ranks/nuclear norm budget. Each subplot shows the stable rank (normalized by the maximum rank) of the converged model across all layers.

Panel of Figure 22 — Figure 22: Normalized stable rank for weight matrices of Qwen3-0.6B models trained with NuMuon under different rank schedulers. Each subplot shows the stable rank (normalized by the maximum rank) of the converged model across all layers.

LLM Input Optimization

See all 22 papers

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model Pretraining Efficient Language Model Training
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Oct 16, 2026Rida Assalouh, Tom Jacobs, Rebekka Burkholz +2Muon Optimizer Structured Sparsity
Inria, Paris, France · ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany +3
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Sep 28, 2026J/K move · Enter open · S save

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Chang-Wei Shi, Xu Wang, Wu-Jun Li

National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Language Model Pretraining Efficient Language Model Training LLM Training

Open paper AskPDF arXiv

The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called matrix-equilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.

Abstract

LLM Input Optimization

Explore similar work

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

Muown: Row-Norm Control for Muon Optimization