T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

Dongyang Li; Mingsong Yan; Zheng Zhang; Sui Tang

cs.LGOct 18, 2026

T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

Authors: Dongyang Li, Mingsong Yan, Zheng Zhang, Sui Tang

Organizations: University of California, Santa Barbara Santa Barbara, CA 93106, USA

Abstract

Muon has recently emerged as a strong optimizer for large language model pretraining by exploiting matrix structure through polar orthogonalization. However, it processes each parameter matrix independently, whereas recent tensor-aware extensions of Muon jointly update groups of eligible matrix parameters through matrix unfoldings. In contrast, we propose T-MUON, a t-SVD-based generalization that groups structurally related Transformer gradients into third-order tensors and defines the corresponding Muon-style update directly through the $M$ -transform. We show that the exact T-MUON update is a steepest-descent direction under the $\star_M$ -spectral norm and establish a convergence guarantee. Across GPT-2-style and LLaMA-style models from 51M to 1.3B parameters trained on FineWeb and FineWeb-Edu, head-wise T-MUON with the identity transform consistently achieves lower validation perplexity than AdamW, Muon, and variants using DFT, DCT, or DST transforms. These gains persist in additional experiments with longer context lengths and larger token budgets, although the preferred tensorization mode varies across settings.

Figures & tables

t-Muon
Optimizer	50M	Small	Base	Large
AdamW	47.353	42.623	38.641	38.058
Muon	42.235	35.483	30.350	27.953
Head- $I$	40.568	33.893	29.074	26.959
Head-stack- $I$	40.515	33.767	29.071	26.966

Table 1: Mean validation perplexity on GPT-2-style models trained on 1B-FineWeb tokens. Lower is better, and the best result in each column is bold.

t-Muon
Optimizer	60M	130M	350M
AdamW	32.166	28.151	24.029
Muon	26.558	22.363	19.386
Head- $I$	25.948	21.915	19.055
Head-stack- $I$	25.930	21.915	19.058

Table 2: Mean validation perplexity on LLaMA-style models trained on 1B-FineWeb tokens. Lower is better, and the best result in each column is bold.

Setting	Ctx.	AdamW	Muon	t-Muon
GPT-Small	2048	65.99	41.87	36.91 [-1pt] Head- $I$
GPT-Base	4096	221.50	55.90	49.77 [-1pt] Layer- $F$
LLaMA-350M	4096	70.25	28.71	27.31 [-1pt] Head-stack- $I$

Table 3: Single-seed validation perplexity under longer contexts. The strongest t-Muon variant for each setting is identified below its score. Ctx. denotes context length in tokens.

Setting	Tokens	AdamW	Muon	t-Muon
GPT-Small	3B	28.97	27.94	27.03 [-1pt] QKV- $I$
GPT-Base	7B	20.96	20.47	19.85 [-1pt] Layer- $I$
LLaMA-130M	2.2B	21.51	19.35	19.00 [-1pt] Head- $I$

Table 4: Mean validation perplexity under larger token budgets for FineWeb datasets. The strongest t-Muon variant for each setting is identified below its score.

Panel of Figure 1 — Figure 1: Training-loss curves of LLaMA-1B on 13.1B FineWeb-Edu tokens. Left: The first 1000 training steps. Right: the last 1000 training steps. Solid lines are the average over the last 50 steps.

t-Muon
Optimizer	60M	130M	350M	1B
Tokens	1.2B	2.7B	7.4B	13.1B
AdamW	23.89	16.59	11.92	–
Muon	20.58	15.30	11.60	9.45
Head- $I$	20.23	15.09	11.41	9.34
Layer- $F$	20.74	15.44	11.69	–

Table 5: Validation perplexity of LLaMA-style models trained on FineWeb-Edu. Single runs at the selected learning rate; all variants are reported in Table 12 . The best result in each column is bold.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$L$	$d$	$H$	$d_{\mathrm{ff}}$	Params. (M)
GPT-50M	8	512	8	2,048	51.475
GPT-Small	12	768	12	3,072	124.439
GPT-Base	24	1,024	16	4,096	354.822
GPT-Large	36	1,280	20	5,120	774.029
LLaMA-60M	8	512	8	1,376	58.074
LLaMA-130M	12	768	12	2,048	134.106

Appendix

Table 6: Architecture configurations used of GPT and LLaMA style models. Here,

L

d

H

, and

d_{\mathrm{ff}}

denote the number of layers, hidden dimension, number of heads, and MLP intermediate dimension, respectively.

Hyperparameter
Context length	1,024
Batch size	512
LR schedule	Cosine
Warmup ratio	$0.1$
Weight decay	$0.1$
Polar approximation	PolarExpress ( Amsel et al., 2026 )

Appendix

Table 7: Training hyperparameters used in the main experiments.

t-Muon (proposed)
Optimizer	GPT-50M	GPT-Small	GPT-Base	GPT-Large
AdamW	47.35 $\pm$ 0.081 (0.005)	42.62 $\pm$ 0.083 (0.005)	38.64 $\pm$ 0.081 (0.001)	38.06 $\pm$ 0.121 (0.001)
Muon	42.24 $\pm$ 0.118 (0.02)	35.48 $\pm$ 0.052 (0.01)	30.35 $\pm$ 0.024 (0.01)	27.95 $\pm$ 0.046 (0.02)
Head- $F$	42.00 $\pm$ 0.079 (0.01)	35.23 $\pm$ 0.064 (0.01)	30.41 $\pm$ 0.048 (0.01)	28.30 $\pm$ 0.009 (0.01)
Head- $C$	41.73 $\pm$ 0.036 (0.01)	35.05 $\pm$ 0.018 (0.01)	30.30 $\pm$ 0.019 (0.01)	28.21 $\pm$ 0.063 (0.01)
Head- $S$	41.83 $\pm$ 0.019 (0.01)	35.09 $\pm$ 0.051 (0.01)	30.34 $\pm$ 0.024 (0.01)	28.22 $\pm$ 0.023 (0.01)

Appendix

Table 8: Complete validation-perplexity comparison on GPT-2-style models trained on 1B FineWeb tokens. Each entry reports mean

\pm

standard deviation over three seeds with the best learning rate in parentheses. The best mean in each column is bold.

t-Muon (proposed)
Optimizer	LLaMA-60M	LLaMA-130M	LLaMA-350M
AdamW	32.17 $\pm$ 0.062 (0.005)	28.15 $\pm$ 0.025 (0.001)	24.03 $\pm$ 0.061 (0.001)
Muon	26.56 $\pm$ 0.022 (0.04)	22.36 $\pm$ 0.012 (0.04)	19.39 $\pm$ 0.003 (0.02)
Head- $F$	26.69 $\pm$ 0.003 (0.04)	22.40 $\pm$ 0.023 (0.04)	19.49 $\pm$ 0.006 (0.02)
Head- $C$	26.40 $\pm$ 0.004 (0.04)	22.31 $\pm$ 0.006 (0.04)	19.45 $\pm$ 0.006 (0.02)
Head- $S$	26.43 $\pm$ 0.029 (0.04)	22.31 $\pm$ 0.013 (0.04)	19.44 $\pm$ 0.004 (0.02)

Appendix

Table 9: Complete validation-perplexity comparison on LLaMA-style models trained on 1B FineWeb tokens. Each entry reports mean

\pm

standard deviation over three seeds with the best learning rate in parentheses. The best mean in each column is bold.

t-Muon (proposed)
Optimizer	GPT-Small, 3B	GPT-Base, 7B	LLaMA-130M, 2.2B
AdamW	28.97 $\pm$ 0.018 (0.01)	20.96 $\pm$ 0.135 (0.005)	21.51 $\pm$ 0.033 (0.005)
Muon	27.94 $\pm$ 0.046 (0.02)	20.47 $\pm$ 0.128 (0.01)	19.35 $\pm$ 0.013 (0.02)
Head- $F$	27.62 $\pm$ 0.021 (0.02)	20.12 $\pm$ 0.110 (0.01)	19.58 $\pm$ 0.006 (0.04)
Head- $I$	27.09 $\pm$ 0.014 (0.02)	19.87 $\pm$ 0.112 (0.01)	19.00 $\pm$ 0.003 (0.02)
Head-stack- $F$	28.33 $\pm$ 0.050 (0.01)	20.51 $\pm$ 0.093 (0.01)	19.59 $\pm$ 0.002 (0.04)

Appendix

Table 10: Complete validation-perplexity comparison under larger token budgets. Each entry reports mean

\pm

standard deviation over three seeds with the best learning rate in parentheses. The best mean in each column is bold.

t-Muon (proposed)
Optimizer	GPT-Small	GPT-Base	LLaMA-350M
Context length	2048	4096	4096
AdamW	65.99 (0.001)	221.50 (0.0005)	70.25 (0.001)
Muon	41.87 (0.01)	55.90 (0.04)	28.71 (0.02)
Head- $F$	39.82 (0.04)	55.45 (0.005)	29.42 (0.02)
Head- $I$	36.91 (0.04)	51.84 (0.01)	27.35 (0.02)

Appendix

Table 11: Complete single-seed validation-perplexity comparison for the longer-context experiments. Results use seed 42. The best learning rate is shown in parentheses. The best value in each column is bold.

t-Muon (proposed)
Optimizer	LLaMA-60M, 1.2B	LLaMA-130M, 2.7B	LLaMA-350M, 7.4B	LLaMA-1B, 13.1B
AdamW	3.1737 / 23.89 (0.005)	2.8086 / 16.59 (0.005)	2.4782 / 11.92 (0.005)	–
Muon	3.0241 / 20.58 (0.04)	2.7277 / 15.30 (0.02)	2.4509 / 11.60 (0.01)	2.2457 / 9.45 (0.01)
Head- $I$	3.0071 / 20.23 (0.04)	2.7140 / 15.09 (0.02)	2.4349 / 11.41 (0.01)	2.2339 / 9.34 (0.01)
Head- $F$	3.0293 / 20.68 (0.04)	2.7426 / 15.53 (0.04)	–	–
Head- $C$	3.0282 / 20.66 (0.04)	2.7381 / 15.46 (0.02)	–	–

Appendix

Table 12: Complete FineWeb-Edu comparison. Each entry reports the final validation loss / perplexity of a single seed, with the selected learning rate in parentheses; the best result in each column is bold.

Optimizer	Models	Learning-rate grid
AdamW	All	$\{5{\times}10^{-4},\,10^{-3},\,5{\times}10^{-3},\,10^{-2}\}$
Muon and t-Muon	GPT-2-style	$\{5{\times}10^{-3},\,10^{-2},\,2{\times}10^{-2},\,4{\times}10^{-2}\}$
Muon and t-Muon	LLaMA-style	$\{5{\times}10^{-3},\,10^{-2},\,2{\times}10^{-2},\,4{\times}10^{-2},\,6{\times}10^{-2},\,8{\times}10^{-2}\}$

Appendix

Table 13: Learning-rate grids swept for each optimizer.

Panel of Figure 2 — Figure 2: Final validation perplexity as a function of the learning rate over the grids in Table 13 .

Panel of Figure 3 — Figure 3: Training-loss curves for GPT-2-style models trained on one billion FineWeb tokens.

Panel of Figure 4 — Figure 4: Training-loss curves for LLaMA-style models trained on one billion FineWeb tokens.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model Pretraining Efficient Language Model Training
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China
Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training
Mar 18, 2026Ben S. Southworth, Stephen ThomasDeep Learning Optimization Muon Optimizer
Theoretical Division, Los Alamos National Laboratory, USA. · Lehigh University, Computer Science
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Memory-Efficient Optimization Muon Optimizer
National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

Sep 28, 2026J/K move · Enter open · S save

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Chang-Wei Shi, Xu Wang, Wu-Jun Li

National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Language Model Pretraining Efficient Language Model Training LLM Training

Open paper AskPDF arXiv

The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called matrix-equilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.

Abstract

Muon Optimizer

Explore similar work

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models