BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

Rida Assalouh; Tom Jacobs; Rebekka Burkholz; Yani Ioannou; Umut Şimşekli

cs.LGOct 16, 2026

BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

Authors: Rida Assalouh, Tom Jacobs, Rebekka Burkholz, Yani Ioannou, Umut Şimşekli

Organizations: Inria, Paris, France · ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Schulich School of Engineering, University of Calgary, Calgary, Alberta, Canada ·

Abstract

Muon has emerged as a strong optimizer for neural network training and admits an interpretation as steepest descent under a spectral-norm geometry. Yet we find that Muon-trained Transformers can degrade sharply when their MLP hidden channels are pruned after training. To reconcile Muon's optimization performance with structured compressibility, we introduce Budget Muon (BMuon), which modifies Muon's spectral geometry so that hidden channels across blocks compete for a shared allocation budget. The resulting oracle is exactly normalized steepest descent under a family of norms that interpolate between Muon's geometry and structured sparse geometries. Our principal sparsity control is an interpretable global channel-allocation budget, rather than a setting-dependent sparsity-penalty coefficient. For linear factorization models, we show that Muon admits minimum-cost exact representations with redundant hidden width, whereas under the BMuon oracle geometry and a sufficiently scarce budget, every minimum-cost representation is width-minimal. For practical training, we derive a minorization-maximization surrogate for the exact allocation problem. On ImageNet-1K with ViT-S/16, BMuon improves over Muon under gradual structured MLP pruning, and its learned allocations provide a more effective pruning criterion than weight magnitude. On a 1B-parameter LLaMA-style model trained on FineWeb, BMuon improves over Muon under post-training MLP channel pruning.

Figures & tables

Figure 1: Post-training global MLP pruning on ImageNet-1K with ViT-S/16. Mean

\pm

std over 3 seeds. See Appendix E.1 .

Figure 2: CIFAR-100 ViT post-training MLP compression. Mean test accuracy

\pm

sample std. over three seeds after compression without recovery. Muon, Muon+GL, and BM-mag use global MLP-channel pruning ranked by magnitude; BM-

\mu

uses

\mu

for ranking, while NuMuon uses parameter-matched truncated-SVD compression of the MLP weights.

Gradual pruning
	CIFAR-100		ImageNet-1K
Method	90%	95%	90%	95%
Muon-mag	$63.00\pm 0.28$	$59.70\pm 0.71$	$65.26\pm 0.19$	$61.24\pm 0.47$
BMuon-mag	$64.78\pm 0.89$	$62.88\pm 0.22$	$68.71\pm 0.11$	$65.77\pm 0.24$
BMuon- $\mu$	$\mathbf{67.24\pm 0.22}$	$\mathbf{63.91\pm 0.07}$	$\mathbf{69.06\pm 0.19}$	$\mathbf{67.04\pm 0.02}$
Architectures trained from scratch

Table 1: Gradual global MLP-channel pruning. Final Top-1 accuracy (%), reported as mean

\pm

sample std. over three seeds. The upper block reports the final pruned models after recovery; the lower block trains the architectures selected by BMuon-

\mu

from scratch with Muon and compares their accuracy with uniform-width models having nearly the same total number of MLP channels. Details in Appendix E.4 .

Figure 3: Post-training MLP-channel pruning at language-model scale. Validation perplexity after global structured pruning of a 1B-parameter LLaMA-style model trained on FineWeb. Mean

\pm

std. over three seeds.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Accuracy of the practical allocation approximation. (a) Certified relative deficit of one practical $\mathrm{NS}_{5}$ -scored MM update. The same update is evaluated on states generated by the 1-step refresh run and by the certified refresh run. At each training stage, the line is the median over three seeds and the shaded region is their minimum–maximum range. The vertical dashed line marks the beginning of the fixed-budget phase $K_{t}/N=0.3$ . (b) Certified relative deficit caused by keeping the allocation installed by the certified refresh run fixed for $s=0,\ldots,4$ optimizer steps. For each seed and each $s$ , we first take the largest deficit over the ten sampled windows; the line is the median of these three seed-wise maxima and the shaded region is their minimum–maximum range.

Figure 5: End-to-end effect of the allocation approximation. We compare the 1-step refresh run with the certified refresh run; both refresh

\mu

every five optimizer steps. (a) CIFAR-100 test accuracy during 100 epochs of training. (b) Test accuracy after global structured MLP-channel pruning of the final trained model, using its final

\mu

to rank channels, without fine-tuning. For each curve, the line is the median over three seeds and the shaded region is their minimum–maximum range.

	$90\%$ pruning		$95\%$ pruning
Seed	BMuon	Muon	BMuon	Muon
0	92.55	92.06	92.47	92.50
1	92.02	90.72	92.27	91.59
2	92.81	93.05	91.57	91.48

Appendix

Table 2: ImageNet-1K training time. Mean epoch time in seconds over each full

150

-epoch gradual-pruning run, excluding evaluation. Differences between optimizers are comparable to seed-to-seed variation.

	BMuon		Muon
Seed	ms/step	Time (h)	ms/step	Time (h)
0	4832.4	8.379	4929.3	8.547
1	4836.0	8.385	4893.5	8.485
2	4906.5	8.507	4932.0	8.551

Appendix

Table 3: LLaMA training time. Mean time per training step and total training time for each full

13.1

B-token run, excluding evaluation and checkpointing. The difference between optimizers is comparable to seed-to-seed variation and does not indicate a slowdown of BMuon.

MLP pruning (%)	Muon + mag.	BMuon + mag.
0	$75.230\pm 0.022$	$74.174\pm 0.220$
10	$64.201\pm 1.184$	$74.183\pm 0.236$
20	$26.762\pm 1.387$	$74.188\pm 0.230$
30	$3.167\pm 0.784$	$74.180\pm 0.228$
40	$0.513\pm 0.211$	$74.182\pm 0.225$
50	$0.249\pm 0.057$	$74.177\pm 0.230$

Appendix

Table 4: ImageNet-1K post-training structured pruning. Top-1 accuracy (%) after global MLP-channel pruning of ViT-S/16 using coupled weight-magnitude ranking. Results are mean

\pm

sample standard deviation over three seeds. No post-pruning recovery is performed.

Compression	Muon	Muon+GL	NuMuon	BM-mag	BM- $\mu$
0%	$71.49\pm 0.21$	$71.25\pm 0.02$	$69.95\pm 0.13$	$71.46\pm 0.15$	$71.46\pm 0.15$
10%	$66.12\pm 0.86$	$71.25\pm 0.02$	$69.90\pm 0.08$	$71.46\pm 0.15$	$71.46\pm 0.15$
20%	$20.26\pm 3.53$	$71.25\pm 0.02$	$69.79\pm 0.17$	$71.46\pm 0.15$	$71.46\pm 0.15$
30%	$5.72\pm 0.55$	$71.25\pm 0.02$	$69.60\pm 0.04$	$71.46\pm 0.15$	$71.46\pm 0.16$
40%	$4.18\pm 0.64$	$71.25\pm 0.02$	$68.70\pm 0.14$	$71.46\pm 0.15$	$71.46\pm 0.16$
50%	$3.77\pm 1.38$	$71.25\pm 0.02$	$66.65\pm 0.36$	$71.46\pm 0.15$	$71.46\pm 0.14$

Appendix

Table 5: CIFAR-100 post-training compression results. Test accuracy (%) reported as mean

\pm

sample standard deviation over three seeds. All compression levels are evaluated independently from the final epoch-200 checkpoint, without post-compression fine-tuning.

Panel of Figure 6 — Figure 6: Effect of the budget on post-training compression. CIFAR-100 test accuracy after global MLP-channel pruning for different budget ratios $K/N$ . Panel (a) uses coupled weight-magnitude ranking, while panel (b) uses the final learned allocation $\mu$ . Mean $\pm$ sample std. over three seeds.

Pruning	Seed 0	Seed 1	Seed 2
90%	$[119,94,18,4,9,25]$	$[120,95,20,4,13,17]$	$[114,99,18,6,13,19]$
95%	$[80,37,4,5,4,4]$	$[78,39,5,4,4,4]$	$[79,37,5,5,4,4]$

Appendix

Table 6: BMuon-

\mu

architectures on CIFAR-100. Per-block MLP widths for blocks

1

–

6

, selected at the end of gradual pruning.

Pruning	Seed	MLP widths, blocks 1–12
90%	0	$[250,168,194,164,159,153,141,140,106,122,177,69]$
	1	$[254,174,204,197,151,132,141,139,98,135,153,65]$
	2	$[245,194,216,199,158,119,131,137,104,121,159,60]$
95%	0	$[181,120,120,106,100,67,58,43,40,34,39,14]$
	1	$[170,121,133,127,91,60,62,53,34,22,43,6]$
	2	$[156,110,107,99,100,73,59,66,37,40,55,20]$

Appendix

Table 7: BMuon-

\mu

architectures on ImageNet-1K. Per-block MLP widths for blocks

1

–

12

, selected at the end of gradual pruning.

MLP pruning	Muon		BMuon
MLP pruning	Magnitude	Activation	Magnitude	Activation	$\mu$
$0\%$	$10.921\pm 0.033$	$10.921\pm 0.033$	$12.843\pm 0.051$	$12.843\pm 0.051$	$12.843\pm 0.051$
$10\%$	$16.100\pm 1.411$	$12.701\pm 0.229$	$12.843\pm 0.051$	$12.843\pm 0.051$	$12.843\pm 0.051$
$20\%$	$22.944\pm 3.728$	$26.249\pm 1.974$	$12.843\pm 0.051$	$12.843\pm 0.051$	$12.843\pm 0.051$
$30\%$	$40.241\pm 8.109$	$62.840\pm 7.917$	$12.843\pm 0.051$	$12.843\pm 0.051$	$12.843\pm 0.051$
$40\%$	$76.861\pm 16.225$	$251.840\pm 55.803$	$12.843\pm 0.051$	$12.843\pm 0.051$	$12.843\pm 0.051$

Appendix

Table 8: Post-training structured pruning of the 1B-parameter LLaMA-style model. Validation perplexity after global SwiGLU-channel pruning. Entries report mean

\pm

sample std. over seeds

\{0,1,2\}

. No post-pruning recovery is performed.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 139 papers

NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Mar 4, 2026Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7LLM Input Optimization LLM Compression
Pluralis Research
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
Muon Meets Mamba: Spectral Optimization for State Space Models
Aug 4, 2026Arslan Battalov, Karim Kramin, Alexander Markotenko +1Neural Network Optimization Muon Optimizer
HSE University · Moscow, Russia

Mar 4, 2026J/K move · Enter open · S save

NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Chamin P Hewa Koneputugodage+6

Pluralis Research

LLM Input Optimization LLM Compression Low-Rank Compression Language Model Pretraining

Open paper AskPDF arXiv

The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-of-the-art compression pipelines leverage the low-rank structure of trained weight matrices, a phenomenon often associated with the properties of popular optimizers such as Adam. In this context, Muon is a recently proposed optimizer that improves LLM pretraining via full-rank update steps, but its induced weight-space structure has not been characterized yet. In this work, we report a surprising empirical finding: despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines. Motivated by this insight, we propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure. Across billion-parameter-scale models, we show that NuMuon increases weight compressibility and improves post-compression model quality under state-of-the-art LLM compression pipelines while retaining Muon's favorable convergence behavior.

Abstract

Muon Optimizer

Explore similar work

NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

Muon Meets Mamba: Spectral Optimization for State Space Models