Approximate Muon with low-rank adapters

Ben Anson; Conor Houghton; Edward Milsom

cs.LGAug 14, 2026

Approximate Muon with low-rank adapters

Authors: Ben Anson, Conor Houghton, Edward Milsom

Organizations: University of Bristol School of Mathematics · University of Bristol School of Engineering Mathematics and Technology · University of Bath Department of Computer Science

Abstract

The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.

Figures & tables

Figure 1: Spectra of the weight change for different LoRA optimizers. We show the average singular values of the weight updates (attention weight, layer 6 of 12), during ReLoRA pretraining (

r=32

). We normalize by

\overline{\sigma}_{1}

, and denote the LoRA rank by the dashed grey line.

Figure 2

Eval	AdamW	Per-factor Muon	LoRA-Muon	Riemannion	sMuon (ours)
Qwen2.5-3B
Piqa	81.7	82.1	81.7	82.7	81.7
Winogrande	62.3	61.2	61.4	61.0	61.5
OBQA	78.4	79.2	76.0	77.4	77.0
Hellaswag	81.7	78.7	76.9	77.6	80.3
BoolQ	67.7	66.5	65.6	67.7	67.4

Table 1: Per-eval accuracy (%), grouped by base model. Each column is an optimizer (best run by validation loss); per row the best optimizer is bolded and cells are shaded green (best) to white (worst).

Eval	AdamW	Per-factor Muon	LoRA-Muon	Riemannion	sMuon (ours)
Qwen2.5-3B
HumanEval	54.9	20.1	43.3	31.1	51.8
HumanEval+	48.8	17.1	39.0	25.6	43.9
MBPP	65.9	69.3	68.0	65.9	62.7
MBPP+	57.1	60.6	58.7	56.9	53.4
Llama-3.2-3B

Table 2: Code pass@1 (%), grouped by base model. Each column is an optimizer (best run by validation loss); per row the best optimizer is bolded and cells are shaded green (best) to white (worst).

	AdamW	Per-factor Muon	LoRA-Muon	Riemannion	sMuon (ours)
Final val loss	3.83	3.70	3.68	3.66	3.66
Optimal LR	0.001	0.006	0.001	0.0006	0.0006

Table 3: ReLoRA pretraining: final validation loss at each optimizer’s optimal learning rate, where the optimal LR minimises the validation loss at the final logged step. Lower is better; the best optimizer is bolded and cells are shaded green (best) to white (worst).

Step time, ms (% total) $\downarrow$	$r=16$	$r=32$	$r=64$	$r=128$
AdamW	$0.66$ ( $0.047$ %)	$0.59$ ( $0.041$ %)	$0.65$ ( $0.046$ %)	$0.78$ ( $0.054$ %)
Per-factor Muon	$1.4$ ( $0.10$ %)	$1.6$ ( $0.11$ %)	$2.0$ ( $0.14$ %)	$3.6$ ( $0.25$ %)
LoRA-Muon	$4.5$ ( $0.32$ %)	$4.8$ ( $0.33$ %)	$6.3$ ( $0.44$ %)	$10$ ( $0.70$ %)
Riemannion	$90$ ( $6.0$ %)	$300$ ( $17$ %)	$700$ ( $33$ %)	$1700$ ( $54$ %)
sMuon (ours)	$8.7$ ( $0.62$ %)	$8.9$ ( $0.62$ %)	$12$ ( $0.80$ %)	$20$ ( $1.4$ %)
Fwd+bwd, ms	$1400$	$1432$	$1429$	$1459$

Table 4: All Muon optimizers are slightly slower than AdamW, but Riemannion adds huge overhead at larger rank. We show mean time for LoRA adapter optimizer step (ms, 2 s.f.), as well as the proportion (%) of the total step time. Figures were obtained while training a 12 layer GPT model with

d_{\text{model}}=768

. We ran these timings on an A100, calculating the mean time over 25 steps, with different ranks

r\in\{2^{4},2^{5},2^{6},2^{7}\}

, batch size 128, 4 gradient accumulation steps.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Common-sense accuracy against learning rate for each base model. Each curve is one optimizer; accuracy is the overall common-sense benchmark accuracy at the final training step.

Figure 3: Validation loss against learning rate for each base model, over the same runs as Figure 2 .

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 146 papers

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Deep Learning Optimization Riemannian Optimization
Ateneo de Manila University · EleutherAI · NaXys +1
Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning
Apr 11, 2026Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4Neural Network Optimization Muon Optimizer
University of California at Santa Barbara · University at Albany, SUNY
Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
Sep 15, 2025Chuan He, Zhanwang Deng, Zhaosong LuNeural Network Optimization Low-Rank Approximation
Department of Mathematics Linköping University, Sweden · Academy for Advanced Interdisciplinary Studies Peking University, Beijing, People’s Republic of China · Department of Industrial and Systems Engineering University of Minnesota, USA

Jun 11, 2026J/K move · Enter open · S save

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

Franz Louis Cesista, Katherine Crowson, Cédric Simal, Stella Biderman

Ateneo de Manila University · EleutherAI · NaXys+1

Deep Learning Optimization Riemannian Optimization Muon Optimizer Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensitive to initialization choices, its optimal learning rates transfer poorly across ranks, and it often fails to beat dense baselines. We derive LoRA-Muon by applying the Muon optimizer's spectral steepest-descent rule to the low-rank setting. Along with our split weight-decay rule, our main claim is that LoRA-Muon is a good low-rank proxy for full-rank Muon and Shampoo-family optimizers. Its optimal learning rates transfer across rank, width, depth, and factor-rescaling. In our compute-matched TinyShakespeare study, a rank-

2

proxy recovers the dense best tested learning rate, and a rank-

32

LoRA-Muon run attains lower mean validation loss than the dense baseline in the seed-averaged sweep. We further show that the Spectron optimizer depends on arbitrary factor scaling, so it would likely be a poor fit when finetuning starts from badly imbalanced factors, and that LoRA-RITE's simplified QR-coordinate core implements the same spectral update. LoRA-Muon computes that update without QR-decomposition and avoids storing second moments, making it more accelerator-friendly and memory-efficient.

Abstract

Muon Optimizer

Explore similar work

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

Muon2^22: Boosting Muon via Adaptive Second-Moment Preconditioning

Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training

Muon $^2$ : Boosting Muon via Adaptive Second-Moment Preconditioning