Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

Alexander Miasnikov; Ruslan Kabirov; Gleb Molodtsov; Dmitrii Vlashchuk; Vladimir Aletov; Nail Bashirov; Daniil Tikhonov; Maxim Krutikov

cs.LGOct 12, 2026

Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

Authors: Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov, Dmitrii Vlashchuk, Vladimir Aletov, Nail Bashirov, Daniil Tikhonov, Maxim Krutikov,

Organizations: BRAIn Lab · HSE University · MBZUAI

Abstract

Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization but its dense polar update remains expensive. We propose Monarch-Muon, a structured matrix-aware optimizer for weights parameterized by Monarch factors. The resulting structure reduces weights, gradients, and optimizer state, and exposes small independent update subproblems in place of a single dense orthogonalization; we formalize this through a blockwise trust-region interpretation. In quality runs, Monarch-Muon improves over Monarch-AdamW at fixed structure: 3.06 vs. 3.09 in the dense 257M model and 2.68 vs. 2.77 in the 1B MoE model; the two-block MoE variant also matches dense AdamW while using 15% fewer active and 25.4% fewer total parameters. In 6.89B-parameter systems measurements, Monarch-Muon reduces peak memory from 68.9 to 52.9 GB, Muon optimizer state from 14.60 to 6.77 GB, and the Muon optimizer step from 1288 to 304 ms.

Figures & tables

Figure 1: Training-step breakdown and optimizer-state memory for the 6.89B dense reference at microbatch 4 and four accumulation steps, from the same runs as Table 2 . Bars show median phase timings; labels give median full-step wall time. Periodic projection refreshes are excluded. Monarch–Muon uses four blocks and reduces both optimizer-state memory and optimizer-step overhead.

Figure 2: MoE pretraining at the same token budget. Two-block Monarch–Muon reaches the reported final validation loss of dense AdamW with fewer active and total parameters.

Figure 3: Learned gate and up projections in layer 0 of the dense quality model. At the same block count, Muon retains larger singular values in the tail than AdamW. Additional layers and components are in Appendix B .

Method	257M	834M	1.4B	3.5B	6.9B
Dense Muon	146.5 (26.2)	428.2 (88.9)	650.4 (156.7)	1473.0 (505.2)	2986.9 (1288.3)
Monarch–Muon	165.8 (8.0)	399.8 (29.2)	535.7 (50.5)	997.9 (140.5)	1638.6 (304.4)
GaLore	126.4 (6.1)	354.6 (14.6)	511.8 (20.4)	1005.7 (45.2)	1778.5 (93.9)
FRUGAL	124.9 (4.9)	349.9 (11.6)	507.5 (16.8)	989.8 (36.0)	1739.7 (69.2)
APOLLO	132.6 (12.5)	365.3 (27.7)	525.7 (32.5)	1016.0 (63.1)	1792.5 (119.3)
Fira	136.6 (16.2)	372.1 (34.6)	538.8 (47.3)	1055.8 (99.0)	1869.4 (199.8)

Table 2: Training costs on one H100 at microbatch 4, indexed by the dense reference model size. Monarch–Muon uses 4 blocks. Top: peak allocated memory and optimizer buffers, including projection matrices (GB). Bottom: steady-state training-step time, with optimizer-step time in parentheses (ms).

Figure 4: Moonlight checkpoints: KL to the original model after replacing one FFN by its 4-block projection (nats/token, log scale). Layer 0 is dense. Layers 1–26 are MoE.

Optimizer	Precision	Val. loss	Peak GB
Monarch–Muon	BF16	3.059	49.47
	FP8 GEMM	3.066	34.28
	FP8 states	3.060	48.52
Monarch–AdamW	BF16	3.088	49.97
	FP8 GEMM	3.093	34.75
	FP8 states	3.087	48.66

Table 5: Two-block Monarch after 39,250 steps. FP8 GEMM compresses computation and activations. FP8 states retain BF16 GEMMs. Validation uses BF16. Peak memory: final step on one H100.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Moonlight-16B-A3B: 4-block weight-projection errors normalized by Gaussian errors at matching shapes. Values near 1 indicate little alignment with Monarch structure. Points denote layers, pooling routed experts by projection type.

Figure 6: Qwen2.5-7B: 2-block weight-projection errors normalized by Gaussian errors at matching shapes. Points denote layers.

Panel of Figure 7 — Figure 7: Layer-wise singular-value spectra for layers 0 and 4 at the end of training. These plots extend Figure 3 from the compact gate/up illustration to all attention and MLP projections.

Panel of Figure 8 — Figure 8: Layer-wise singular-value spectra for layers 8 and 11 at the end of training. The same comparison shows that Monarch–Muon generally preserves a heavier spectral tail than Monarch–AdamW at fixed Monarch structure.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
May 26, 2026Jiacheng Li, Jianchao Tan, Hongtao Xu +5Language Model Pretraining Gradient Descent
Meituan, Beijing, China
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Memory-Efficient Optimization Muon Optimizer
National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
Reassessing Muon for Matrix Factorization
Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerDeep Learning Optimization Muon Optimizer
†. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

May 26, 2026J/K move · Enter open · S save

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

Jiacheng Li, Jianchao Tan, Hongtao Xu, Jiaqi Zhang+4

Meituan, Beijing, China

Language Model Pretraining Gradient Descent Muon Optimizer Accelerated Gradient Methods

Open paper AskPDF arXiv

The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term introduces curvature-sensitive corrections while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.

Abstract

Muon Optimizer

Explore similar work

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

Reassessing Muon for Matrix Factorization