MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

Athanasios Glentis; Dawei Li; Mingyi Hong

cs.LGOct 27, 2026

MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

Authors: Athanasios Glentis, Dawei Li, Mingyi Hong

Organizations: University of Minnesota

Abstract

Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the accumulated effective learning rate, $C_T=\sum_{t<T}η_t/\|W_t\|_F$ , where $η_t$ is the nominal learning rate and $η_t/\|W_t\|_F$ is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to $\sqrt{t}$ after $t$ steps. Consequently, MuonGS guarantees $C_T=Ω(\sqrt{T})$ , with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.

Figures & tables

Figure 1: One step of vanilla Muon, MuonW, MuonH and MuonGS from the same

W_{t}

and the same direction

O_{t}

, for a small (left) and a large (right) weight norm. Vanilla Muon’s radial part (dashed) grows the norm at first order; MuonH retracts to

\|W\|_{F}=R

(dotted); MuonGS keeps the tangential part only, so the norm changes at second order. MuonW’s decay

-\eta\lambda W_{t}

scales with

\|W_{t}\|_{F}

but the actual step depends on the size of

W_{t}

; the decay therefore hardly slows down the norm growth for a small norm but can shrink the norm for a large one.

Figure 2: An attainable gap in accumulation growth. Exact Muon is run on

F(W)=\exp(\operatorname{tr}W)

from

W_{0}=-4I_{2}

for

10^{5}

steps, with

\eta=0.04

and momentum coefficient

\beta=0.95

. MuonW uses

\lambda=0.2

, and MuonH uses

R=4\sqrt{2}

. All methods have the same orthogonalized direction

I_{2}

, which MuonH rescales to

D_{t}=4I_{2}

. Left: total accumulation. Right: accumulation over

[T,2T)

; the dotted line is

\log 2/\sqrt{2}

Figure 3: Accumulate effective learning rate $C_{T}$ vs training horizon $T$ . Results from training a 160M GPT-style model on FineWeb for 100,000 steps (about 50B tokens) with a constant learning rate of $0.005$ . Left: total accumulation $C_{T}$ . Right: accumulation over $[T,2T)$ .

Figure 4: Accumulate effective learning rate

C_{T}

vs learning rate scale

\eta

. Results from training a 160M GPT-style model on FineWeb for 20,000 steps (about 10.5B tokens). Each point represents a learning-rate and weight-decay setting; solid curves connect settings with the same weight decay (WD). WD= 0 is vanilla Muon, and positive WD corresponds to MuonW. Numbers beside the points report perplexity. Stars mark the lowest perplexity among the tested learning rates for each WD; the dashed line connects these stars for MuonW.

Control	Norm bound	$C_{T}$	$C_{2T}-C_{T}$
Vanilla Muon	$O(t)$	$\Omega(\log T)$	$\Omega(1)$
Weight decay	$O(1)$	$\Omega(T)$	$\Omega(T)$
Fixed norm	$R$	$\eta T/R$	$\eta T/R$
GS	$O(\sqrt{t})$	$\Omega(\sqrt{T})$	$\Omega(\sqrt{T})$

Table 1: Norm and accumulation guarantees at a constant positive learning rate.

Figure 5: Validation perplexity versus learning rate for MuonGS and MuonH across the two model sizes and Chinchilla ration

1{\times}

–

8{\times}

. Stars and dashed horizontal lines indicate the best observed perplexity in each panel.

Figure 6: Estimated token-equivalent speedup of MuonGS over MuonH for GPT-160M (left) and GPT-400M (right) across Chinclilla ratios

1{\times}

–

8{\times}

Figure 7: Validation perplexity versus learning rate for MuonGS and MuonH for an 135M active / 1B total MoE model for two different training durations. Stars and dashed horizontal lines indicate the best observed perplexity in each panel.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Validation perplexity vs learning rate for MuonGS, MuonH, Muon and MuonW with varying weight decay. Both models are trained for 20k steps (about 10.5B tokens) on FineWeb.

Figure 9: Weight-norm evolution for the 160M (left) and 400M (right) models over 20k training steps, using each method’s best learning rate. Here, $w_{t}$ concatenates the matrix weights updated by Muon variants, excluding auxiliary AdamW parameters. MuonW uses weight decay $0.0125$ which was found to be the best among the ones tested as shown 8

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 151 papers

May 11, 2026

Muown: Row-Norm Control for Muon Optimization

Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the

\ell_\infty

geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.

Kai Lion, Florian Hübler, Bingcong Li +2

Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

Language Model Pretraining Muon Optimizer

Ask PDF

Feb 25, 2026

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition is that polar iterations, by flattening the singular spectrum to all ones, should also eliminate column- and row-wise norm imbalance in the update. We show that this is not true in practice: practical polar steps can substantially amplify the imbalance. We term this the post-polar imbalanced update problem, and prove that such imbalance tightens the second-order term in a blockwise descent analysis, weakening Muon's per-step descent guarantee. Motivated by this analysis, we propose Muon+, a one-line fix that inserts a single normalization step after polar orthogonalization. Muon+ adds no optimizer state. Across pre-training experiments on GPT and LLaMA models from 60M to 7B parameters, spanning both compute-optimal budgets and extended token-to-parameter ratios up to approximately 200, Muon+ consistently outperforms Muon in terms of training and validation perplexity, leading to significant overall pre-training speedup.

Ruijie Zhang, Yequan Zhao, Ziyue Liu +4

University of California at Santa Barbara

Muon Optimizer Deep Learning Optimization Language Model Pretraining

Ask PDF

Sep 28, 2026

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called matrix-equilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.

Chang-Wei Shi, Xu Wang, Wu-Jun Li

National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Language Model Pretraining Efficient Language Model Training LLM Training

Ask PDF

Abstract

Muon Optimizer

Explore similar work

Muown: Row-Norm Control for Muon Optimization

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining