Muon Operates Beyond the Edge of Stability

Niko Covic; Baraq Lipshitz; Alexander Padula; Stefan Visnjic

cs.LGOct 16, 2026

Muon Operates Beyond the Edge of Stability

Authors: Niko Covic, Baraq Lipshitz, Alexander Padula, Stefan Visnjic

Organizations: Department of Informatics, ETH Zurich, Switzerland.

Abstract

We study the Muon optimizer using sharpness and provide the first empirical evidence that the Flat Minima Hypothesis also applies to Muon. Further, we derive Muon's implicit preconditioner to analyze its potential Edge of Stability behavior. Experiments on CIFAR-10 show sharpness strongly correlates with generalization gap within each optimizer (Pearson $r = 0.45$ - $0.69$ , $p \leq 10^{-4}$ ), confirming the hypothesis holds for Muon similarly to Adam and SGD. However, Muon operates above the classical sharpness-based Adaptive Edge of Stability threshold that bounds diagonally-preconditioned optimizers like Adam and RMSprop, yet remains stable. Our findings suggest Muon operates in a fundamentally different regime than diagonally-adaptive methods. Defining the exact stability boundary for Muon remains an open question. All code is available at https://github.com/NikoCovic/DL-project.

Figures & tables

Figure 1: Per-optimizer correlation of raw/adaptive sharpness with generalization gap using a fixed LR. Each point is the final checkpoint for 1 of 64 runs per optimizer.

Figure 2: The training loss (left), raw sharpness (middle) and Effective Sharpness (right) of VanillaMuon (

\eta,\mu

) with

\mu=0.9

and two different

\eta

. The dashed lines in the right plot represent the threshold where it is expected for Muon to oscillate, computed as

(2+2\mu)/\eta

( Goh, 2017 ; Cohen et al., 2024 ) . Clearly, VanillaMuon (

\eta,\mu

) operates beyond this threshold.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

LR	Optimizer	Train Acc	Val Acc	Acc Gap	Raw ( $\times 10^{2}$ )	ASAM ( $\times 10^{-2}$ )
Fixed	Normalized Muon	0.97 $\pm$ 0.005	0.90 $\pm$ 0.007	0.07 $\pm$ 0.007	0.69 $\pm$ 0.075	0.82 $\pm$ 0.229
Fixed	Decoupled Muon	0.98 $\pm$ 0.004	0.90 $\pm$ 0.006	0.08 $\pm$ 0.007	0.58 $\pm$ 0.080	0.96 $\pm$ 0.307
Fixed	Coupled SGD	0.87 $\pm$ 0.012	0.80 $\pm$ 0.057	0.07 $\pm$ 0.051	0.62 $\pm$ 0.135	2.06 $\pm$ 0.911
Fixed	Coupled Adam	0.91 $\pm$ 0.007	0.87 $\pm$ 0.012	0.04 $\pm$ 0.014	1.34 $\pm$ 0.266	1.93 $\pm$ 0.626
LDS	Normalized Muon	1.00 $\pm$ 0.001	0.94 $\pm$ 0.002	0.06 $\pm$ 0.002	0.45 $\pm$ 0.078	0.71 $\pm$ 0.097
LDS	Decoupled Muon	1.00 $\pm$ 0.001	0.93 $\pm$ 0.002	0.07 $\pm$ 0.002	0.13 $\pm$ 0.076	0.70 $\pm$ 0.283

Appendix

Table 1: For each Learning Rate scheduler (LR) and Optimizer pairing, we report Validation Accuracy (Val Acc), Generalization Gap (Gap), Raw Sharpness (Raw), and Adaptive Sharpness (ASAM) scores. Each entry aggregates the mean and standard deviation across 64 runs.

Figure 3: Per-optimizer correlation of raw/adaptive sharpness with generalization gap using a LDS.

Figure 4: The training loss (top), raw sharpness (middle) and Effective Sharpness (bottom) of VanillaAdam (

\eta,\beta_{1},\beta_{2}

) with

\beta_{1}=0.9,\beta_{2}=0.999

and three different

\eta

(left) and RMSprop (

\eta,\alpha

) (right) with

\alpha=0.99

and three different

\eta

. The dashed lines in the bottom plots represent the threshold where it is expected for the algorithms to oscillate, computed as

2/\eta

for RMSprop (

\eta,\alpha

) and

(2+2\beta_{1})/\eta

for VanillaAdam (

\eta,\beta_{1},\beta_{2}

) ( Goh, 2017 ) . VanillaAdam oscillates around the threshold and RMSprop slightly above the threshold, exactly as observed in previous work ( Cohen et al., 2024 )

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Sep 10, 2026Zhuanghua Liu, Menglian Wang, Luo LuoMuon Optimizer LLM Training
National University of Singapore · Wonders Information Co., Ltd. · Fudan University
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
Why Muon Outperforms Adam: A Curvature Perspective
Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2Loss Landscape Sharpness Neural Network Optimization
National University of Singapore · Yale University · University of Minnesota

Sep 10, 2026J/K move · Enter open · S save

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

Zhuanghua Liu, Menglian Wang, Luo Luo

National University of Singapore · Wonders Information Co., Ltd. · Fudan University

Muon Optimizer LLM Training

Open paper AskPDF arXiv

Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations.

Abstract

Muon Optimizer

Explore similar work

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

Why Muon Outperforms Adam: A Curvature Perspective