The Newton-Muon Optimizer

Zhehang Du; Weijie Su

math.OCApr 1, 2026

The Newton-Muon Optimizer

Authors: Zhehang Du, Weijie Su

Organizations: University of Pennsylvania

Abstract

The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains largely elusive. In this paper, we introduce a surrogate model that not only sheds new light on the design of Muon, but more importantly leads to a new optimizer. In the same spirit as the derivation of Newton's method, the surrogate approximates the loss as a quadratic function of the perturbation to a weight matrix $W$ using only three matrices: the gradient $G$ , an output-space curvature matrix $H$ , and the data matrix $Z$ that stacks the layer inputs. By minimizing this surrogate in one step and adopting a certain isotropic assumption on the weights, we obtain the closed-form update rule (up to momentum and weight decay) $W \leftarrow W - η\cdot \mathrm{msgn}(G(ZZ^\top)^{-1})$ , where $η$ is the learning rate and $\mathrm{msgn}(X)=UV^\top$ if $X=USV^\top$ is a compact singular value decomposition. This new optimization method, which we refer to as Newton-Muon, shows that standard Muon can be interpreted as an implicit Newton-type method that neglects the right preconditioning induced by the input second moment. Empirically, on a reproduction of the earliest publicly released Modded-NanoGPT speedrun configuration using Muon for GPT-2 pretraining, Newton-Muon reaches the target validation loss in 6% fewer iteration steps and reduces wall-clock training time by about 4%.

Figures & tables

Figure 1: Top: short track Record #4 validation loss comparison on the Modded-NanoGPT speedrun benchmark. Record #4 is the earliest publicly released configuration using Muon, and our reproduction on a single H100 GPU is denoted Muon. Newton–Muon adds the activation right-preconditioner before the Newton–Schulz iterations. Newton–Muon reaches the Muon baseline final validation loss in

6\%

fewer steps; despite a

1.8\%

higher per-step cost from right-preconditioning, it reduces wall-clock time to that loss by about

4\%

. Bottom: CIFAR-10 experiments (Appendix C ) on a 32-layer residual MLP show that Newton–Muon outperforms both Muon and AdamW in both per-step efficiency and overall wall-clock time.

Panel of Figure 2 — Figure 2: Left: standard Muon is orthogonally equivariant. Right: Newton–Muon takes the pair $(G,Z)$ as input, and the diagram commutes if the right rotation of $G$ is accompanied by the transformation $Z\mapsto O_{n}^{\top}Z$ .

Figure 3: Numerical study with spiked activation second moment

\mathrm{diag}(\kappa,1,\ldots,1)

and

\kappa=64

. Top: baseline case

(N,p)=(8192,0.3)

. Middle: more uniform curvature

(N,p)=(8192,2.4)

. Bottom: smaller sample size

(N,p)=(1024,0.3)

. The left column shows the spectrum of

{H}

, and the right column shows the corresponding mean absolute scores

s({Q})

Method	Loss	Time (s)
AdamW	3.3801	7228.4
Muon	3.2793	7314.1
Newton–Muon	3.2611	7443.3

Table 1: Short track Record #4 setting (single H100).

Figure 4: Refresh ablation for Newton–Muon on short track Record #4. Grouped bar plot over refresh interval

k

, with one bar per EWMA coefficient

\beta

(ridge scaling fixed at

\gamma=0.2

and learning rate fixed at

0.0040

Figure 5: Left: Ridge-scaling ablation for Newton–Muon on short track Record #4. Bar plot over ridge scaling

\gamma

with

k=32

\beta=0.95

, and learning rate

0.0040

fixed. Right: Learning-rate ablation for Newton–Muon on short track Record #4. Bar plot over learning rate with

k=32

\beta=0.95

, and

\gamma=0.2

fixed.

Method	Run 1	Run 2	Run 3	Run 4	Avg. loss	Avg. time (s)
AdamW	3.4677	3.4631	3.4640	3.4566	3.4628	4272.0
Muon	3.2758	3.2777	3.2783	3.2830	3.2787	4305.9
Newton–Muon	3.2733	3.2736	3.2740	3.2745	3.2739	4342.4

Table 2: Short track Record #28 setting (single L40S).

Figure 6: Validation loss trajectories for the short track Record #28 setting (single L40S). We plot the second-best run for AdamW, Muon, and Newton–Muon, and show loss versus step (left) and loss versus wall-clock time (right).

Method	Run 1	Run 2	Run 3	Avg. loss
Muon	2.9190	2.9208	2.9190	2.9196
Newton–Muon	2.9175	2.9191	2.9181	2.9183

Table 3: Medium track Record #17 setting. Wall-clock time is not reported since we did not observe a substantial improvement.

Module	$n$	$\kappa({Z}{Z}^{\top})$	$d_{\max}/d_{\min}$	$\bar{o}/d_{\mathrm{mean}}$	$\gamma/d_{\mathrm{mean}}$
Attn out	512	$9.18\times 10^{3}$	$2.84\times 10^{1}$	$9.89\times 10^{1}$	$1.0\times 10^{-6}$
Attn QKV	512	$2.04\times 10^{6}$	$3.00\times 10^{0}$	$6.59\times 10^{1}$	$1.0\times 10^{-6}$
MLP expansion	512	$\infty$	$3.01\times 10^{0}$	$6.56\times 10^{1}$	$1.0\times 10^{-4}$
MLP contraction	2048	–	$7.23\times 10^{1}$	$1.88\times 10^{2}$	$1.0\times 10^{-6}$
Attn out	512	$2.34\times 10^{3}$	$3.02\times 10^{1}$	$7.21\times 10^{1}$	$1.0\times 10^{-6}$
Attn QKV	512	$9.77\times 10^{4}$	$1.66\times 10^{2}$	$4.22\times 10^{1}$	$1.0\times 10^{-6}$

Table 4: Diagnostics of the activation second moment

{Z}{Z}^{\top}

for four modules at step 1000 (top block) and step 50000 (bottom block). Here

d_{\min}

d_{\mathrm{mean}}

, and

d_{\max}

are the minimum, mean, and maximum diagonal entries of

{Z}{Z}^{\top}

\bar{o}=(1/n)\sum_{i}\sum_{j\neq i}|({Z}{Z}^{\top})_{ij}|

, and

\kappa({Z}{Z}^{\top})

is the condition number of

{Z}{Z}^{\top}

. The value

\kappa({Z}{Z}^{\top})

is reported when eigenvalues were explicitly computed; “

\infty

” indicates numerical instability in the eigenspectrum estimate (a tiny negative eigenvalue at step 1000), and “–” indicates that the spectrum was not explicitly computed at

n=2048

. The last column reports the relative damping level selected by the adaptive Cholesky procedure.

Figure 7: Quadratic score comparison on four layer matrices from Pythia at checkpoints step 1000 (top) and step 50000 (bottom), corresponding to the one-step analysis in ( 20 ).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\varepsilon$	Total	$s_{\mathrm{out}}$	SYPP	Polynomial $q_{k}(x)$
0.0015	12	0.030717	3	$q_{1}(x)=1.991037-15.856588x+31.760959x^{2}$
			4	$q_{2}(x)=0.102569+0.102569x+7.383161x^{2}+7.383161x^{3}$
			3	$q_{3}(x)=1+2.541910x+2.541910x^{2}$
			2	$q_{4}(x)=1+1.192261x+1.192261x^{2}$
0.0015	13	0.004865	3	$q_{1}(x)=1.991037-15.856588x+31.760959x^{2}$
			3	$q_{2}(x)=1+3.839962x+3.839963x^{2}$

Appendix

Table 5: Polynomial plans with CMAX =32 (the maximum absolute polynomial coefficient), INTERVAL_PAD_REL =0.001, and NOISE_ABS =0.001. Total denotes the total number of symmetric polynomial products (SYPP) used by the plan, and the SYPP column reports the SYPP cost of each polynomial

q_{k}

. The quantity

s_{\mathrm{out}}

is the certified final residual bound, i.e., an upper bound on

\left\|{I}_{n}-{K}_{\gamma}\widetilde{{K}_{\gamma}^{-1}}\right\|_{2}

Figure 8: Baseline configuration (

N=8192

p=0.3

): spectrum of

{H}

(left) and mean absolute scores

s({Q})

for GD, Muon, Newton–Muon, and Newton (right).

Figure 9: More top-uniform curvature (

N=8192

p=2.4

): spectrum of

{H}

(left) and mean absolute scores (right).

Figure 10: Smaller-

N

data (

N=1024

p=0.3

): spectrum of

{H}

(left) and mean absolute scores (right).

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

May 19, 2026

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

Matrix-structured parameters frequently appear in many artificial intelligence models such as large language models. More recently, an efficient Muon optimizer is designed for matrix parameters of large-scale models, and shows markedly faster convergence than the vector-wise algorithms. Although some works have begun to study convergence properties (i.e., optimization error) of the Muon optimizer, its generalization properties (i.e., generalization error) is still not established. Thus, in this paper, we study generalization error of the Muon optimizer based on algorithmic stability and mathematical induction, and prove that the Muon has a generalization error of

O\big(\frac{1}{Nκ^{T}}\big)

, where

N

is training sample size, and

T

denotes iteration number, and

κ>0

denotes minimum difference between singular values of gradient estimate. To enhance generalization of the Muon, we propose an effective mixed Muon (MiMuon) optimizer by cautiously using orthogonalization of gradient, which is a hybrid of Muon and momentum-based SGD optimizers. Then we prove that our MiMuon optimizer has a lower generalization error of

O\big(\frac{1}{N}\big)

than

O\big(\frac{1}{Nκ^{T}}\big)

of Muon optimizer, since

κ

generally is very small. Meanwhile, we also studied the convergence properties of our MiMuon algorithm, and prove that our MiMuon algorithm has the same convergence rate of

O(\frac{1}{T^{1/4}})

as the Muon algorithm. Some numerical experimental results on training large models including Qwen3-0.6B and YOLO26m demonstrate efficiency of the MiMuon optimizer.

Feihu Huang, Yuning Luo, Songcan Chen

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore

Neural Network Generalization Deep Learning Optimization Muon Optimizer Stochastic Gradient Descent

Ask PDF

Dec 4, 2025

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges. However, these methods rely on a costly gradient orthogonalization step. Even efficient iterative approximations such as Newton-Schulz remain expensive, typically requiring dozens of matrix multiplications to converge. We introduce a pre-conditioning procedure that improves the initialization of the Newton--Schulz iterations while incurring negligible overhead. Furthermore, our pre-conditioning reduces the initial polar error and enables the removal of one Newton-Schulz iteration (out of the five iterations usually used in practice). The resulting implementation significantly reduces Muon's overhead. At the end-to-end training level, we observe consistent runtime improvements across speed-run and standard benchmarks, including

\sim

3% reductions in training time on multiple fast training benchmarks, while matching reference performance on both language and vision tasks. Crucially, these improvements require no hyperparameter tuning and can be adopted as a simple drop-in replacement. Beyond empirical gains, we provide theoretical insight into the geometry of the update and its potential robustness against feature collapse. Our code is publicly available on github, in optax and huggingface kernels.

Thibaut Boissin, Thomas Massena, Franck Mamalet +1

IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2

Newton-Schulz Iteration Orthogonal Optimization Neural Network Optimization Muon Optimizer Second-Order Optimization

Ask PDF

May 26, 2026

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term introduces curvature-sensitive corrections while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.

Jiacheng Li, Jianchao Tan, Hongtao Xu +5

Meituan, Beijing, China

Language Model Pretraining Gradient Descent Muon Optimizer Accelerated Gradient Methods

Ask PDF

Abstract

Neural Network Optimization

Explore similar work

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training