The Newton-Muon Optimizer

Zhehang Du; Weijie Su

math.OCApr 1, 2026

The Newton-Muon Optimizer

Authors: Zhehang Du, Weijie Su

Organizations: University of Pennsylvania

Abstract

The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains largely elusive. In this paper, we introduce a surrogate model that not only sheds new light on the design of Muon, but more importantly leads to a new optimizer. In the same spirit as the derivation of Newton's method, the surrogate approximates the loss as a quadratic function of the perturbation to a weight matrix $W$ using only three matrices: the gradient $G$ , an output-space curvature matrix $H$ , and the data matrix $Z$ that stacks the layer inputs. By minimizing this surrogate in one step and adopting a certain isotropic assumption on the weights, we obtain the closed-form update rule (up to momentum and weight decay) $W \leftarrow W - η\cdot \mathrm{msgn}(G(ZZ^\top)^{-1})$ , where $η$ is the learning rate and $\mathrm{msgn}(X)=UV^\top$ if $X=USV^\top$ is a compact singular value decomposition. This new optimization method, which we refer to as Newton-Muon, shows that standard Muon can be interpreted as an implicit Newton-type method that neglects the right preconditioning induced by the input second moment. Empirically, on a reproduction of the earliest publicly released Modded-NanoGPT speedrun configuration using Muon for GPT-2 pretraining, Newton-Muon reaches the target validation loss in 6% fewer iteration steps and reduces wall-clock training time by about 4%.

Figures & tables

Figure 1: Top: short track Record #4 validation loss comparison on the Modded-NanoGPT speedrun benchmark. Record #4 is the earliest publicly released configuration using Muon, and our reproduction on a single H100 GPU is denoted Muon. Newton–Muon adds the activation right-preconditioner before the Newton–Schulz iterations. Newton–Muon reaches the Muon baseline final validation loss in

6\%

fewer steps; despite a

1.8\%

higher per-step cost from right-preconditioning, it reduces wall-clock time to that loss by about

4\%

. Bottom: CIFAR-10 experiments (Appendix C ) on a 32-layer residual MLP show that Newton–Muon outperforms both Muon and AdamW in both per-step efficiency and overall wall-clock time.

Panel of Figure 2 — Figure 2: Left: standard Muon is orthogonally equivariant. Right: Newton–Muon takes the pair $(G,Z)$ as input, and the diagram commutes if the right rotation of $G$ is accompanied by the transformation $Z\mapsto O_{n}^{\top}Z$ .

Figure 3: Numerical study with spiked activation second moment

\mathrm{diag}(\kappa,1,\ldots,1)

and

\kappa=64

. Top: baseline case

(N,p)=(8192,0.3)

. Middle: more uniform curvature

(N,p)=(8192,2.4)

. Bottom: smaller sample size

(N,p)=(1024,0.3)

. The left column shows the spectrum of

{H}

, and the right column shows the corresponding mean absolute scores

s({Q})

Method	Loss	Time (s)
AdamW	3.3801	7228.4
Muon	3.2793	7314.1
Newton–Muon	3.2611	7443.3

Table 1: Short track Record #4 setting (single H100).

Figure 4: Refresh ablation for Newton–Muon on short track Record #4. Grouped bar plot over refresh interval

k

, with one bar per EWMA coefficient

\beta

(ridge scaling fixed at

\gamma=0.2

and learning rate fixed at

0.0040

Figure 5: Left: Ridge-scaling ablation for Newton–Muon on short track Record #4. Bar plot over ridge scaling

\gamma

with

k=32

\beta=0.95

, and learning rate

0.0040

fixed. Right: Learning-rate ablation for Newton–Muon on short track Record #4. Bar plot over learning rate with

k=32

\beta=0.95

, and

\gamma=0.2

fixed.

Method	Run 1	Run 2	Run 3	Run 4	Avg. loss	Avg. time (s)
AdamW	3.4677	3.4631	3.4640	3.4566	3.4628	4272.0
Muon	3.2758	3.2777	3.2783	3.2830	3.2787	4305.9
Newton–Muon	3.2733	3.2736	3.2740	3.2745	3.2739	4342.4

Table 2: Short track Record #28 setting (single L40S).

Figure 6: Validation loss trajectories for the short track Record #28 setting (single L40S). We plot the second-best run for AdamW, Muon, and Newton–Muon, and show loss versus step (left) and loss versus wall-clock time (right).

Method	Run 1	Run 2	Run 3	Avg. loss
Muon	2.9190	2.9208	2.9190	2.9196
Newton–Muon	2.9175	2.9191	2.9181	2.9183

Table 3: Medium track Record #17 setting. Wall-clock time is not reported since we did not observe a substantial improvement.

Module	$n$	$\kappa({Z}{Z}^{\top})$	$d_{\max}/d_{\min}$	$\bar{o}/d_{\mathrm{mean}}$	$\gamma/d_{\mathrm{mean}}$
Attn out	512	$9.18\times 10^{3}$	$2.84\times 10^{1}$	$9.89\times 10^{1}$	$1.0\times 10^{-6}$
Attn QKV	512	$2.04\times 10^{6}$	$3.00\times 10^{0}$	$6.59\times 10^{1}$	$1.0\times 10^{-6}$
MLP expansion	512	$\infty$	$3.01\times 10^{0}$	$6.56\times 10^{1}$	$1.0\times 10^{-4}$
MLP contraction	2048	–	$7.23\times 10^{1}$	$1.88\times 10^{2}$	$1.0\times 10^{-6}$
Attn out	512	$2.34\times 10^{3}$	$3.02\times 10^{1}$	$7.21\times 10^{1}$	$1.0\times 10^{-6}$
Attn QKV	512	$9.77\times 10^{4}$	$1.66\times 10^{2}$	$4.22\times 10^{1}$	$1.0\times 10^{-6}$

Table 4: Diagnostics of the activation second moment

{Z}{Z}^{\top}

for four modules at step 1000 (top block) and step 50000 (bottom block). Here

d_{\min}

d_{\mathrm{mean}}

, and

d_{\max}

are the minimum, mean, and maximum diagonal entries of

{Z}{Z}^{\top}

\bar{o}=(1/n)\sum_{i}\sum_{j\neq i}|({Z}{Z}^{\top})_{ij}|

, and

\kappa({Z}{Z}^{\top})

is the condition number of

{Z}{Z}^{\top}

. The value

\kappa({Z}{Z}^{\top})

is reported when eigenvalues were explicitly computed; “

\infty

” indicates numerical instability in the eigenspectrum estimate (a tiny negative eigenvalue at step 1000), and “–” indicates that the spectrum was not explicitly computed at

n=2048

. The last column reports the relative damping level selected by the adaptive Cholesky procedure.

Figure 7: Quadratic score comparison on four layer matrices from Pythia at checkpoints step 1000 (top) and step 50000 (bottom), corresponding to the one-step analysis in ( 20 ).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\varepsilon$	Total	$s_{\mathrm{out}}$	SYPP	Polynomial $q_{k}(x)$
0.0015	12	0.030717	3	$q_{1}(x)=1.991037-15.856588x+31.760959x^{2}$
			4	$q_{2}(x)=0.102569+0.102569x+7.383161x^{2}+7.383161x^{3}$
			3	$q_{3}(x)=1+2.541910x+2.541910x^{2}$
			2	$q_{4}(x)=1+1.192261x+1.192261x^{2}$
0.0015	13	0.004865	3	$q_{1}(x)=1.991037-15.856588x+31.760959x^{2}$
			3	$q_{2}(x)=1+3.839962x+3.839963x^{2}$

Appendix

Table 5: Polynomial plans with CMAX =32 (the maximum absolute polynomial coefficient), INTERVAL_PAD_REL =0.001, and NOISE_ABS =0.001. Total denotes the total number of symmetric polynomial products (SYPP) used by the plan, and the SYPP column reports the SYPP cost of each polynomial

q_{k}

. The quantity

s_{\mathrm{out}}

is the certified final residual bound, i.e., an upper bound on

\left\|{I}_{n}-{K}_{\gamma}\widetilde{{K}_{\gamma}^{-1}}\right\|_{2}

Figure 8: Baseline configuration (

N=8192

p=0.3

): spectrum of

{H}

(left) and mean absolute scores

s({Q})

for GD, Muon, Newton–Muon, and Newton (right).

Figure 9: More top-uniform curvature (

N=8192

p=2.4

): spectrum of

{H}

(left) and mean absolute scores (right).

Figure 10: Smaller-

N

data (

N=1024

p=0.3

): spectrum of

{H}

(left) and mean absolute scores (right).

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenNeural Network Generalization Deep Learning Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore
Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz Iteration Orthogonal Optimization
IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
May 26, 2026Jiacheng Li, Jianchao Tan, Hongtao Xu +5Language Model Pretraining Gradient Descent
Meituan, Beijing, China

May 19, 2026J/K move · Enter open · S save

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

Feihu Huang, Yuning Luo, Songcan Chen

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore

Neural Network Generalization Deep Learning Optimization Muon Optimizer Stochastic Gradient Descent

Open paper AskPDF arXiv

Matrix-structured parameters frequently appear in many artificial intelligence models such as large language models. More recently, an efficient Muon optimizer is designed for matrix parameters of large-scale models, and shows markedly faster convergence than the vector-wise algorithms. Although some works have begun to study convergence properties (i.e., optimization error) of the Muon optimizer, its generalization properties (i.e., generalization error) is still not established. Thus, in this paper, we study generalization error of the Muon optimizer based on algorithmic stability and mathematical induction, and prove that the Muon has a generalization error of

O\big(\frac{1}{Nκ^{T}}\big)

, where

N

is training sample size, and

T

denotes iteration number, and

κ>0

denotes minimum difference between singular values of gradient estimate. To enhance generalization of the Muon, we propose an effective mixed Muon (MiMuon) optimizer by cautiously using orthogonalization of gradient, which is a hybrid of Muon and momentum-based SGD optimizers. Then we prove that our MiMuon optimizer has a lower generalization error of

O\big(\frac{1}{N}\big)

than

O\big(\frac{1}{Nκ^{T}}\big)

of Muon optimizer, since

κ

generally is very small. Meanwhile, we also studied the convergence properties of our MiMuon algorithm, and prove that our MiMuon algorithm has the same convergence rate of

O(\frac{1}{T^{1/4}})

as the Muon algorithm. Some numerical experimental results on training large models including Qwen3-0.6B and YOLO26m demonstrate efficiency of the MiMuon optimizer.

Abstract

Neural Network Optimization

Explore similar work

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training