Does Muon Need Fine-Grained Spectral Shaping?

Meher Chaitanya; Tianyi Zhou; Aristides Gionis

cs.AIOct 5, 2026

Does Muon Need Fine-Grained Spectral Shaping?

Authors: Meher Chaitanya, Tianyi Zhou, Aristides Gionis

Organizations: KTH Royal Institute of Technology Stockholm, Sweden

Abstract

Muon combines current and past gradients into matrix momentum. For $M=UΣV^\top$ , the idealized polar update $Q=UV^\top$ gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately $94$ -- $97\%$ of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by $0.022\%$ of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of $0.073$ -- $0.147\%$ . These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.

Figures & tables

Figure 1: Held-out loss versus training tokens on Wiki-zh, one column per model scale. Top: the full run, with the pre-adaptation loss

\ell_{0}

dotted and the zoomed interval shaded. Bottom: the shaded interval at a finer vertical scale. The BB-Static curve uses

p=1.5\%

, the leading fraction of Spectra , so the two can be compared directly. Appendix G.3 shows the other corpora.

Model	Method	FineMath-4	FineMath-3	CodeParrot	Wiki-de	Wiki-zh	Wiki-ar
14M	AdamW	$3.102{\scriptstyle\,\pm\,0.003}$	$2.650{\scriptstyle\,\pm\,0.002}$	$1.929{\scriptstyle\,\pm\,0.003}$	$3.324{\scriptstyle\,\pm\,0.001}$	$2.801{\scriptstyle\,\pm\,0.001}$	$2.055{\scriptstyle\,\pm\,0.001}$
	Muon	$3.086{\scriptstyle\,\pm\,0.002}$	$2.640{\scriptstyle\,\pm\,0.002}$	$1.902{\scriptstyle\,\pm\,0.004}$	$3.283{\scriptstyle\,\pm\,0.001}$	$2.745{\scriptstyle\,\pm\,0.001}$	$2.006{\scriptstyle\,\pm\,0.000}$
	Freon	$\mathbf{3.083}{\scriptstyle\,\pm\,0.002}$	$\mathbf{2.638}{\scriptstyle\,\pm\,0.002}$	$\underline{1.899}{\scriptstyle\,\pm\,0.004}$	$3.279{\scriptstyle\,\pm\,0.001}$	$2.742{\scriptstyle\,\pm\,0.001}$	$2.008{\scriptstyle\,\pm\,0.001}$
	DeVA	$3.110{\scriptstyle\,\pm\,0.004}$	$2.661{\scriptstyle\,\pm\,0.002}$	$1.917{\scriptstyle\,\pm\,0.005}$	$3.286{\scriptstyle\,\pm\,0.001}$	$2.753{\scriptstyle\,\pm\,0.001}$	$2.013{\scriptstyle\,\pm\,0.001}$
	Spectra	$3.138{\scriptstyle\,\pm\,0.004}$	$2.682{\scriptstyle\,\pm\,0.002}$	$1.955{\scriptstyle\,\pm\,0.004}$	$3.328{\scriptstyle\,\pm\,0.003}$	$2.811{\scriptstyle\,\pm\,0.001}$	$2.058{\scriptstyle\,\pm\,0.002}$
	BB-Static †	$\mathbf{3.083}{\scriptstyle\,\pm\,0.002}$	$\mathbf{2.638}{\scriptstyle\,\pm\,0.002}$	$\mathbf{1.898}{\scriptstyle\,\pm\,0.003}$	$\underline{3.278}{\scriptstyle\,\pm\,0.001}$	$\underline{2.738}{\scriptstyle\,\pm\,0.001}$	$\mathbf{1.998}{\scriptstyle\,\pm\,0.001}$

Table 1: Final validation loss after about 100M tokens, reported as mean

\pm

standard deviation over three seeds. All runs use the shared learning rate of their scale. For each model–corpus setting, BB-Static † reports the fraction with the lowest mean loss among the five in Table 9 . Bold and Underline mark the lowest and second-lowest means, respectively in each setting, including ties. Appendix G.1 reports the pre-adaptation losses

\ell_{0}

	vs. Muon		best of all		$\Delta$ vs. Muon	tokens saved (%)
Method	wins	tied	wins	tied	(pp of $\ell_{0}$ )	mean	median
Muon (reference)	–	–	4	1	–	–	–
AdamW	0	0	0	0	$+1.299\pm 0.010$	–	–
Freon	19	1	$\underline{5}$	3	$-0.022\pm 0.010$	$7.78$	$7.71$
DeVA	0	0	0	0	$+0.811\pm 0.015$	–	–
Spectra	0	0	0	0	$+2.321\pm 0.047$	–	–

Table 2: Summary over the 30 model–corpus settings after about 100M tokens. BB-Static † uses the fraction selection of Table 1 . Counts compare three-seed mean final losses. vs. Muon : settings with lower ( wins ) or equal ( tied ) loss than Muon . Best of all : settings with the lowest loss of all rows, alone ( wins ) or shared ( tied ). BB-Static (5%) can at best tie BB-Static † , which includes its runs.

\Delta

vs. Muon : the loss difference from Muon as a percentage of

\ell_{0}

, averaged over settings within each seed. We report the mean

\pm

std dev of the three seed averages. Negative values favor the method. Tokens saved : the mean and median share of the 100M-token budget saved in reaching Muon ’s final loss, over the winning settings and the seeds that end below Muon (Table 10 ).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Tokens (M)	Bulk (%)	$\Gamma$ ( $\times$ )	$S_{\mathrm{bu}}$ (%)	$S_{\mathrm{bu}}^{\mathrm{abs}}$ (%)
FineMath-4	0.0	95.3	66.4	22.6	23.5
	25.0	97.1	85.1	25.8	28.0
	50.0	96.6	92.3	19.7	23.6
	99.9	96.2	33.2	18.6	43.5
CodeParrot	0.0	95.5	74.8	22.6	22.3
	25.0	96.5	72.0	26.0	27.8

Appendix

Table 3: Raw-gradient-frame band content. Each row pools 24 matrices and 12,288 singular directions from a Pythia-70M Muon trajectory. Bulk (%) is the pooled fraction of directions whose singular values lie at or below the raw-gradient noise edge

\tau^{G}_{t,\ell}

;

\Gamma

is the ratio of mean absolute per-direction alignment in spikes to bulk.

S_{\mathrm{bu}}

and

S_{\mathrm{bu}}^{\mathrm{abs}}

are the signed and absolute shares in equation 32 , computed from pooled sums. Every analyzed matrix has positive signed bulk alignment. Token labels are rounded.

Corpus	Tokens (M)	Bulk (%)	$\Gamma$ ( $\times$ )	$S_{\mathrm{bu}}$ (%)	$S_{\mathrm{bu}}^{\mathrm{abs}}$ (%)
FineMath-4	0.0	95.3	66.4	22.6	23.5
	25.0	97.1	56.8	24.7	36.8
	50.0	96.8	78.1	10.0	28.0
	99.9	96.6	18.9	63.5	59.8
CodeParrot	0.0	95.5	74.8	22.6	22.3
	25.0	96.5	60.7	19.4	31.3

Appendix

Table 4: Nesterov-frame band content. The trajectories, checkpoints, and reference gradients are paired with Table 3 , but the singular frame and noise edge

\tau^{M}_{t,\ell}

are those of Muon ’s Nesterov input

M_{t,\ell}

. Each row pools 24 matrices and 12,288 directions, using the same statistics as Table 3 . Matched scalar rescaling explains the agreement at 0.0M. Every analyzed matrix has positive signed bulk alignment.

Corpus	Frame	$A_{\mathrm{sp}}$	$A_{\mathrm{bu}}$	$A_{\mathrm{all}}$	$B_{\mathrm{all}}$	$C$
FineMath-4	Gradient	0.5946	0.1363	0.7308	1.2160	0.601
	Nesterov	0.0749	0.1300	0.2049	0.8594	0.238
CodeParrot	Gradient	1.0155	0.2028	1.2183	1.6182	0.753
	Nesterov	0.1132	0.0957	0.2088	0.8246	0.253
Wiki-ar	Gradient	0.9131	0.2013	1.1144	1.3890	0.802
	Nesterov	0.4201	0.1422	0.5623	0.9577	0.587

Appendix

Table 5: Signed contributions and cancellation at 99.9M tokens.

A_{\mathrm{sp}}

and

A_{\mathrm{bu}}

are pooled signed alignments;

A_{\mathrm{all}}

is their sum and

B_{\mathrm{all}}

sums absolute alignments.

C=|A_{\mathrm{all}}|/B_{\mathrm{all}}

measures the fraction remaining after signed cancellation. Every row pools 24 matrices, and every matrix has positive signed bulk alignment. Displayed sums are rounded independently.

Corpus	Frame	$\overline{\alpha}_{\mathrm{bu}}/\overline{\alpha}_{\mathrm{sp}}$	$R_{\mathrm{mat}}(2)$	$R_{\mathrm{mat}}(4)$
FineMath-4	Gradient	0.009	0.601	0.397
	Nesterov	0.062	0.828	0.738
CodeParrot	Gradient	0.007	0.590	0.381
	Nesterov	0.032	0.739	0.604
Wiki-ar	Gradient	0.012	0.602	0.395
	Nesterov	0.019	0.638	0.450

Appendix

Table 6: Pre-update reference-alignment trade-off at 99.9M tokens. The per-direction ratio uses pooled signed means

\overline{\alpha}_{b}=A_{b}/N_{b}

; the ratio

\Gamma

in Appendix D uses absolute alignments.

R_{\mathrm{mat}}(\rho)

applies each matrix’s own norm-matching factor before summing, as in equation 36 . Values below one indicate lower alignment than the flat profile in the same frame. Each row uses the same 24 matrices and reference gradient as the preceding diagnostics, without learning-rate or shape multipliers.

		Loss reduction $\widehat{B}_{\epsilon}$		Slope $\widehat{S}_{\epsilon}$
		( $10^{-4}$ )		( $10^{-3}$ )
Corpus	Tokens (M)	$\epsilon=0.25$	$\epsilon=0.5$	$\epsilon=0.25$	$\epsilon=0.5$
FineMath-4	25.0	+2.524	+3.725	+1.709	+2.774
CodeParrot	25.0	+1.842	+2.574	+1.350	+2.320
Wiki-ar	25.0	+6.593	+10.226	+4.082	+5.944
FineMath-4	50.0	-0.604	-1.191	-0.177	+0.032

Appendix

Table 7: Single-update validation-loss improvement (

K=0

). Both

\rho=1.25

and

\rho=1.5

lower post-update validation loss relative to the exact flat update

\rho=1

in five of six tested states. Each row is a Pythia-70M corpus–checkpoint state with a common gain across 24 hidden matrices and separate per-matrix Frobenius norm matching.

\widehat{B}_{\epsilon}

measures the loss reduction relative to

\rho=1

, whereas

\widehat{S}_{\epsilon}

compares the bulk-emphasizing and bulk-suppressing branches through equation 37 . Loss reductions and raw central finite-difference slopes are reported in units of

10^{-4}

and

10^{-3}

, respectively. All six states are shown; token labels are rounded, and the duplicate

\rho=1

branches have identical recorded losses.

Corpus	$\ell_{0}$	$\Delta$	$\ell_{0}$	$\Delta$	$\ell_{0}$	$\Delta$	$\ell_{0}$	$\Delta$	$\ell_{0}$	$\Delta$
	14M		31M		70M		160M		410M
Wiki-de	4.530	$-1.247{\scriptstyle\,\pm\,0.001}$	4.009	$-1.050{\scriptstyle\,\pm\,0.002}$	3.699	$-1.019{\scriptstyle\,\pm\,0.003}$	3.171	$-0.719{\scriptstyle\,\pm\,0.002}$	2.626	$-0.445{\scriptstyle\,\pm\,0.001}$
Wiki-zh	3.715	$-0.970{\scriptstyle\,\pm\,0.001}$	3.465	$-0.994{\scriptstyle\,\pm\,0.001}$	3.329	$-1.118{\scriptstyle\,\pm\,0.005}$	3.090	$-1.035{\scriptstyle\,\pm\,0.001}$	2.570	$-0.691{\scriptstyle\,\pm\,0.002}$
FineMath-4	3.572	$-0.486{\scriptstyle\,\pm\,0.002}$	3.278	$-0.502{\scriptstyle\,\pm\,0.001}$	3.088	$-0.537{\scriptstyle\,\pm\,0.002}$	2.753	$-0.481{\scriptstyle\,\pm\,0.003}$	2.421	$-0.375{\scriptstyle\,\pm\,0.005}$
Wiki-ar	2.936	$-0.930{\scriptstyle\,\pm\,0.000}$	2.761	$-0.906{\scriptstyle\,\pm\,0.001}$	2.747	$-1.045{\scriptstyle\,\pm\,0.000}$	2.620	$-1.012{\scriptstyle\,\pm\,0.000}$	2.150	$-0.698{\scriptstyle\,\pm\,0.001}$
FineMath-3	2.930	$-0.290{\scriptstyle\,\pm\,0.002}$	2.692	$-0.261{\scriptstyle\,\pm\,0.002}$	2.566	$-0.269{\scriptstyle\,\pm\,0.004}$	2.320	$-0.254{\scriptstyle\,\pm\,0.002}$	2.050	$-0.159{\scriptstyle\,\pm\,0.002}$

Appendix

Table 8: Corpus adaptation across five model scales.

\ell_{0}

is the held-out loss before continued pretraining.

\Delta=\ell_{\mathrm{final}}-\ell_{0}

is the change after about 100M tokens with Muon , reported as mean

\pm

standard deviation over three seeds. Negative values indicate improvement.

\ell_{0}

has no spread, because all seeds start from the same checkpoint and use the same held-out windows. Rows are ordered by decreasing

\ell_{0}

, which gives the same order at every scale. Corpora are grouped into mathematics, code, and multilingual Wikipedia.

Model	Method	FineMath-4	FineMath-3	CodeParrot	Wiki-de	Wiki-zh	Wiki-ar
14M	BB-Static (1.5%)	$3.088{\scriptstyle\,\pm\,0.003}$	$2.642{\scriptstyle\,\pm\,0.002}$	$1.903{\scriptstyle\,\pm\,0.003}$	$\mathbf{3.278}{\scriptstyle\,\pm\,0.001}$	$2.741{\scriptstyle\,\pm\,0.000}$	$1.999{\scriptstyle\,\pm\,0.001}$
	BB-Static (2%)	$3.086{\scriptstyle\,\pm\,0.002}$	$2.639{\scriptstyle\,\pm\,0.001}$	$1.899{\scriptstyle\,\pm\,0.003}$	$3.279{\scriptstyle\,\pm\,0.000}$	$2.739{\scriptstyle\,\pm\,0.001}$	$\mathbf{1.998}{\scriptstyle\,\pm\,0.001}$
	BB-Static (5%)	$3.085{\scriptstyle\,\pm\,0.003}$	$2.639{\scriptstyle\,\pm\,0.002}$	$1.900{\scriptstyle\,\pm\,0.003}$	$\mathbf{3.278}{\scriptstyle\,\pm\,0.001}$	$\mathbf{2.738}{\scriptstyle\,\pm\,0.001}$	$2.000{\scriptstyle\,\pm\,0.000}$
	BB-Static (15%)	$\mathbf{3.083}{\scriptstyle\,\pm\,0.002}$	$2.639{\scriptstyle\,\pm\,0.002}$	$\mathbf{1.898}{\scriptstyle\,\pm\,0.003}$	$3.279{\scriptstyle\,\pm\,0.001}$	$2.742{\scriptstyle\,\pm\,0.001}$	$2.006{\scriptstyle\,\pm\,0.001}$
	BB-Static (20%)	$\mathbf{3.083}{\scriptstyle\,\pm\,0.002}$	$\mathbf{2.638}{\scriptstyle\,\pm\,0.002}$	$1.899{\scriptstyle\,\pm\,0.004}$	$3.282{\scriptstyle\,\pm\,0.002}$	$2.745{\scriptstyle\,\pm\,0.001}$	$2.009{\scriptstyle\,\pm\,0.001}$
31M	BB-Static (1.5%)	$2.776{\scriptstyle\,\pm\,0.001}$	$2.433{\scriptstyle\,\pm\,0.002}$	$1.650{\scriptstyle\,\pm\,0.002}$	$2.956{\scriptstyle\,\pm\,0.001}$	$\mathbf{2.463}{\scriptstyle\,\pm\,0.001}$	$\mathbf{1.848}{\scriptstyle\,\pm\,0.001}$

Appendix

Table 9: Fixed-fraction sweep for BB-Static : final validation loss after about 100M training tokens, reported as mean

\pm

standard deviation over three seeds. For

p\in\{0.015,0.02,0.05,0.15,0.20\}

, the leading band contains

r_{t}=\lceil pd\rceil

singular directions, where

d=\min\{m,n\}

and

p

is held fixed within each run. All fractions use

\rho=4

and the shared learning rate of their scale, with the same training budget, schedule, and seeds as Table 1 . For each model–corpus setting, the fraction with the lowest three-seed mean gives the BB-Static † entry in Table 1 . Table 2 summarizes the shared configuration

p=5\%

, and Figure 1 shows

p=1.5\%

. Bold marks the lowest mean among the fractions in each setting, including ties at the printed precision.

Arm	Model	FineMath-4	FineMath-3	CodeParrot	Wiki-de	Wiki-zh	Wiki-ar
Freon	14M	$8.3{\scriptstyle\,\pm\,0.7}$	$3.5{\scriptstyle\,\pm\,2.9}$	$7.1{\scriptstyle\,\pm\,1.6}$	$4.9{\scriptstyle\,\pm\,2.6}$	$4.4{\scriptstyle\,\pm\,2.6}$	–
	31M	$7.8{\scriptstyle\,\pm\,1.8}$	$8.7{\scriptstyle\,\pm\,3.5}$	$3.3{\scriptstyle\,\pm\,2.9}$	$4.4{\scriptstyle\,\pm\,1.7}$	$4.7{\scriptstyle\,\pm\,0.7}$	–
	70M	$11.1{\scriptstyle\,\pm\,1.4}$	$10.6{\scriptstyle\,\pm\,3.2}$	$12.5{\scriptstyle\,\pm\,0.7}$	$9.8{\scriptstyle\,\pm\,2.6}$	$10.9{\scriptstyle\,\pm\,3.3}$	$10.2{\scriptstyle\,\pm\,1.5}$
	160M	–	–	–	–	–	–
	410M	$9.9{\scriptstyle\,\pm\,3.1}$	$9.6{\scriptstyle\,\pm\,3.5}$	–	$6.0{\scriptstyle\,\pm\,5.1}$	–	–
BB-Static (5%)	14M	$4.6{\scriptstyle\,\pm\,0.4}$	$3.4{\scriptstyle\,\pm\,2.9}$	$6.1{\scriptstyle\,\pm\,2.1}$	$6.3{\scriptstyle\,\pm\,2.1}$	$10.5{\scriptstyle\,\pm\,2.4}$	$12.3{\scriptstyle\,\pm\,2.2}$

Appendix

Table 10: Token savings against Muon , as a percentage of the 100M-token budget, for each method and model–corpus setting. Muon is the reference, so it has no block of its own. A cell is filled only where the method’s three-seed mean final loss is below Muon ’s in Table 1 , and “–” marks settings where Muon is at least as good. A filled cell reports

1-D^{\star}/100\mathrm{M}

, where

D^{\star}

is the number of tokens at which the method’s curve first reaches Muon ’s 100M-token loss in the same seed. Values are mean

\pm

standard deviation over the seeds in which the method ends below Muon . A superscript

k/3

marks a setting won in only

k

of the three seeds. DeVA and Spectra never beat Muon , so they are omitted. The last block, BB-Static † , uses the best of the five fractions in each setting (Table 9 ). Because this fraction is chosen on the same runs, its savings are optimistic. Separately, intermediate points of a cosine schedule are evaluated before the learning rate has fully decayed.

Figure 2: Learning-rate sweeps on FineMath-4 with a 20M-token budget and one seed. Curves show final validation loss against multiples of each scale’s reference rate. Grey bands mark each method’s best multiplier. BB-Static uses

p=20\%

in the sweep. Table 11 lists the full results.

Model	Method	0.01 $\times$	0.05 $\times$	0.1 $\times$	0.25 $\times$	0.5 $\times$	1 $\times$	2 $\times$	4 $\times$	best
14M	Muon	3.335	3.232	3.209	3.227	3.281	3.354	3.459	3.581	0.1 $\times$
	Freon	3.338	3.231	3.207	3.219	3.265	3.335	3.427	3.540	0.1 $\times$
	DeVA	3.266	3.235	3.226	3.232	3.267	3.328	3.421	3.615	0.1 $\times$
	Spectra	3.272	3.257	3.250	3.259	3.304	3.400	3.555	4.458	0.1 $\times$
	BB-Static (20%)	3.340	3.232	3.207	3.217	3.259	3.332	3.422	3.543	0.1 $\times$
	BB-Adaptive	3.336	3.230	3.207	3.223	3.273	3.347	3.448	3.572	0.1 $\times$

Appendix

Table 11: Final held-out loss on FineMath-4 after 20M tokens with one seed, at multiples of each scale’s reference rate. Bold marks each method’s lowest loss, and underline marks its second lowest. BB-Static uses

p=20\%

. The 100M-token runs use one shared multiplier per scale, the best for most methods:

0.1

0.1

0.25

0.1

, and

0.05

from 14M to 410M.

Model	Corpus	$\rho{=}2$	$\rho{=}4$	$\rho{=}6$	$\rho{=}8$	$\rho{=}10$	best	range
14M	FineMath-4	3.209	3.209	3.209	3.209	3.210	6	0.000
	FineMath-3	2.737	2.737	2.736	2.737	2.735	10	0.002
	CodeParrot	2.046	2.046	2.048	2.048	2.048	4	0.002
	Wiki-de	3.518	3.522	3.523	3.526	3.528	2	0.010
	Wiki-zh	2.960	2.962	2.964	2.965	2.967	2	0.007
	Wiki-ar	2.184	2.188	2.190	2.193	2.194	2	0.011

Appendix

Table 12:

\rho

tuning across all 30 model–corpus pairs, measured by final held-out validation loss after 20M tokens. The best loss in each row is bold , and the second-best is underlined . The range column is the difference between the largest and smallest loss, computed before rounding; its median is

0.009

. At each model scale, the sweep uses the same peak learning rate as the main 100M-token BB-Adaptive runs (Table 11 ), held fixed across

\rho

values and corpora. The sweep uses a different seed from the learning-rate sweep, so its

\rho=4

entries on FineMath-4 differ from the BB-Adaptive entries of Table 11 by at most

0.008

Model	matrices	median $\rho_{\mathrm{eff}}$	p5–p95	$\rho_{\mathrm{eff}}-\rho$	Ritz residual
Pythia-14M	144	3.980	3.88–4.07	$-0.020$	4.1 $\times 10^{-7}$
Pythia-31M	144	3.980	3.91–4.04	$-0.020$	4.7 $\times 10^{-7}$
Pythia-70M	144	4.005	3.95–4.06	$+0.005$	6.2 $\times 10^{-7}$
Pythia-160M	288	3.963	3.92–4.05	$-0.037$	1.5 $\times 10^{-6}$
Pythia-410M	576	3.965	3.93–4.02	$-0.035$	8.2 $\times 10^{-6}$

Appendix

Table 13: Numerical diagnostics for BB-Adaptive at the target gain

\rho=4

, summarized by model scale. “matrices” counts the instrumented two-dimensional weight matrices across the six corpora. We report the median effective gain

\rho_{\mathrm{eff}}

, its 5th–95th percentile interval, and the difference between the median and the target gain. The final column reports the median Rayleigh–Ritz residual of the retained spike subspaces.

Arm	14M	31M	70M	160M	410M
Muon (ms/step)	$173{\scriptstyle\,\pm\,5}$	$308{\scriptstyle\,\pm\,1}$	$617{\scriptstyle\,\pm\,0}$	$635{\scriptstyle\,\pm\,1}$	$1748{\scriptstyle\,\pm\,10}$
AdamW	$0.93{\scriptstyle\,\pm\,0.05}\times$	$0.92{\scriptstyle\,\pm\,0.01}\times$	$0.98{\scriptstyle\,\pm\,0.01}\times$	$0.98{\scriptstyle\,\pm\,0.00}\times$	$0.98{\scriptstyle\,\pm\,0.01}\times$
Freon	$1.17{\scriptstyle\,\pm\,0.01}\times$	$1.10{\scriptstyle\,\pm\,0.00}\times$	$1.06{\scriptstyle\,\pm\,0.00}\times$	$1.17{\scriptstyle\,\pm\,0.00}\times$	$1.20{\scriptstyle\,\pm\,0.00}\times$
DeVA	$1.13{\scriptstyle\,\pm\,0.00}\times$	$1.10{\scriptstyle\,\pm\,0.00}\times$	$1.11{\scriptstyle\,\pm\,0.00}\times$	$1.27{\scriptstyle\,\pm\,0.00}\times$	$1.34{\scriptstyle\,\pm\,0.00}\times$
Spectra	$0.98{\scriptstyle\,\pm\,0.01}\times$	$1.00{\scriptstyle\,\pm\,0.00}\times$	$1.00{\scriptstyle\,\pm\,0.00}\times$	$1.00{\scriptstyle\,\pm\,0.00}\times$	$1.00{\scriptstyle\,\pm\,0.00}\times$
BB-Static (1.5%)	$1.59{\scriptstyle\,\pm\,0.05}\times$	$1.32{\scriptstyle\,\pm\,0.01}\times$	$1.19{\scriptstyle\,\pm\,0.00}\times$	$1.27{\scriptstyle\,\pm\,0.00}\times$	$1.27{\scriptstyle\,\pm\,0.01}\times$

Appendix

Table 14: Per-step wall-clock time at each model scale. For each run, the per-step time is its total wall-clock time divided by its number of steps. Entries are mean

\pm

standard deviation over the runs at that scale, across six corpora and three seeds. The Muon row reports milliseconds per step; all other rows report runtime relative to Muon at the same scale. Fixed-rank timings cover

p=1.5\%

and

5\%

. Measurements were collected on shared cluster nodes and are subject to scheduling variability. The ratios within a scale are stable.

Model	$d_{\mathrm{model}}$	micro-batch $\times$ accum	steps	tokens	Muon ms/step
Pythia-14M	128	32 $\times$ 1	3051	100M	$173{\scriptstyle\,\pm\,5}$
Pythia-31M	256	32 $\times$ 1	3051	100M	$308{\scriptstyle\,\pm\,1}$
Pythia-70M	512	32 $\times$ 1	3051	100M	$617{\scriptstyle\,\pm\,0}$
Pythia-160M	768	32 $\times$ 1	3051	100M	$635{\scriptstyle\,\pm\,1}$
Pythia-410M	1024	32 $\times$ 1	3051	100M	$1748{\scriptstyle\,\pm\,10}$

Appendix

Table 15: Pythia model configurations and training budgets.

d_{\mathrm{model}}

denotes the hidden width, and “micro-batch

\times

accum” gives the number of sequences per microbatch and the number of gradient accumulation steps. All scales use an effective batch of 32 sequences of length 1024, giving 32,768 tokens per step and approximately 100M tokens over 3,051 steps. The Muon timing entries reproduce those in Table 14 : mean

\pm

standard deviation over six corpora and three seeds at each scale.

Figure 3: Final held-out validation loss as a function of model size, with one panel per corpus and all seven optimization methods shown. Across all corpora, final loss decreases consistently as model size increases, matching the expected scaling trend. The rate of decrease differs by corpus, reflecting differences in adaptation difficulty, but the overall monotonic improvement with scale remains stable. The BB-Static curve uses

p=20\%

(Table 9 ); the other curves match Table 1 .

Figure 4: Held-out loss vs. tokens on FineMath-4, one column per scale. Top: the full run, with the pre-adaptation loss

\ell_{0}