cs.AIOct 5, 2026

Does Muon Need Fine-Grained Spectral Shaping?

Authors: Meher Chaitanya, Tianyi Zhou, Aristides Gionis

Organizations: KTH Royal Institute of Technology Stockholm, Sweden

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Muon combines current and past gradients into matrix momentum. For M=UΣV⊤M=UΣV^\top, the idealized polar update Q=UV⊤Q=UV^\top gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately 9494--97%97\% of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by 0.022%0.022\% of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of 0.0730.073--0.147%0.147\%. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.

Figures & tables

Figure 1: Held-out loss versus training tokens on Wiki-zh, one column per model scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoomed interval shaded. Bottom: the shaded interval at a finer vertical scale. The BB-Static curve uses p=1.5%p=1.5\% , the leading fraction of Spectra , so the two can be compared directly. Appendix G.3 shows the other corpora.
ModelMethodFineMath-4FineMath-3CodeParrotWiki-deWiki-zhWiki-ar
14MAdamW3.102±0.0033.102{\scriptstyle\,\pm\,0.003}2.650±0.0022.650{\scriptstyle\,\pm\,0.002}1.929±0.0031.929{\scriptstyle\,\pm\,0.003}3.324±0.0013.324{\scriptstyle\,\pm\,0.001}2.801±0.0012.801{\scriptstyle\,\pm\,0.001}2.055±0.0012.055{\scriptstyle\,\pm\,0.001}
Muon3.086±0.0023.086{\scriptstyle\,\pm\,0.002}2.640±0.0022.640{\scriptstyle\,\pm\,0.002}1.902±0.0041.902{\scriptstyle\,\pm\,0.004}3.283±0.0013.283{\scriptstyle\,\pm\,0.001}2.745±0.0012.745{\scriptstyle\,\pm\,0.001}2.006±0.0002.006{\scriptstyle\,\pm\,0.000}
Freon3.083±0.002\mathbf{3.083}{\scriptstyle\,\pm\,0.002}2.638±0.002\mathbf{2.638}{\scriptstyle\,\pm\,0.002}1.899‾±0.004\underline{1.899}{\scriptstyle\,\pm\,0.004}3.279±0.0013.279{\scriptstyle\,\pm\,0.001}2.742±0.0012.742{\scriptstyle\,\pm\,0.001}2.008±0.0012.008{\scriptstyle\,\pm\,0.001}
DeVA3.110±0.0043.110{\scriptstyle\,\pm\,0.004}2.661±0.0022.661{\scriptstyle\,\pm\,0.002}1.917±0.0051.917{\scriptstyle\,\pm\,0.005}3.286±0.0013.286{\scriptstyle\,\pm\,0.001}2.753±0.0012.753{\scriptstyle\,\pm\,0.001}2.013±0.0012.013{\scriptstyle\,\pm\,0.001}
Spectra3.138±0.0043.138{\scriptstyle\,\pm\,0.004}2.682±0.0022.682{\scriptstyle\,\pm\,0.002}1.955±0.0041.955{\scriptstyle\,\pm\,0.004}3.328±0.0033.328{\scriptstyle\,\pm\,0.003}2.811±0.0012.811{\scriptstyle\,\pm\,0.001}2.058±0.0022.058{\scriptstyle\,\pm\,0.002}
BB-Static †3.083±0.002\mathbf{3.083}{\scriptstyle\,\pm\,0.002}2.638±0.002\mathbf{2.638}{\scriptstyle\,\pm\,0.002}1.898±0.003\mathbf{1.898}{\scriptstyle\,\pm\,0.003}3.278‾±0.001\underline{3.278}{\scriptstyle\,\pm\,0.001}2.738‾±0.001\underline{2.738}{\scriptstyle\,\pm\,0.001}1.998±0.001\mathbf{1.998}{\scriptstyle\,\pm\,0.001}
Table 1: Final validation loss after about 100M tokens, reported as mean ±\pm standard deviation over three seeds. All runs use the shared learning rate of their scale. For each model–corpus setting, BB-Static † reports the fraction with the lowest mean loss among the five in Table 9 . Bold and Underline mark the lowest and second-lowest means, respectively in each setting, including ties. Appendix G.1 reports the pre-adaptation losses ℓ0\ell_{0} .
 vs. Muonbest of allΔ\Delta vs. Muontokens saved (%)
Methodwinstiedwinstied(pp of ℓ0\ell_{0} )meanmedian
Muon (reference)––41–––
AdamW0000+1.299±0.010+1.299\pm 0.010––
Freon1915‾\underline{5}3−0.022±0.010-0.022\pm 0.0107.787.787.717.71
DeVA0000+0.811±0.015+0.811\pm 0.015––
Spectra0000+2.321±0.047+2.321\pm 0.047––
Table 2: Summary over the 30 model–corpus settings after about 100M tokens. BB-Static † uses the fraction selection of Table 1 . Counts compare three-seed mean final losses. vs. Muon : settings with lower ( wins ) or equal ( tied ) loss than Muon . Best of all : settings with the lowest loss of all rows, alone ( wins ) or shared ( tied ). BB-Static (5%) can at best tie BB-Static † , which includes its runs. Δ\Delta vs. Muon : the loss difference from Muon as a percentage of ℓ0\ell_{0} , averaged over settings within each seed. We report the mean ±\pm std dev of the three seed averages. Negative values favor the method. Tokens saved : the mean and median share of the 100M-token budget saved in reaching Muon ’s final loss, over the winning settings and the seeds that end below Muon (Table 10 ).
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

CorpusTokens (M)Bulk (%)Γ\Gamma ( ×\times )SbuS_{\mathrm{bu}} (%)SbuabsS_{\mathrm{bu}}^{\mathrm{abs}} (%)
FineMath-40.095.366.422.623.5
 25.097.185.125.828.0
 50.096.692.319.723.6
 99.996.233.218.643.5
CodeParrot0.095.574.822.622.3
 25.096.572.026.027.8
Appendix
Table 3: Raw-gradient-frame band content. Each row pools 24 matrices and 12,288 singular directions from a Pythia-70M Muon trajectory. Bulk (%) is the pooled fraction of directions whose singular values lie at or below the raw-gradient noise edge τt,ℓG\tau^{G}_{t,\ell} ; Γ\Gamma is the ratio of mean absolute per-direction alignment in spikes to bulk. SbuS_{\mathrm{bu}} and SbuabsS_{\mathrm{bu}}^{\mathrm{abs}} are the signed and absolute shares in equation 32 , computed from pooled sums. Every analyzed matrix has positive signed bulk alignment. Token labels are rounded.
CorpusTokens (M)Bulk (%)Γ\Gamma ( ×\times )SbuS_{\mathrm{bu}} (%)SbuabsS_{\mathrm{bu}}^{\mathrm{abs}} (%)
FineMath-40.095.366.422.623.5
 25.097.156.824.736.8
 50.096.878.110.028.0
 99.996.618.963.559.8
CodeParrot0.095.574.822.622.3
 25.096.560.719.431.3
Appendix
Table 4: Nesterov-frame band content. The trajectories, checkpoints, and reference gradients are paired with Table 3 , but the singular frame and noise edge τt,ℓM\tau^{M}_{t,\ell} are those of Muon ’s Nesterov input Mt,ℓM_{t,\ell} . Each row pools 24 matrices and 12,288 directions, using the same statistics as Table 3 . Matched scalar rescaling explains the agreement at 0.0M. Every analyzed matrix has positive signed bulk alignment.
CorpusFrameAspA_{\mathrm{sp}}AbuA_{\mathrm{bu}}AallA_{\mathrm{all}}BallB_{\mathrm{all}}CC
FineMath-4Gradient0.59460.13630.73081.21600.601
 Nesterov0.07490.13000.20490.85940.238
CodeParrotGradient1.01550.20281.21831.61820.753
 Nesterov0.11320.09570.20880.82460.253
Wiki-arGradient0.91310.20131.11441.38900.802
 Nesterov0.42010.14220.56230.95770.587
Appendix
Table 5: Signed contributions and cancellation at 99.9M tokens. AspA_{\mathrm{sp}} and AbuA_{\mathrm{bu}} are pooled signed alignments; AallA_{\mathrm{all}} is their sum and BallB_{\mathrm{all}} sums absolute alignments. C=∣Aall∣/BallC=|A_{\mathrm{all}}|/B_{\mathrm{all}} measures the fraction remaining after signed cancellation. Every row pools 24 matrices, and every matrix has positive signed bulk alignment. Displayed sums are rounded independently.
CorpusFrameα‾bu/α‾sp\overline{\alpha}_{\mathrm{bu}}/\overline{\alpha}_{\mathrm{sp}}Rmat(2)R_{\mathrm{mat}}(2)Rmat(4)R_{\mathrm{mat}}(4)
FineMath-4Gradient0.0090.6010.397
 Nesterov0.0620.8280.738
CodeParrotGradient0.0070.5900.381
 Nesterov0.0320.7390.604
Wiki-arGradient0.0120.6020.395
 Nesterov0.0190.6380.450
Appendix
Table 6: Pre-update reference-alignment trade-off at 99.9M tokens. The per-direction ratio uses pooled signed means α‾b=Ab/Nb\overline{\alpha}_{b}=A_{b}/N_{b} ; the ratio Γ\Gamma in Appendix D uses absolute alignments. Rmat(ρ)R_{\mathrm{mat}}(\rho) applies each matrix’s own norm-matching factor before summing, as in equation 36 . Values below one indicate lower alignment than the flat profile in the same frame. Each row uses the same 24 matrices and reference gradient as the preceding diagnostics, without learning-rate or shape multipliers.
  Loss reduction B^ϵ\widehat{B}_{\epsilon}Slope S^ϵ\widehat{S}_{\epsilon}
  ( 10−410^{-4} )( 10−310^{-3} )
CorpusTokens (M)ϵ=0.25\epsilon=0.25ϵ=0.5\epsilon=0.5ϵ=0.25\epsilon=0.25ϵ=0.5\epsilon=0.5
FineMath-425.0+2.524+3.725+1.709+2.774
CodeParrot25.0+1.842+2.574+1.350+2.320
Wiki-ar25.0+6.593+10.226+4.082+5.944
FineMath-450.0-0.604-1.191-0.177+0.032
Appendix
Table 7: Single-update validation-loss improvement ( K=0K=0 ). Both ρ=1.25\rho=1.25 and ρ=1.5\rho=1.5 lower post-update validation loss relative to the exact flat update ρ=1\rho=1 in five of six tested states. Each row is a Pythia-70M corpus–checkpoint state with a common gain across 24 hidden matrices and separate per-matrix Frobenius norm matching. B^ϵ\widehat{B}_{\epsilon} measures the loss reduction relative to ρ=1\rho=1 , whereas S^ϵ\widehat{S}_{\epsilon} compares the bulk-emphasizing and bulk-suppressing branches through equation 37 . Loss reductions and raw central finite-difference slopes are reported in units of 10−410^{-4} and 10−310^{-3} , respectively. All six states are shown; token labels are rounded, and the duplicate ρ=1\rho=1 branches have identical recorded losses.
 14M31M70M160M410M
Corpusℓ0\ell_{0}Δ\Deltaℓ0\ell_{0}Δ\Deltaℓ0\ell_{0}Δ\Deltaℓ0\ell_{0}Δ\Deltaℓ0\ell_{0}Δ\Delta
Wiki-de4.530−1.247±0.001-1.247{\scriptstyle\,\pm\,0.001}4.009−1.050±0.002-1.050{\scriptstyle\,\pm\,0.002}3.699−1.019±0.003-1.019{\scriptstyle\,\pm\,0.003}3.171−0.719±0.002-0.719{\scriptstyle\,\pm\,0.002}2.626−0.445±0.001-0.445{\scriptstyle\,\pm\,0.001}
Wiki-zh3.715−0.970±0.001-0.970{\scriptstyle\,\pm\,0.001}3.465−0.994±0.001-0.994{\scriptstyle\,\pm\,0.001}3.329−1.118±0.005-1.118{\scriptstyle\,\pm\,0.005}3.090−1.035±0.001-1.035{\scriptstyle\,\pm\,0.001}2.570−0.691±0.002-0.691{\scriptstyle\,\pm\,0.002}
FineMath-43.572−0.486±0.002-0.486{\scriptstyle\,\pm\,0.002}3.278−0.502±0.001-0.502{\scriptstyle\,\pm\,0.001}3.088−0.537±0.002-0.537{\scriptstyle\,\pm\,0.002}2.753−0.481±0.003-0.481{\scriptstyle\,\pm\,0.003}2.421−0.375±0.005-0.375{\scriptstyle\,\pm\,0.005}
Wiki-ar2.936−0.930±0.000-0.930{\scriptstyle\,\pm\,0.000}2.761−0.906±0.001-0.906{\scriptstyle\,\pm\,0.001}2.747−1.045±0.000-1.045{\scriptstyle\,\pm\,0.000}2.620−1.012±0.000-1.012{\scriptstyle\,\pm\,0.000}2.150−0.698±0.001-0.698{\scriptstyle\,\pm\,0.001}
FineMath-32.930−0.290±0.002-0.290{\scriptstyle\,\pm\,0.002}2.692−0.261±0.002-0.261{\scriptstyle\,\pm\,0.002}2.566−0.269±0.004-0.269{\scriptstyle\,\pm\,0.004}2.320−0.254±0.002-0.254{\scriptstyle\,\pm\,0.002}2.050−0.159±0.002-0.159{\scriptstyle\,\pm\,0.002}
Appendix
Table 8: Corpus adaptation across five model scales. ℓ0\ell_{0} is the held-out loss before continued pretraining. Δ=ℓfinal−ℓ0\Delta=\ell_{\mathrm{final}}-\ell_{0} is the change after about 100M tokens with Muon , reported as mean ±\pm standard deviation over three seeds. Negative values indicate improvement. ℓ0\ell_{0} has no spread, because all seeds start from the same checkpoint and use the same held-out windows. Rows are ordered by decreasing ℓ0\ell_{0} , which gives the same order at every scale. Corpora are grouped into mathematics, code, and multilingual Wikipedia.
ModelMethodFineMath-4FineMath-3CodeParrotWiki-deWiki-zhWiki-ar
14MBB-Static (1.5%)3.088±0.0033.088{\scriptstyle\,\pm\,0.003}2.642±0.0022.642{\scriptstyle\,\pm\,0.002}1.903±0.0031.903{\scriptstyle\,\pm\,0.003}3.278±0.001\mathbf{3.278}{\scriptstyle\,\pm\,0.001}2.741±0.0002.741{\scriptstyle\,\pm\,0.000}1.999±0.0011.999{\scriptstyle\,\pm\,0.001}
BB-Static (2%)3.086±0.0023.086{\scriptstyle\,\pm\,0.002}2.639±0.0012.639{\scriptstyle\,\pm\,0.001}1.899±0.0031.899{\scriptstyle\,\pm\,0.003}3.279±0.0003.279{\scriptstyle\,\pm\,0.000}2.739±0.0012.739{\scriptstyle\,\pm\,0.001}1.998±0.001\mathbf{1.998}{\scriptstyle\,\pm\,0.001}
BB-Static (5%)3.085±0.0033.085{\scriptstyle\,\pm\,0.003}2.639±0.0022.639{\scriptstyle\,\pm\,0.002}1.900±0.0031.900{\scriptstyle\,\pm\,0.003}3.278±0.001\mathbf{3.278}{\scriptstyle\,\pm\,0.001}2.738±0.001\mathbf{2.738}{\scriptstyle\,\pm\,0.001}2.000±0.0002.000{\scriptstyle\,\pm\,0.000}
BB-Static (15%)3.083±0.002\mathbf{3.083}{\scriptstyle\,\pm\,0.002}2.639±0.0022.639{\scriptstyle\,\pm\,0.002}1.898±0.003\mathbf{1.898}{\scriptstyle\,\pm\,0.003}3.279±0.0013.279{\scriptstyle\,\pm\,0.001}2.742±0.0012.742{\scriptstyle\,\pm\,0.001}2.006±0.0012.006{\scriptstyle\,\pm\,0.001}
BB-Static (20%)3.083±0.002\mathbf{3.083}{\scriptstyle\,\pm\,0.002}2.638±0.002\mathbf{2.638}{\scriptstyle\,\pm\,0.002}1.899±0.0041.899{\scriptstyle\,\pm\,0.004}3.282±0.0023.282{\scriptstyle\,\pm\,0.002}2.745±0.0012.745{\scriptstyle\,\pm\,0.001}2.009±0.0012.009{\scriptstyle\,\pm\,0.001}
31MBB-Static (1.5%)2.776±0.0012.776{\scriptstyle\,\pm\,0.001}2.433±0.0022.433{\scriptstyle\,\pm\,0.002}1.650±0.0021.650{\scriptstyle\,\pm\,0.002}2.956±0.0012.956{\scriptstyle\,\pm\,0.001}2.463±0.001\mathbf{2.463}{\scriptstyle\,\pm\,0.001}1.848±0.001\mathbf{1.848}{\scriptstyle\,\pm\,0.001}
Appendix
Table 9: Fixed-fraction sweep for BB-Static : final validation loss after about 100M training tokens, reported as mean ±\pm standard deviation over three seeds. For p∈{0.015,0.02,0.05,0.15,0.20}p\in\{0.015,0.02,0.05,0.15,0.20\} , the leading band contains rt=⌈pd⌉r_{t}=\lceil pd\rceil singular directions, where d=min⁡{m,n}d=\min\{m,n\} and pp is held fixed within each run. All fractions use ρ=4\rho=4 and the shared learning rate of their scale, with the same training budget, schedule, and seeds as Table 1 . For each model–corpus setting, the fraction with the lowest three-seed mean gives the BB-Static † entry in Table 1 . Table 2 summarizes the shared configuration p=5%p=5\% , and Figure 1 shows p=1.5%p=1.5\% . Bold marks the lowest mean among the fractions in each setting, including ties at the printed precision.
ArmModelFineMath-4FineMath-3CodeParrotWiki-deWiki-zhWiki-ar
Freon14M8.3±0.78.3{\scriptstyle\,\pm\,0.7}3.5±2.93.5{\scriptstyle\,\pm\,2.9}7.1±1.67.1{\scriptstyle\,\pm\,1.6}4.9±2.64.9{\scriptstyle\,\pm\,2.6}4.4±2.64.4{\scriptstyle\,\pm\,2.6}–
31M7.8±1.87.8{\scriptstyle\,\pm\,1.8}8.7±3.58.7{\scriptstyle\,\pm\,3.5}3.3±2.93.3{\scriptstyle\,\pm\,2.9}4.4±1.74.4{\scriptstyle\,\pm\,1.7}4.7±0.74.7{\scriptstyle\,\pm\,0.7}–
70M11.1±1.411.1{\scriptstyle\,\pm\,1.4}10.6±3.210.6{\scriptstyle\,\pm\,3.2}12.5±0.712.5{\scriptstyle\,\pm\,0.7}9.8±2.69.8{\scriptstyle\,\pm\,2.6}10.9±3.310.9{\scriptstyle\,\pm\,3.3}10.2±1.510.2{\scriptstyle\,\pm\,1.5}
160M––––––
410M9.9±3.19.9{\scriptstyle\,\pm\,3.1}9.6±3.59.6{\scriptstyle\,\pm\,3.5}–6.0±5.16.0{\scriptstyle\,\pm\,5.1}––
BB-Static (5%)14M4.6±0.44.6{\scriptstyle\,\pm\,0.4}3.4±2.93.4{\scriptstyle\,\pm\,2.9}6.1±2.16.1{\scriptstyle\,\pm\,2.1}6.3±2.16.3{\scriptstyle\,\pm\,2.1}10.5±2.410.5{\scriptstyle\,\pm\,2.4}12.3±2.212.3{\scriptstyle\,\pm\,2.2}
Appendix
Table 10: Token savings against Muon , as a percentage of the 100M-token budget, for each method and model–corpus setting. Muon is the reference, so it has no block of its own. A cell is filled only where the method’s three-seed mean final loss is below Muon ’s in Table 1 , and “–” marks settings where Muon is at least as good. A filled cell reports 1−D⋆/100M1-D^{\star}/100\mathrm{M} , where D⋆D^{\star} is the number of tokens at which the method’s curve first reaches Muon ’s 100M-token loss in the same seed. Values are mean ±\pm standard deviation over the seeds in which the method ends below Muon . A superscript k/3k/3 marks a setting won in only kk of the three seeds. DeVA and Spectra never beat Muon , so they are omitted. The last block, BB-Static † , uses the best of the five fractions in each setting (Table 9 ). Because this fraction is chosen on the same runs, its savings are optimistic. Separately, intermediate points of a cosine schedule are evaluated before the learning rate has fully decayed.
Figure 2: Learning-rate sweeps on FineMath-4 with a 20M-token budget and one seed. Curves show final validation loss against multiples of each scale’s reference rate. Grey bands mark each method’s best multiplier. BB-Static uses p=20%p=20\% in the sweep. Table 11 lists the full results.
ModelMethod0.01 ×\times0.05 ×\times0.1 ×\times0.25 ×\times0.5 ×\times1 ×\times2 ×\times4 ×\timesbest
14MMuon3.3353.2323.2093.2273.2813.3543.4593.5810.1 ×\times
Freon3.3383.2313.2073.2193.2653.3353.4273.5400.1 ×\times
DeVA3.2663.2353.2263.2323.2673.3283.4213.6150.1 ×\times
Spectra3.2723.2573.2503.2593.3043.4003.5554.4580.1 ×\times
BB-Static (20%)3.3403.2323.2073.2173.2593.3323.4223.5430.1 ×\times
BB-Adaptive3.3363.2303.2073.2233.2733.3473.4483.5720.1 ×\times
Appendix
Table 11: Final held-out loss on FineMath-4 after 20M tokens with one seed, at multiples of each scale’s reference rate. Bold marks each method’s lowest loss, and underline marks its second lowest. BB-Static uses p=20%p=20\% . The 100M-token runs use one shared multiplier per scale, the best for most methods: 0.10.1 , 0.10.1 , 0.250.25 , 0.10.1 , and 0.050.05 from 14M to 410M.
ModelCorpusρ=2\rho{=}2ρ=4\rho{=}4ρ=6\rho{=}6ρ=8\rho{=}8ρ=10\rho{=}10bestrange
14MFineMath-43.2093.2093.2093.2093.21060.000
FineMath-32.7372.7372.7362.7372.735100.002
CodeParrot2.0462.0462.0482.0482.04840.002
Wiki-de3.5183.5223.5233.5263.52820.010
Wiki-zh2.9602.9622.9642.9652.96720.007
Wiki-ar2.1842.1882.1902.1932.19420.011
Appendix
Table 12: ρ\rho tuning across all 30 model–corpus pairs, measured by final held-out validation loss after 20M tokens. The best loss in each row is bold , and the second-best is underlined . The range column is the difference between the largest and smallest loss, computed before rounding; its median is 0.0090.009 . At each model scale, the sweep uses the same peak learning rate as the main 100M-token BB-Adaptive runs (Table 11 ), held fixed across ρ\rho values and corpora. The sweep uses a different seed from the learning-rate sweep, so its ρ=4\rho=4 entries on FineMath-4 differ from the BB-Adaptive entries of Table 11 by at most 0.0080.008 .
Modelmatricesmedian ρeff\rho_{\mathrm{eff}}p5–p95ρeff−ρ\rho_{\mathrm{eff}}-\rhoRitz residual
Pythia-14M1443.9803.88–4.07−0.020-0.0204.1 ×10−7\times 10^{-7}
Pythia-31M1443.9803.91–4.04−0.020-0.0204.7 ×10−7\times 10^{-7}
Pythia-70M1444.0053.95–4.06+0.005+0.0056.2 ×10−7\times 10^{-7}
Pythia-160M2883.9633.92–4.05−0.037-0.0371.5 ×10−6\times 10^{-6}
Pythia-410M5763.9653.93–4.02−0.035-0.0358.2 ×10−6\times 10^{-6}
Appendix
Table 13: Numerical diagnostics for BB-Adaptive at the target gain ρ=4\rho=4 , summarized by model scale. “matrices” counts the instrumented two-dimensional weight matrices across the six corpora. We report the median effective gain ρeff\rho_{\mathrm{eff}} , its 5th–95th percentile interval, and the difference between the median and the target gain. The final column reports the median Rayleigh–Ritz residual of the retained spike subspaces.
Arm14M31M70M160M410M
Muon (ms/step)173±5173{\scriptstyle\,\pm\,5}308±1308{\scriptstyle\,\pm\,1}617±0617{\scriptstyle\,\pm\,0}635±1635{\scriptstyle\,\pm\,1}1748±101748{\scriptstyle\,\pm\,10}
AdamW0.93±0.05×0.93{\scriptstyle\,\pm\,0.05}\times0.92±0.01×0.92{\scriptstyle\,\pm\,0.01}\times0.98±0.01×0.98{\scriptstyle\,\pm\,0.01}\times0.98±0.00×0.98{\scriptstyle\,\pm\,0.00}\times0.98±0.01×0.98{\scriptstyle\,\pm\,0.01}\times
Freon1.17±0.01×1.17{\scriptstyle\,\pm\,0.01}\times1.10±0.00×1.10{\scriptstyle\,\pm\,0.00}\times1.06±0.00×1.06{\scriptstyle\,\pm\,0.00}\times1.17±0.00×1.17{\scriptstyle\,\pm\,0.00}\times1.20±0.00×1.20{\scriptstyle\,\pm\,0.00}\times
DeVA1.13±0.00×1.13{\scriptstyle\,\pm\,0.00}\times1.10±0.00×1.10{\scriptstyle\,\pm\,0.00}\times1.11±0.00×1.11{\scriptstyle\,\pm\,0.00}\times1.27±0.00×1.27{\scriptstyle\,\pm\,0.00}\times1.34±0.00×1.34{\scriptstyle\,\pm\,0.00}\times
Spectra0.98±0.01×0.98{\scriptstyle\,\pm\,0.01}\times1.00±0.00×1.00{\scriptstyle\,\pm\,0.00}\times1.00±0.00×1.00{\scriptstyle\,\pm\,0.00}\times1.00±0.00×1.00{\scriptstyle\,\pm\,0.00}\times1.00±0.00×1.00{\scriptstyle\,\pm\,0.00}\times
BB-Static (1.5%)1.59±0.05×1.59{\scriptstyle\,\pm\,0.05}\times1.32±0.01×1.32{\scriptstyle\,\pm\,0.01}\times1.19±0.00×1.19{\scriptstyle\,\pm\,0.00}\times1.27±0.00×1.27{\scriptstyle\,\pm\,0.00}\times1.27±0.01×1.27{\scriptstyle\,\pm\,0.01}\times
Appendix
Table 14: Per-step wall-clock time at each model scale. For each run, the per-step time is its total wall-clock time divided by its number of steps. Entries are mean ±\pm standard deviation over the runs at that scale, across six corpora and three seeds. The Muon row reports milliseconds per step; all other rows report runtime relative to Muon at the same scale. Fixed-rank timings cover p=1.5%p=1.5\% and 5%5\% . Measurements were collected on shared cluster nodes and are subject to scheduling variability. The ratios within a scale are stable.
Modeldmodeld_{\mathrm{model}}micro-batch ×\times accumstepstokensMuon ms/step
Pythia-14M12832 ×\times 13051100M173±5173{\scriptstyle\,\pm\,5}
Pythia-31M25632 ×\times 13051100M308±1308{\scriptstyle\,\pm\,1}
Pythia-70M51232 ×\times 13051100M617±0617{\scriptstyle\,\pm\,0}
Pythia-160M76832 ×\times 13051100M635±1635{\scriptstyle\,\pm\,1}
Pythia-410M102432 ×\times 13051100M1748±101748{\scriptstyle\,\pm\,10}
Appendix
Table 15: Pythia model configurations and training budgets. dmodeld_{\mathrm{model}} denotes the hidden width, and “micro-batch ×\times accum” gives the number of sequences per microbatch and the number of gradient accumulation steps. All scales use an effective batch of 32 sequences of length 1024, giving 32,768 tokens per step and approximately 100M tokens over 3,051 steps. The Muon timing entries reproduce those in Table 14 : mean ±\pm standard deviation over six corpora and three seeds at each scale.
Figure 3: Final held-out validation loss as a function of model size, with one panel per corpus and all seven optimization methods shown. Across all corpora, final loss decreases consistently as model size increases, matching the expected scaling trend. The rate of decrease differs by corpus, reflecting differences in adaptation difficulty, but the overall monotonic improvement with scale remains stable. The BB-Static curve uses p=20%p=20\% (Table 9 ); the other curves match Table 1 .
Figure 4: Held-out loss vs. tokens on FineMath-4, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the yy -ranges differ per panel.
Figure 5: Held-out loss vs. tokens on FineMath-3, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the yy -ranges differ per panel.
Figure 6: Held-out loss vs. tokens on CodeParrot, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the yy -ranges differ per panel.
Figure 7: Held-out loss vs. tokens on Wiki-de, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the yy -ranges differ per panel.
Figure 8: Held-out loss vs. tokens on Wiki-ar, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0\ell_{0} dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the yy -ranges differ per panel.

Explore similar work

CardsList
Jun 11, 2026cs.LG

Muonp^p: Muon with Fractional Spectral Powers

Muon is an increasingly widely used optimizer that replaces a gradient G=USV⊤G=USV^\top with its polar factor UV⊤UV^\top, thereby flattening the singular spectrum. However, full flattening discards singular-value information that may matter for adaptation. We introduce Muonp^p, a Muon-style optimizer that instead uses fractional spectral-power updates USpV⊤US^pV^\top for rational p∈(0,1)p\in(0,1), interpolating between Muon and gradient descent. To make it practical, we prove that fractional spectral powers cannot be computed by any fixed univariate polynomial iteration, and furthermore derive low-degree odd bivariate recurrences that approximate USpV⊤US^pV^\top using only matrix multiplications, preserving Muon's matrix-multiplication-only structure and compute complexity. We show that Muonp^p maximizes the linear improvement in loss under the Schatten qq-norm for q=1+1pq=1+\frac{1}{p}. Empirically, Muonp^p is especially effective for finetuning: on billion-scale models, Muonp^p improves validation perplexity and downstream task performance. We further analyze when Muonp^p is less suitable, through the lens of spectral geometry. Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.

Yihe Dong, Will Sawin

Princeton University

MuonSpectralGradient DescentMatrix Multiplication
AskPDF
Jun 7, 2026cs.LG

The Spectral Dynamics and Noise Geometry of Muon

Muon replaces a matrix gradient G=UΣV⊤G=UΣV^\top by its polar factor UV⊤UV^\top. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso Poggio

Massachusetts Institute of Technology

MuonSpectralPretraining
AskPDF
May 16, 2026cs.LG

DynMuon: A Dynamic Spectral Shaping View of Muon

In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the usual update matrix M=UΣV⊤M=UΣV^\top with its polar factor UV⊤UV^\top. In this work, we consider a class of Muon-like updates, where we replace the update MM with UΣpV⊤UΣ^p V^\top for some parameter pp. We call this a "spectral-shaping" operation, and develop a theory of how to pick pp which depends on (a) local curvature of the loss function, (b) noise stemming from stochastic gradients and label noise, and (c) training stage. Our theory and experimentation reveal a previously overlooked behavior: positive pp helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative pp helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we propose DynMuon, an efficient dynamic spectral shaping method that schedules pp from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show that DynMuon consistently achieves lower validation loss than Muon, while requiring 10.6-26.5% fewer steps to reach the same target loss. Our code is available at https://github.com/fzwark/DynMuon.

Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1

University of Wisconsin–Madison · MIT · Elorian AI

MuonSpectralLoss Function
AskPDF