Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training

Abdelrahman Abouzeid

cs.AIApr 2, 2026

Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training

Authors: Abdelrahman Abouzeid

Organizations: Georgia Institute of Technology

Abstract

In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this assumption can fail: Dynamic Erf (Derf; Chen & Liu, 2025) suffers a large negative interaction with Muon (Jordan, 2024), with its gap to RMSNorm growing from +0.31 nats under AdamW to +0.97 under Muon, approximately three times larger. Dynamic Tanh (DyT; Zhu et al., 2025), included as a bounded-normalizer control, shows no such penalty. Our evidence points to two failure modes of erf under Muon's faster spectral-norm growth: saturation (lossy compression) and scale blindness (discarding activation magnitude). An EMA-blend that reintroduces running scale estimates recovers ~84% of the gap. Separately, reducing Derf's alpha from its published default (0.5 to 0.3) recovers ~80% by keeping erf in its near-linear regime, where it approximately preserves relative scale; this setting is not the published default of Chen & Liu (2025). Using Derf's published default alpha with Muon incurs a 0.66-nat interaction penalty without producing NaNs or divergence, making the failure easy to miss in short pilot runs.

Figures & tables

	AdamW	Muon
RMSNorm	3.883 $\pm$ 0.008	3.322 $\pm$ 0.003
Derf	4.192 $\pm$ 0.003	4.289 $\pm$ 0.017
DyT	4.201 $\pm$ 0.008	3.541 $\pm$ 0.003

Table 1: Left: Validation loss at 1000 steps (1B params, 3 seeds/cell). Right: Gap to RMSNorm under each optimizer; interaction

=

Muon gap

-

AdamW gap. Under AdamW, Derf and DyT have near-identical gaps (

+

0.31,

+

0.32). Under Muon, Derf’s gap triples (

+

0.66 interaction) while DyT’s shrinks (

-

0.10). Bootstrap 95% CIs: Derf

[0.64,0.68]

, DyT

[-0.12,-0.08]

(Appendix B ).

Interaction at 1000: Muon gap $+$ 0.967, AdamW gap $+$ 0.308 $\Rightarrow$ +0.659
Step	Derf+Muon gap	EMA-blend gap ( $\lambda\!=\!0.9$ )
100	$+$ 0.13	$+$ 0.43
300	$+$ 0.71	$+$ 0.35
500	$+$ 0.96	$+$ 0.19
700	$+$ 1.01	$+$ 0.16
1000	$+$ 0.97	$+$ 0.15

Table 2: Gap to RMSNorm+Muon over training (3-seed averages). Derf+Muon widens and plateaus; EMA-blend (

\lambda\!=\!0.9

) narrows monotonically from

+

0.43 to

+

0.15. Both trends are partially confounded with cosine decay.

Figure 1: Overview of the optimizer-driven saturation mechanism. At the tested learning rates, Muon grows weights

{\sim}2.2\times

faster than AdamW, pushing Derf into steep saturation (83% of elements at

\pm 1

). AdamW’s slower growth keeps Derf closer to its useful regime (10% saturation). Muon+EMA (bottom) preserves Muon’s fast weight growth but uses a running

\hat{\sigma}

to keep the post-blend erf argument near unit scale (std 1.3, 2% post-blend saturation), recovering

{\sim}84\%

of the Derf penalty. Weight and saturation numbers from layer 15, step 950 (seed 42); quality gaps at step 1000.

Figure 2: Rescaling (RMSNorm) preserves relative magnitudes at any input scale; squashing (Derf) destroys them outside a narrow operating regime.

Metric (L15, step 950)	Derf+Muon	Derf+AdamW	EMA-blend
MLP weight $\\|W\\|_{F}$	155.8	69.6	163.5
erf input std	11.8	1.8	1.3
erf saturated (%)	83%	10%	2%

Table 3: Per-layer diagnostics at layer 15, step 950 of the main 1000-step seed-42 runs (per-layer breakdown in Appendix Figure 5 ). At the tested learning rates, Muon grows weights

2.2\times

faster, producing

{\sim}8\times

more saturation. EMA-blend has Muon-scale weights but keeps the post-blend erf argument in a low-saturation regime via

\hat{\sigma}

Hypothesis	Intervention	Outcome	Conclusion
Large gradients	Grad clipping	Silent degrad. (gap 0.97)	Not the cause
Weight norms alone	Weight decay	No effect (gap 0.97)	Too weak
Alpha can self-correct	Alpha LR $\uparrow$	No effect (gap 0.97)	Ineffective in saturation regime
Growth rate alone	Residual scaling	Diverges	Insufficient recovery
Saturation alone	asinh compress.	49% recovery (gap 0.50)	Helps, not sufficient
Missing scale info.	EMA-blend	84% recovery (gap 0.15)	Saturation + scale

Table 4: Alternative-explanation tests (targeted probes). Rows 1–4 and asinh are single-seed probes; EMA-blend is replicated with 3 seeds. Only EMA-blend, which restores scale information, achieves substantial recovery. The 35 pp gap between asinh and EMA-blend (similar erf input distributions, different recovery) is consistent with scale information being a distinct factor. Percentages are relative to the full Derf+Muon gap of

+

0.97 nats.

Figure 3: Validation loss over 1000 steps. EMA-blend (

\lambda\!=\!0.9

) converges to within 0.15 nats of RMSNorm+Muon while outperforming RMSNorm+AdamW by 0.41 nats. Derf+Muon fails to outperform Derf+AdamW. Multi-seed variance is negligible. Three-seed results appear in Appendix A .

$\lambda$	Val Loss	Gap to RMS+Muon
0.0 (vanilla Derf)	4.289 $\pm$ 0.017	$+$ 0.967
0.5	3.588 $\pm$ 0.060	$+$ 0.267
0.7	3.504 $\pm$ 0.003	$+$ 0.182
0.9	3.476 $\pm$ 0.001	$+$ 0.155
1.0	6.64 $\pm$ 0.001 (plateau)	$+$ 3.318

Table 5: EMA-blend sweep over

\lambda

at 1B, 1000 steps. Performance improves monotonically from

\lambda\!=\!0.5

\lambda\!=\!0.9

, then collapses at

\lambda\!=\!1.0

\lambda\!=\!0.9

is best; both

\lambda\!=\!0.7

and

\lambda\!=\!0.9

are replicated with 3 seeds.

Method	Allreduces / step	Relative	Quality
RMSNorm	33 $\times$ 64 $\times$ 2 = 4,224	$1\times$	best
Derf + EMA	33 $\times$ 1 = 33	$\sim$ 0.008 $\times$	$-$ 0.15 nats
DyT	0	$0\times$	$-$ 0.22 nats
Derf (no EMA)	0	$0\times$	$-$ 0.97 nats

Table 6: Theoretical per-step allreduce operation count under tensor parallelism (forward

+

backward). The

128\times

reduction in operation count (4,224 vs. 33) translates to

7.8\times

measured norm-layer wall-clock speedup at 8-way TP (Figure 4 ); the gap between theoretical and measured reflects allreduce fusion and overlap in modern TP implementations. DyT and vanilla Derf are fully reduction-free, but only DyT preserves quality with Muon.

Figure 4: Tensor parallelism benchmark on H100 NVLink (1/2/4/8 GPUs; training uses a single H200, Section 3 ). Norm-layer wall-clock time at hidden

=

2048. DyT and Derf are fully reduction-free; EMA requires 33 allreduces/step vs. RMSNorm’s 4,224. At 8-way TP: DyT

9.4\times

, Derf

9.3\times

, EMA

7.8\times

faster than RMSNorm. At 1-GPU (no communication), the gap is pure compute:

{\sim}3.2\times

for all three, confirming the speedup scales with TP degree.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Config	s42	s43	s44	Avg
RMSNorm+Muon	3.321	3.319	3.325	3.322
RMSNorm+AdamW	3.874	3.888	3.888	3.883
DyT+Muon	3.545	3.538	3.541	3.541
DyT+AdamW	4.204	4.192	4.207	4.201
EMA-blend ( $\lambda$ =0.9)	3.476	3.476	3.477	3.476
EMA-blend ( $\lambda$ =0.7)	3.502	3.503	3.507	3.504

Appendix

Table 7: Validation loss at step 1000 across seeds. All configs have 3 seeds. Std

\leq

0.017.

Contrast	Mean	95% CI
Derf gap (Muon)	$+$ 0.967	$[0.95,0.98]$
Derf gap (AdamW)	$+$ 0.308	$[0.30,0.32]$
Derf interaction	$+$ 0.659	$[0.64,0.68]$
Ratio (Muon/AdamW gap)	$3.1\times$	$[3.0,3.2]$
DyT gap (Muon)	$+$ 0.220	$[0.21,0.23]$
DyT gap (AdamW)	$+$ 0.318	$[0.30,0.33]$

Appendix

Table 8: Key contrasts with bootstrap 95% CIs (10K resamples,

n\!=\!3

/cell). CIs are lower bounds on true uncertainty given small

n

Layer	attn	ffn	Interpretation
0	0.50 $\to$ 0.43	0.50 $\to$ 0.42	Early shrinks
7	0.50 $\to$ 0.50	0.50 $\to$ 0.47	Middle stable
15	0.50 $\to$ 0.53	0.50 $\to$ 0.45	FFN shrinks

Appendix

Table 9: Residual scaling parameters at step 400 in the targeted probe.

Metric (L15, step 999)	DyT+Muon	DyT+AdamW	Derf+Muon	Derf+AdamW
MLP weight $\\|W\\|_{F}$	144	70	156	70
Nonlinearity input std	2.96	1.92	10.65	1.76
Saturated (%)	27%	12%	79%	9%
FFN $\alpha$ (init $\to$ final)	0.30 $\to$ 0.175	0.30 $\to$ 0.40	0.50 $\to$ 0.36	0.50 $\to$ 0.44

Appendix

Table 10: DyT vs. Derf diagnostics at layer 15, step 999 (seed 42). Despite comparable Muon-driven weight growth, DyT has far lower saturation because tanh has a wider useful regime and DyT’s

\alpha

adapts more aggressively.

Run	Steps	Val Loss	Muon grad norm
RMSNorm+Muon	999	3.79	0.066
RMSNorm+AdamW	999	3.89	n/a
DyT+Muon ( $\alpha=0.5/0.3$ )	999	4.38	0.089
Derf+Muon ( $\alpha=0.5$ )	186 ∗	6.07	0.184

Appendix

Table 11: Exploratory 125M pilot results. DyT+Muon shows a moderate gap (

+

0.59) at 125M; at 1B, DyT+Muon gap narrows to

+

0.22 (Table 1 ), consistent with DyT scaling better than Derf under Muon. ∗ Derf+Muon terminated by dataset timeout, not divergence, so the 125M pilot is not used to draw scale-dependent conclusions about the Derf interaction.

Muon LR	RMSNorm	Derf	Derf gap
0.02	3.321	4.271	$+$ 0.950
0.01	3.379	3.656	$+$ 0.277
0.005	3.577	3.954	$+$ 0.377

Appendix

Table 12: Muon LR probe (val loss at step 1000; single seed s42). Lowering LR from 0.02 to 0.01 reduces the Derf gap, consistent with slower weight growth producing less saturation. The non-monotonic 0.005 result reflects undertraining.

Figure 5: Per-layer diagnostics at step 340 of the main 1000-step seed-42 runs. Saturation, weight growth, and erf input magnitude increase with depth. For EMA-blend, the saturation panel uses post-blend saturation of the actual erf argument. Muon drives the vanilla Derf metrics higher than AdamW, while EMA-blend decouples weight growth from erf input.

Figure 6: Norm output variance by layer at step 340 of the main 1000-step seed-42 runs. Derf+AdamW and EMA-blend lines begin at layer

{\sim}7

because earlier layers have variance

<0.4

, below the plotted range. Counterintuitively, Derf+Muon has the highest variance among Derf configs (0.94 at L15). This is not because it preserves information, but because 80% saturation pushes outputs to

\pm 1

, a bimodal distribution with high variance but substantially reduced representational resolution.

Figure 7: erf saturation at layer 15 over 1000 training steps. Derf+Muon saturates rapidly and stays at

{\sim}80\%

. Derf+AdamW peaks at

{\sim}50\%

around step 50 then recedes as AdamW’s adaptive

\sqrt{v_{t}}

dampens growth, evidence that AdamW also fights Derf’s saturation, but wins the fight. For EMA-blend (labeled “post-blend”), we plot saturation of the actual erf argument after blending; it falls below 10% once

\hat{\sigma}

converges and reaches

{\sim}1\%

by late training.

Figure 8: Gap trajectories (1000-step cosine schedule). Left: Derf+Muon’s gap widens monotonically and plateaus. Right: EMA-blend’s gap peaks at

+

0.53 around step

{\sim}175

(before

\hat{\sigma}

converges) then narrows. Both trends are partially confounded with cosine decay.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Sep 29, 2026Zixuan Gong, Zeyu Gan, Jiaye Teng +1Language Model Pretraining Matrix Optimization
Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
May 26, 2026Mingze Wang, Shuchen Zhu, Yuxin Fang +3Language Model Pretraining Weight Decay
ByteDance Seed · Peking University
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model Pretraining Efficient Language Model Training
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

Sep 29, 2026J/K move · Enter open · S save

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

Zixuan Gong, Zeyu Gan, Jiaye Teng, Yong Liu

Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics

Language Model Pretraining Matrix Optimization LLM Training

Open paper AskPDF arXiv

Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish

\mathcal{O}(T^{-1/2})

convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at https://github.com/zx-gong/NormPre.

Abstract

Neural Network Optimization

Explore similar work

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining