Spectral Condition for $μ$P under Width-Depth Scaling

Chenyu Zheng; Rongzhen Wang; Xinyu Zhang; Chongxuan Li

cs.LGFeb 28, 2026

Spectral Condition for $μ$ P under Width-Depth Scaling

Authors: Chenyu Zheng, Rongzhen Wang, Xinyu Zhang, Chongxuan Li

Organizations: Gaoling School of AI, Renmin University of China · ByteDance Seed

Abstract

Generative foundation models are increasingly scaled in both width and depth, posing significant challenges for stable feature learning and reliable hyperparameter (HP) transfer across model sizes. While maximal update parameterization ( $μ$ P) has provided a principled solution to both problems for width scaling, existing extensions to the joint width-depth scaling regime remain fragmented, architecture- and optimizer-specific, and often rely on technically involved theories. In this work, we develop a simple and unified spectral framework for $μ$ P under joint width-depth scaling. For deep residual networks whose residual blocks contain $k$ transformations, the framework specifies how the norms of weights and their per-step updates should scale with width and depth. It reveals a fundamental transition from $k=1$ to $k\geq 2$ , unifying previously disparate $μ$ P formulations and identifying the $k\geq 2$ case as more appropriate for practical architectures with multi-transformation branches such as Transformers. Building on this framework, we derive a general recipe for implementing $μ$ P across a broad class of optimizers by mapping spectral constraints to concrete HP parameterizations, recovering existing results and extending them to additional optimizers. Finally, experiments on GPT-2 style language models show that the $μ$ P formulation derived from the $k\geq 2$ case achieves stable feature learning and robust HP transfer under width-depth scaling, whereas standard parameterization and $μ$ P in the $k=1$ case often fail to do so. These results support the practical effectiveness of the proposed spectral framework.

Figures & tables

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$

Table 1:

\mu

P implementation of Condition 3.1 (

k=2

) for Muon-Kimi [ 26 ] under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

Panel of Figure 1 — Figure 1: Feature learning and HP transfer of Muon-Kimi-AdamW under SP and $\mu$ P. We train GPT-2 style models with Muon-Kimi-AdamW using SP and $\mu$ P derived from Condition 3.1 (see Tables 1 and 6 ). $\mu$ P maintains stable feature norms and enables robust HP transfer across both width and depth scaling, while generally achieving lower loss than SP as the model size increases. The detailed numerical values are provided in Appendix E.3.2 .

Panel of Figure 2 — Figure 2: Validating the role of residual block depth $k$ . We compare two $\mu$ P implementations for GPT-2-style models trained with Muon-Kimi-AdamW and Muon-AdamW: Depth- $\mu$ P-style formulation from Condition B.1 ( $k=1$ ) and CompleteP-style formulation from Condition 3.1 ( $k\geq 2$ ). Condition 3.1 yields more stable HP transfer, empirically supporting it as the appropriate $\mu$ P condition for architectures with multi-transformation residual branches. The numerical values are provided in Appendix E.3 .

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer family	Detailed parameterization
Muon-Kimi	Table 3
Muon / Shampoo / SOAP	Table 4
SGD	Table 5
AdamW / Lion / Sophia	Table 6
SSO	Table 7

Appendix

Table 2: Overview of

\mu

P implementation from Condition 3.1 (

k=2

) for optimizer families with weight decay. Optimizers in the same row share the same

\mu

P scaling rules.

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$

Appendix

Table 3:

\mu

P implementation of Condition 3.1 (

k=2

) for Muon-Kimi [ 26 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )
Shampoo $\varepsilon$	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{L}^{2}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )

Appendix

Table 4:

\mu

P implementation of Condition 3.1 (

k=2

) for Muon [ 21 ] , Shampoo [ 12 ] , and SOAP [ 38 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

	Input weights & biases	Hidden weights	Output weights	Hidden biases
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}r_{L}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}r_{L}r_{n}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/r_{L}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/(r_{L}r_{n})$ ( $\lambda_{\mathrm{base}}$ )

Appendix

Table 5:

\mu

P implementation of Condition 3.1 (

k=2

) for SGD with weight decay under width–depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image. The initial variance of input bias is

\sigma^{2}_{\mathrm{base}}

	Input weights & biases	Hidden weights	Output weights	Hidden biases
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/r_{n}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}$
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}r_{n}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}$
AdamW $\varepsilon$	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/(r_{L}r_{n})$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/(r_{L}r_{n})$ ( $\varepsilon_{\mathrm{base}}$ )

Appendix

Table 6:

\mu

P implementation of Condition 3.1 (

k=2

) for AdamW [ 27 ] , Lion [ 7 ] , and Sophia [ 25 ] with weight decay under width–depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image. The initial variance of input bias is

\sigma^{2}_{\mathrm{base}}

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/r_{L}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )

Appendix

Table 7:

\mu

P implementation of Condition 3.1 (

k=2

) for SSO [ 39 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

Optimizer family	Detailed parameterization
Muon-Kimi	Table 9
Muon / Shampoo / SOAP	Table 10
SGD	Table 11
AdamW / Lion / Sophia	Table 12
SSO	Table 13

Appendix

Table 8: Overview of

\mu

P implementation from Condition B.1 (

k=1

) for optimizer families with weight decay. Optimizers in the same row share the same

\mu

P scaling rules.

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/\sqrt{r_{n}r_{L}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$

Appendix

Table 9:

\mu

P implementation of Condition B.1 (

k=1

) for Muon-Kimi [ 26 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}\sqrt{r_{n}}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}/\sqrt{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )
Shampoo $\varepsilon$	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{L}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )

Appendix

Table 10:

\mu

P implementation of Condition B.1 (

k=1

) for Muon [ 21 ] , Shampoo [ 12 ] , and SOAP [ 38 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

	Input weights & biases	Hidden weights	Output weights	Hidden biases
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}r_{n}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}/(r_{n}\sqrt{r_{L}})$ ( $\lambda_{\mathrm{base}}$ )

Appendix

Table 11:

\mu

P implementation of Condition B.1 (

k=1

) for SGD with weight decay under width–depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image. The initial variance of input bias is

\sigma^{2}_{\mathrm{base}}

	Input weights & biases	Hidden weights	Output weights	Hidden biases
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/(r_{n}\sqrt{r_{L}})$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}r_{n}$ ( $\lambda_{\mathrm{base}}$ )	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}$
AdamW $\varepsilon$	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/(r_{n}\sqrt{r_{L}})$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/r_{n}$ ( $\varepsilon_{\mathrm{base}}$ )	$\varepsilon_{\mathrm{base}}/(r_{n}\sqrt{r_{L}})$ ( $\varepsilon_{\mathrm{base}}$ )

Appendix

Table 12:

\mu

P implementation of Condition B.1 (

k=1

) for AdamW [ 27 ] , Lion [ 7 ] , and Sophia [ 25 ] with weight decay under width–depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image. The initial variance of input bias is

\sigma^{2}_{\mathrm{base}}

	Input weights	Hidden weights	Output weights
Block Multiplier	$\alpha_{\mathrm{base}}$	$\alpha_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\alpha_{\mathrm{base}}$ )	$\alpha_{\mathrm{base}}/r_{n}$ ( $\alpha_{\mathrm{base}}$ )
Initial Variance	$\sigma^{2}_{\mathrm{base}}/d_{0}$ or $\sigma^{2}_{\mathrm{base}}$	$\sigma^{2}_{\mathrm{base}}/r_{n}$ ( $\sigma^{2}_{\mathrm{base}}$ )	$\sigma^{2}_{\mathrm{base}}$
Learning Rate	$\eta_{\mathrm{base}}$	$\eta_{\mathrm{base}}/\sqrt{r_{L}}$ ( $\eta_{\mathrm{base}}$ )	$\eta_{\mathrm{base}}{r_{n}}$ ( $\eta_{\mathrm{base}}$ )
Weight Decay	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}$	$\lambda_{\mathrm{base}}/{r_{n}}$ ( $\lambda_{\mathrm{base}}$ )

Appendix

Table 13:

\mu

P implementation of Condition B.1 (

k=1

) for SSO [ 39 ] with weight decay under width-depth scaling. Entries in purple indicate differences between

\mu

P and SP, while gray shows the corresponding SP choices. Here,

r_{n}

and

r_{L}

denote the width and depth scaling ratios relative to the base model. The variance of input weights is

\sigma^{2}_{\mathrm{base}}

for language and

\sigma^{2}_{\mathrm{base}}/d_{0}

for image.

URL	Citation	License
https://github.com/EleutherAI/nanoGPT-mup/tree/completep	[ 10 ]	MIT
https://github.com/karpathy/nanoGPT	[ 23 ]	MIT
https://skylion007.github.io/OpenWebTextCorpus/	[ 11 ]	Creative Commons CC0

Appendix

Table 14: Used assets and their licenses.

Figure 3: Feature learning and HP transfer of Muon-Kimi-AdamW under SP and $\mu$ P from Condition 3.1 ( $k\geq 2$ ) without LayerNorm. First, in terms of training stability, SP becomes increasingly prone to loss divergence as depth increases in the absence of LayerNorm, whereas $\mu$ P enables stable training. Second, unlike SP, $\mu$ P preserves HP transferability at large depths without LayerNorm.

Panel of Figure 4 — Figure 4: Weight decay transfer of Muon-Kimi-AdamW under SP and $\mu$ P from Condition 3.1 ( $k\geq 2$ ). We train GPT-2 style models with Muon-Kimi and AdamW using SP and $\mu$ P derived from Condition 3.1 (see Tables 3 and 6 ). $\mu$ P enables robust HP transfer across both width and depth scaling, while consistently achieving lower loss than SP as the model depth increases. The corresponding numerical values are reported in Appendix E.3.2 .

$n/\log_{2}(\eta_{\mathrm{base}})$	-10	-9	-8	-7	-6	-5
128	5.127	4.685	4.45	4.373	4.364	4.372
256	4.552	4.219	4.081	4.053	4.062	4.091
512	4.093	3.886	3.819	3.833	3.837	4.062
1024	3.817	3.699	3.672	3.68	3.952	5.603
2048	3.654	3.571	3.555	3.798	5.472	6.438
4096	3.56	3.516	3.747	5.557	6.159	6.552

Appendix

Table 15: For Muon-Kimi-AdamW, SP fails to transfer the optimal base learning rate across widths. This table reports the numerical results corresponding to Figure 1 , where the best validation loss for each width is highlighted in bold .

$n/\log_{2}(\eta_{\mathrm{base}})$	-10	-9	-8	-7	-6	-5
128	4.875	4.53	4.42	4.374	4.383	4.397
256	4.561	4.227	4.081	4.059	4.079	4.104
512	4.305	3.974	3.83	3.811	3.828	3.873
1024	4.125	3.798	3.654	3.646	3.676	3.726
2048	3.957	3.636	3.516	3.515	3.552	3.689
4096	3.882	3.531	3.446	3.461	3.523	3.752

Appendix

Table 16: For Muon-Kimi-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across widths, and achieves lower loss than SP as the width increases. This table reports the numerical results corresponding to Figure 1 , where the best validation loss for each width is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-9	-8	-7	-6	-5
4	4.219	4.081	4.056	4.067	4.09
8	4.109	3.985	3.952	3.973	4.013
16	4.016	3.893	3.864	3.889	3.929
32	3.949	3.824	3.799	3.82	3.885
64	3.916	3.777	3.747	3.777	3.91
128	3.898	3.75	3.723	3.772	4.031

Appendix

Table 17: For Muon-Kimi-AdamW, with LayerNorm, SP transfers the optimal base learning rate across the tested depths. This table reports the numerical results corresponding to Figure 1 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-9	-8	-7	-6	-5
4	4.228	4.081	4.06	4.075	4.098
8	4.089	3.972	3.938	3.957	3.988
16	4.01	3.886	3.85	3.874	3.907
32	3.96	3.826	3.8	3.828	3.879
64	3.917	3.771	3.747	3.796	3.942
128	3.878	3.715	3.694	3.754	4.002

Appendix

Table 18: For Muon-Kimi-AdamW, with LayerNorm,

\mu

P from Condition 3.1 (

k\geq 2

) transfers the optimal base learning rate across depths, and achieves lower loss than SP as the depth increases. This table reports the numerical results corresponding to Figure 1 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-9	-8	-7	-6	-5	-4
4	4.225	4.081	4.06	4.067	4.095	4.971
8	4.149	3.988	3.923	3.934	3.951	4.03
16	4.198	3.963	3.871	3.857	3.88	3.922
32	4.325	3.987	3.843	3.796	3.814	3.855
64	4.517	4.052	3.825	3.728	3.725	3.768
128	4.666	4.242	3.912	3.745	3.7	3.746

Appendix

Table 19: For Muon-Kimi-AdamW, with LayerNorm,

\mu

P from Condition B.1 (

k=1

) fails to transfer the optimal base learning rate across depths. This table reports the numerical results corresponding to Figure 2 , where the best validation loss for each depth is highlighted in bold . The implementation can be found in Table 9 and 12 .

$L/\log_{2}(\eta_{\mathrm{base}})$	-13	-12	-11	-10	-9
4	7.318	6.394	5.784	5.169	13.77
8	6.775	5.974	5.426	4.811	NaN
16	6.115	5.631	5.052	4.409	10.814
32	5.809	5.328	4.706	4.233	4.282
64	5.519	5.038	4.516	4.189	7.251
128	5.316	4.896	4.484	4.313	NaN

Appendix

Table 20: For Muon-Kimi-AdamW, without LayerNorm, SP fails to preserve stable training. NaN data points indicate training instability, where the loss explodes.

$L/\log_{2}(\eta_{\mathrm{base}})$	-11	-10	-9	-8	-7
4	5.791	5.169	11.6	NaN	345.85
8	5.741	5.084	131.43	NaN	NaN
16	5.73	5.059	8.732	246.45	122.99
32	5.734	5.069	4.275	3.964	3.894
64	5.73	5.051	4.253	3.912	3.815
128	5.728	5.052	4.214	3.862	3.742

Appendix

Table 21: For Muon-Kimi-AdamW, without LayerNorm,

\mu

P from Condition 3.1 (

k\geq 2

) has stable runs and approximately transfers the optimal base learning rate at large depth

L\geq 32

. NaN data points indicate training instability, where the loss explodes. The best validation loss for each depth is highlighted in bold .

$n/\log_{2}(\lambda_{\mathrm{base}})$	-4	-3	-2	-1	0
128	4.361	4.359	4.375	4.478	4.975
256	4.026	4.019	4.013	4.061	4.327
512	3.809	3.79	3.77	3.789	3.971
1024	3.642	3.614	3.585	3.582	3.724
2048	3.574	3.514	3.467	3.45	3.545
4096	4.711	4.032	3.478	3.406	3.444

Appendix

Table 22: For Muon-Kimi-AdamW, SP fails to transfer the optimal base weight decay across widths. This table reports the numerical results corresponding to Figure 4 , where the best validation loss for each width is highlighted in bold .

$n/\log_{2}(\lambda_{\mathrm{base}})$	-4	-3	-2	-1	0
128	4.359	4.35	4.352	4.406	4.74
256	4.039	4.029	4.033	4.073	4.357
512	3.785	3.767	3.766	3.815	4.109
1024	3.622	3.617	3.602	3.659	3.966
2048	3.493	3.48	3.475	3.539	3.847
4096	3.412	3.398	3.392	3.463	3.782

Appendix

Table 23: For Muon-Kimi-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base weight decay across widths. This table reports the numerical results corresponding to Figure 4 , where the best validation loss for each width is highlighted in bold .

$L/\log_{2}(\lambda_{\mathrm{base}})$	-4	-3	-2	-1	0
4	4.03	4.021	4.017	4.062	4.328
8	3.919	3.911	3.905	3.965	4.314
16	3.842	3.832	3.832	3.905	4.259
32	3.784	3.773	3.781	3.876	4.246
64	3.734	3.726	3.735	3.862	4.244
128	3.675	3.661	3.68	3.831	4.242

Appendix

Table 24: This table reports the numerical results corresponding to depth scaling of Muon-Kimi-AdamW under SP in Figure 4 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\lambda_{\mathrm{base}})$	-4	-3	-2	-1	0
4	4.041	4.037	4.033	4.071	4.343
8	3.905	3.896	3.89	3.954	4.267
16	3.834	3.824	3.825	3.906	4.232
32	3.776	3.765	3.77	3.863	4.192
64	3.699	3.686	3.693	3.807	4.143
128	3.669	3.657	3.669	3.794	4.137

Appendix

Table 25: For Muon-Kimi-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base weight decay across depths, and achieves lower loss than SP as the depth increases. This table reports the numerical results corresponding to Figure 4 , where the best validation loss for each depth is highlighted in bold .

Panel of Figure 5 — Figure 5: Feature learning and HP transfer of Muon-AdamW under SP and $\mu$ P. We train GPT-2 style models with Muon-AdamW using SP and $\mu$ P derived from Condition 3.1 (see Tables 4 and 6 ). $\mu$ P maintains stable feature norms and enables robust HP transfer across both width and depth scaling, while generally achieving lower loss than SP as the model size increases. The detailed numerical values are provided in Appendix E.3.3 .

$n/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
128	4.88	4.544	4.416	4.377	4.375
256	4.38	4.168	4.061	4.067	4.055
512	4.054	3.888	3.831	3.819	3.818
1024	3.848	3.711	3.655	3.648	3.923
2048	3.742	3.595	3.532	3.553	5.137
4096	3.735	3.586	3.543	3.795	5.992

Appendix

Table 26: For Muon-AdamW, SP fails to transfer the optimal base learning rate across widths. This table reports the numerical results corresponding to Figure 5 , where the best validation loss for each width is highlighted in bold .

$n/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
128	4.671	4.473	4.385	4.385	4.387
256	4.403	4.187	4.076	4.073	4.077
512	4.169	3.947	3.834	3.823	3.825
1024	4.02	3.75	3.677	3.668	3.682
2048	3.868	3.624	3.548	3.571	3.669
4096	3.771	3.537	3.486	3.532	4.672

Appendix

Table 27: For Muon-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across widths, and achieves lower loss than SP as the width increases. This table reports the numerical results corresponding to Figure 5 , where the best validation loss for each width is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
4	4.381	4.169	4.064	4.062	4.055
8	4.267	4.059	3.968	3.956	3.961
16	4.201	3.973	3.893	3.883	3.925
32	4.153	3.917	3.841	3.857	4.047
64	4.124	3.884	3.804	3.868	4.772
128	4.084	3.835	3.766	3.886	5.798

Appendix

Table 28: For Muon-AdamW, SP shifts the optimal base learning rate across depths. This table reports the numerical results corresponding to Figure 5 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
4	4.402	4.187	4.082	4.072	4.075
8	4.259	4.044	3.961	3.946	3.981
16	4.198	3.972	3.888	3.888	3.95
32	4.152	3.91	3.832	3.854	3.995
64	4.09	3.842	3.767	3.849	4.223
128	4.076	3.819	3.753	3.869	4.415

Appendix

Table 29: For Muon-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across depths, and achieves lower loss than SP as the depth increases. This table reports the numerical results corresponding to Figure 5 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3	-2
4	4.402	4.189	4.078	4.07	4.073	4.068
8	4.323	4.067	3.97	3.937	3.96	3.983
16	4.368	4.052	3.909	3.862	3.881	3.947
32	4.477	4.073	3.884	3.809	3.823	3.902
64	4.649	4.12	3.874	3.754	3.735	3.834
128	4.793	4.262	3.946	3.78	3.722	3.795

Appendix

Table 30: For Muon-AdamW,

\mu

P from Condition B.1 (

k=1

Panel of Figure 6 — Figure 6: Feature learning and HP transfer of Shampoo-AdamW under SP and $\mu$ P. We train GPT-2 style models with Shampoo-AdamW using SP and $\mu$ P derived from Condition 3.1 (see Tables 4 and 6 ). $\mu$ P maintains stable feature norms and enables robust HP transfer across both width and depth scaling, while generally achieving lower loss than SP as the model size increases. The detailed numerical values are provided in Appendix E.3.4 .

Figure 7: Validating the role of residual block depth $k$ . We compare two $\mu$ P implementations for GPT-2-style models trained with Shampoo-AdamW: Depth- $\mu$ P-style formulation from Condition B.1 ( $k=1$ ) and CompleteP-style formulation from Condition 3.1 ( $k\geq 2$ ). Condition 3.1 yields more stable HP transfer and lower validation loss, empirically supporting it as the appropriate $\mu$ P condition for architectures with multi-transformation residual branches. The numerical values are provided in Appendix E.3.4 .

$n/\log_{2}(\eta_{\mathrm{base}})$	-8	-7	-6	-5	-4
128	4.925	4.749	4.614	4.516	4.521
256	4.688	4.407	4.294	4.239	4.236
512	4.481	4.242	4.147	4.084	4.097
1024	4.385	4.229	4.059	3.992	4.067
2048	4.338	4.205	4.042	3.981	6.002
4096	4.366	4.247	4.148	4.604	7.476

Appendix

Table 31: For Shampoo-AdamW, SP fails to transfer the optimal base learning rate across widths. This table reports the numerical results corresponding to Figure 6 , where the best validation loss for each width is highlighted in bold .

$n/\log_{2}(\eta_{\mathrm{base}})$	-8	-7	-6	-5	-4
128	4.969	4.796	4.61	4.548	4.552
256	4.689	4.404	4.291	4.246	4.249
512	4.44	4.159	4.057	4.012	4.012
1024	4.272	3.995	3.919	3.874	3.887
2048	4.131	3.873	3.787	3.754	3.771
4096	3.995	3.761	3.699	3.658	3.694

Appendix

Table 32: For Shampoo-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) transfers the optimal base learning rate across widths, and achieves lower loss than SP as the width increases. This table reports the numerical results corresponding to Figure 6 , where the best validation loss for each width is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
4	4.696	4.617	4.503	4.425	4.62
8	4.721	4.638	4.511	4.421	4.634
16	4.728	4.662	4.61	4.491	4.811
32	4.783	4.754	4.72	4.704	4.966
64	4.886	4.855	4.851	4.802	5.131
128	5.033	5.023	4.994	4.983	5.276

Appendix

Table 33: For Shampoo-AdamW, SP yields increasing validation loss under depth scaling. This table reports the numerical results corresponding to Figure 6 .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
4	4.696	4.605	4.511	4.487	4.623
8	4.567	4.391	4.335	4.339	4.535
16	4.544	4.293	4.265	4.273	4.415
32	4.451	4.267	4.172	4.166	4.373
64	4.407	4.198	4.134	4.109	4.625
128	4.381	4.176	4.112	4.163	5.603

Appendix

Table 34: For Shampoo-AdamW,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across depths, and achieves lower loss than SP as the depth increases. This table reports the numerical results corresponding to Figure 6 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-7	-6	-5	-4	-3
4	4.704	4.613	4.534	4.471	4.616
8	4.693	4.472	4.383	4.349	4.548
16	4.749	4.56	4.337	4.314	4.425
32	4.82	4.597	4.439	4.277	4.377
64	4.913	4.651	4.457	4.314	4.433
128	5.045	4.787	4.575	4.428	4.493

Appendix

Table 35: For Shampoo-AdamW,

\mu

P from Condition B.1 (

k=1

) gives weaker depth scaling and higher validation loss than

\mu

P from Condition 3.1 (

k\geq 2

). This table reports the numerical results corresponding to Figure 7 .

Panel of Figure 8 — Figure 8: Feature learning and HP transfer of Sophia under SP and $\mu$ P. We train GPT-2 style models with Sophia using SP and $\mu$ P derived from Condition 3.1 (see Table 6 ). $\mu$ P maintains stable feature norms and enables robust HP transfer across both width and depth scaling. The detailed numerical values are provided in Appendix E.3.5 .

Figure 9: Validating the role of residual block depth $k$ . We compare two $\mu$ P implementations for GPT-2-style models trained with Sophia: Depth- $\mu$ P-style formulation from Condition B.1 ( $k=1$ ) and CompleteP-style formulation from Condition 3.1 ( $k\geq 2$ ). Condition 3.1 yields more stable HP transfer, empirically supporting it as the appropriate $\mu$ P condition for architectures with multi-transformation residual branches. The numerical values are provided in Appendix E.3.5 .

$n/\log_{2}(\eta_{\mathrm{base}})$	-16	-15	-14	-13	-12	-11
128	7.485	6.982	6.43	5.935	5.54	5.648
256	6.293	5.97	5.603	5.232	5.143	5.361
512	5.506	5.257	4.946	5.005	4.986	5.28
1024	4.924	4.682	4.536	4.48	4.708	5.988
2048	4.457	4.289	4.157	4.249	5.651	6.169
4096	4.321	4.209	4.413	5.677	5.964	6.101

Appendix

Table 36: For Sophia, SP fails to transfer the optimal base learning rate across widths. This table reports the numerical results corresponding to Figure 8 , where the best validation loss for each width is highlighted in bold .

$n/\log_{2}(\eta_{\mathrm{base}})$	-15	-14	-13	-12	-11	-10
128	6.133	5.788	5.446	5.474	5.571	5.781
256	5.969	5.613	5.235	5.143	5.362	5.756
512	5.903	5.521	5.124	4.955	5.113	5.707
1024	5.863	5.455	5.055	4.98	4.976	5.598
2048	5.827	5.419	5.01	4.932	4.919	5.555
4096	5.833	5.406	4.991	4.831	4.88	5.419

Appendix

Table 37: For Sophia,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across widths larger than

256

. This table reports the numerical results corresponding to Figure 8 , where the best validation loss for each width is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-15	-14	-13	-12	-11
4	5.967	5.61	5.235	5.249	5.314
8	5.902	5.547	5.189	4.999	5.315
16	5.881	5.541	5.226	5.263	5.76
32	5.894	5.58	5.267	5.525	5.737
64	5.922	5.65	5.402	6.047	6.29
128	6.041	5.731	5.492	5.95	6.389

Appendix

Table 38: For Sophia, SP yields increasing validation loss under depth scaling. This table reports the numerical results corresponding to Figure 8 .

$L/\log_{2}(\eta_{\mathrm{base}})$	-15	-14	-13	-12	-11
4	5.966	5.603	5.242	5.319	5.375
8	5.895	5.544	5.178	4.935	5.225
16	5.861	5.532	5.19	4.994	5.362
32	5.812	5.497	5.162	4.906	5.237
64	5.837	5.527	5.238	5.154	5.402
128	5.861	5.599	5.329	5.384	5.915

Appendix

Table 39: For Sophia,

\mu

P from Condition 3.1 (

k\geq 2

) approximately transfers the optimal base learning rate across depths, and achieves lower loss than SP as the depth increases. This table reports the numerical results corresponding to Figure 8 , where the best validation loss for each depth is highlighted in bold .

$L/\log_{2}(\eta_{\mathrm{base}})$	-15	-14	-13	-12	-11	-10	-9
4	5.969	5.609	5.234	5.329	5.356	5.938	6.264
8	5.966	5.612	5.231	4.936	5.212	5.515	6.252
16	5.99	5.67	5.302	5.052	4.939	5.438	6.061
32	6.028	5.73	5.402	5.095	5.026	5.014	5.97
64	6.051	5.785	5.515	5.231	5.072	5.077	5.218
128	6.066	5.83	5.609	5.39	5.252	5.118	5.359

Appendix

Table 40: For Sophia,

\mu

P from Condition B.1 (

k=1

) fails to transfer the optimal base learning rate across depths. This table reports the numerical results corresponding to Figure 9 , where the best validation loss for each depth is highlighted in bold . The implementation can be found in Table 12 .

Figure 10: Validation of Assumption G.1 (weight update). During the training, the ratio $\frac{\|{\bm{W}}_{l}+\Delta{\bm{W}}_{l}\|_{\mathrm{R}}}{\|{\bm{W}}_{l}\|_{\mathrm{R}}+\|\Delta{\bm{W}}_{l}\|_{\mathrm{R}}}$ remains constant near 1 across depth for the input layer and residual block layers, showing non-vanishing updates throughout multiple-step training.

Figure 11: Validation of Assumption G.1 (feature update). During the training, the ratio $\frac{\|{\bm{h}}_{l}+\Delta{\bm{h}}_{l}\|_{\mathrm{R}}}{\|{\bm{h}}_{l}\|_{\mathrm{R}}+\|\Delta{\bm{h}}_{l}\|_{\mathrm{R}}}$ remains around constant near 1 across varying depths, showing non-vanishing updates throughout multiple-step training.

Figure 12: Validation of Assumption G.2 (stable activation). During the training, the ratio of post-activation to pre-activation norms $\frac{\|\phi({\bm{W}}_{l}{\bm{h}}_{l-1})\|_{\mathrm{R}}}{\|{\bm{W}}_{l}{\bm{h}}_{l-1}\|_{\mathrm{R}}}$ remains stable across varying depths, confirming that the ReLU activation does not collapse the norm in non-linear networks.

Figure 13: Validation of Assumption G.3 (per-sample update alignment). We report the averaged ratio of $\frac{\|\frac{1}{B}\Delta{\bm{W}}_{l}^{(i)}{\bm{h}}_{l-1}^{(i)}\|_{\mathrm{R}}}{\|\Delta{\bm{W}}_{l}{\bm{h}}_{l-1}^{(i)}\|_{\mathrm{R}}}$ across the batch of data. The values remain stable across varying depths, suggesting that the batch update does not alter the depth-wise scaling of the single-sample update.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 199 papers

Feb 24, 2026

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization

(μ

)

is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving

μ

P for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive

μ

P for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our

μ

P derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.

Akshita Gupta, Marieme Ngom, Sam Foreman +1

Purdue University · Argonne National Laboratory

Hyperparameter Transfer Adaptive Gradient Methods

Ask PDF

Feb 11, 2026

$μ$ pscaling small models: Principled warm starts and hyperparameter transfer

Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets. To improve efficiency, recent work has explored model upscaling: initializing larger models from trained smaller ones to accelerate convergence. However, this method can be sensitive to hyperparameters that need to be tuned at the target upscaled model size, which is prohibitively costly to do directly. It remains unclear whether tuning hyperparameters on smaller models and extrapolating via scaling laws is sound in this setting. We address this with principled approaches to width-based upscaling and efficient hyperparameter tuning in this setting. Motivated by

μ

P and any-dimensional architectures, we introduce a general upscaling method that, like Net2Net, copies and perturbs weights, but uses theoretically grounded, width-dependent scalings for the perturbation noise and optimizer hyperparameters. First, we prove that under zero perturbation, the upscaled model is functionally equivalent to the base model throughout training. Second, we extend the

μ

P theory to enable infinite-width limit analysis and establish hyperparameter transfer for upscaled models, greatly reducing the tuning cost. We empirically demonstrate that this method is effective on realistic datasets and architectures.

Yuxin Ma, Nan Chen, Mateo Díaz +3

Department of Applied Mathematics and Statistics Johns Hopkins University

Neural Network Optimization Hyperparameter Transfer Neural Scaling Laws Hyperparameter Optimization

Ask PDF

May 14, 2026

GQA-μP: The maximal parameterization update for grouped query attention

Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization (μP) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of Yang et al. (2023a), we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μP scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay.

Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7

UC Davis & MBZUAI IFM · USC & MBZUAI IFM · USC +4

Grouped-Query Attention Representation Learning Hyperparameter Transfer Efficient Attention

Ask PDF

Abstract

Deep Learning Optimization

Explore similar work

Extending μμμP: Spectral Conditions for Feature Learning Across Optimizers

μμμpscaling small models: Principled warm starts and hyperparameter transfer

GQA-μP: The maximal parameterization update for grouped query attention

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers

$μ$ pscaling small models: Principled warm starts and hyperparameter transfer