Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Liliang Ren; Yang Liu; Yelong Shen; Weizhu Chen

cs.LGMar 30, 2026

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Authors: Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen

Organizations: Microsoft

Abstract

Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth- $μ$ P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the "magic exponent" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\times$ compute efficiency over a strong Muon baseline at $6\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$ -values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.

Figures & tables

Figure 1: Left: Loss vs. LR at different token budgets. Right: Fitted optimal LR vs. training tokens on log-log scale, showing a clean power-law relationship with exponent $0.32$ . The exact values are reported in Table 4 .

Parameter	Scheme	LR mult.	Init. std.	Res. mult.	Weight mult.	WD
Embedding/Vector	$\mu$ P	$\propto 1$	$\propto 1$	—	$\propto 1$	$\propto 1$
	$\mu$ P++	$\propto 1/\sqrt{d}$	$\propto 1$	—	$\propto 1$	0
	HyperP	$\propto 1/\sqrt{d}$	$\propto 1$	—	$\propto 1$	0
Unembedding	$\mu$ P	$\propto 1$	$\propto 1$	—	$\propto 1/w$	$\propto 1$
	$\mu$ P++	$\propto 1/\sqrt{d}$	$\propto 1$	—	$\propto 1/w$	0
	HyperP	$\propto 1/\sqrt{d}$	$\propto 1$	—	$\propto 1$	0

Table 1: Differences between

\mu

\mu

P++ [ 38 ] and HyperP under Muon-based optimizers [ 20 ] . LR mult. denotes the per-parameter multiplier applied on top of the global learning-rate (

\eta

), Init. std. means the standard deviation of the initialization, Res. mult. is the multiplier applied to the output of residual branches and WD denotes the weight decay.

w

is the model width,

d

means model depth and

T

is the training tokens.

\mu

P and

\mu

P++ applies Muon for matix-like parameters with adjustments of LR and WD following [ 33 ] and AdamW for unembedding and vector-like parameters, while HyperP adopts MuonH [ 47 ] for matrix-like parameters, AdamH for unembedding weights and AdamW for vector-like parameters.

Figure 2: Validation loss vs. learning rate for Muon (sweeping weight decay $\lambda$ ) and MuonH ( $\lambda{=}0$ ). MuonH achieves comparable optimality with a simpler hyperparameter space.

Method	Fitted $\eta^{*}$	Best Val Loss	Weight Decay
Muon (best $\lambda{=}10^{-3}$ )	0.0222	2.479	$10^{-3}$
MuonH ( $\lambda{=}0$ )	0.0155	2.475	0

Table 2: Fitted optimal learning rate

\eta*

and validation loss between MuonH and Muon. MuonH matches Muon while eliminating weight decay.

Figure 3: Loss vs. LR curves across model sizes with Depth- $\mu$ P (left) and without Depth- $\mu$ P (right). Depth- $\mu$ P keeps the optimal LR stable at $\eta^{*}\approx 0.014$ – $0.016$ across all depths, while the optimum drifts from $\eta^{*}=0.016$ at $d{=}8$ to $\eta^{*}=0.008$ at $d{=}24$ without Depth- $\mu$ P.

Figure 4: Left: Loss vs. LR at different batch sizes. Right: Optimal LR vs. batch size on log-log scale. The exact values are reported in Table 8 .

Figure 5: Loss vs. LR curves for three auxiliary loss weights. The curves nearly overlap, indicating robustness on $\gamma$ under hypersphere optimization. The exact values across all LR and $\gamma$ combinations are reported in Table 9 .

$\gamma$	Best Val Loss	Mean MaxVio
$10^{-3}$	2.334	0.848
$10^{-2}$	2.336	0.132
$10^{-1}$	2.332	0.086

Table 3: Effect of

\gamma

on global load balancing. Mean max violation measures worst-case load imbalance across experts at

\eta^{*}=0.012

from the global batch.

Figure 6: Left: Loss vs. LR across sparsity levels. Right: Optimal loss follows a power law in the MoE sparsity. The exact values are reported in Table 10 .

Figure 7: Loss vs. LR across top- $k$ values with and without SqrtGate. The exact optimal learning rates and losses are provided in Table 11 .

Figure 8: Loss vs. LR across depths with HyperP (left) and without HyperP (right). HyperP keeps the curves aligned and preserves a common base optimum at $\eta_{0}=0.02$ from $d{=}8$ through $d{=}20$ .

Figure 9: Left: Loss vs. FLOPs with power-law fits for all four methods. Right: Compute Efficiency Leverage (CEL) relative to the Muon baseline. MuonH+HyperP achieves 1.58 $\times$ CEL than the MuonH baseline, while the MuonH+HyperP MoE models achieve 3.38 $\times$ CEL over the dense model baselines. The exact values are reported in Table 12 .

Figure 10: Stability metrics across training for MoE models at depths $d\in\{8,12,16,20\}$ under the same transferred LR. All metrics are bounded and non-increasing with scales.

Figure 11: Relative error in optimal LR (Left) and optimal loss (Right) estimates vs. number of sweep points. The exact values are reported in Table 14 .

Figure 12: Small-scale LR sweeps at $d{=}8$ , 10.4B tokens. Left: Dense attention normalization variants. GA QK-Norm achieves the lowest loss with a slightly shifted optimal LR. We exclude the LR ${=}0.02$ data points for dense models because the large learning rate leads to phase changes that harm fitting goodness. Right: MoE architecture variants. SharedExp + SqrtGate achieves the best loss while all variants maintain similar optimal LRs ( $\eta^{*}\approx 0.0135$ – $0.0137$ ). The exact values are reported in Table 15 and Table 16 .

Figure 13: Dense architecture scaling. Left: Loss vs. FLOPs with power-law fits $L=A\cdot C^{-b}$ . Right: Compute Efficiency Leverage (CEL) over the baseline.

Figure 14: MoE architecture scaling. Left: Loss vs. FLOPs with power-law fits. All properly-tuned variants (LR ${=}0.014$ ) outperform the lower-LR baseline. Right: Compute efficiency leverage over the SharedExp+SqrtGate (LR ${=}0.01$ ) baseline.

Figure 15: Stability comparison of architecture ablations. Left: Router $Z$ -values for MoE variants at $d{=}16$ . Router logits are exploding without SqrtGate. Right: MLP output RMS for dense variants at $d{=}20$ . QKNorm and Baseline exhibit a large spike at around 110B training tokens, while GatedAttn+QKNorm maintains the most stable RMS throughout training.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Training Tokens	Fitted $\eta^{*}$	Fitted Min Loss
10.4B	0.01515	2.4741
20.8B	0.01208	2.4189
41.6B	0.00958	2.3773
83.2B	0.00772	2.3456
166.4B	0.00635	2.3214

Appendix

Table 4: Optimal LR vs. training token budget under fine-grid sweeping with quadratic fitting.

Depth ( $d$ )	Params	$\eta{=}0.002$	$\eta{=}0.004$	$\eta{=}0.006$	$\eta{=}0.008$	$\eta{=}0.010$	$\eta{=}0.012$	$\eta{=}0.014$	$\eta{=}0.016$	$\eta{=}0.018$	$\eta{=}0.020$	Optimal $\eta$
8	208M	2.684	2.569	2.521	2.498	2.485	2.474	2.470	2.469	2.473	2.492	0.016
12	570M	2.523	2.405	2.355	2.328	2.315	2.308	2.309	2.319	2.351	2.386	0.012
16	1.24B	2.426	2.307	2.256	2.230	2.220	2.225	2.251	2.288	2.299	2.315	0.010
20	2.31B	2.354	2.235	2.189	2.166	2.169	2.191	2.218	2.235	2.246	2.264	0.008
24	3.90B	2.300	2.183	2.140	2.126	2.142	2.165	2.184	2.196	2.212	2.221	0.008

Appendix

Table 5: Validation loss vs. LR across model depth at a fixed token budget of 10.4B without Depth-

\mu

Depth ( $d$ )	Params	$\eta{=}0.002$	$\eta{=}0.004$	$\eta{=}0.006$	$\eta{=}0.008$	$\eta{=}0.010$	$\eta{=}0.012$	$\eta{=}0.014$	$\eta{=}0.016$	$\eta{=}0.018$	$\eta{=}0.020$	Optimal $\eta$
8	208M	2.682	2.568	2.520	2.496	2.484	2.476	2.473	2.474	2.477	2.479	0.014
12	570M	2.568	2.437	2.377	2.347	2.331	2.319	2.316	2.315	2.317	2.321	0.016
16	1.24B	2.495	2.359	2.297	2.264	2.245	2.235	2.229	2.225	2.225	2.234	0.016
20	2.31B	2.445	2.309	2.246	2.211	2.188	2.177	2.171	2.169	2.172	2.188	0.016
24	3.90B	2.413	2.272	2.208	2.172	2.150	2.137	2.132	2.132	2.137	2.152	0.014

Appendix

Table 6: Validation loss vs. LR across model depth at a fixed token budget of 10.4B with Depth-

\mu

Depth ( $d$ )	w/ Depth- $\mu$ P $\eta^{*}$	w/ Depth- $\mu$ P Loss	w/o Depth- $\mu$ P $\eta^{*}$	w/o Depth- $\mu$ P Loss
8	0.014	2.4734	0.016	2.4693
12	0.016	2.3150	0.012	2.3079
16	0.016	2.2250	0.010	2.2196
20	0.016	2.1690	0.008	2.1656
24	0.014	2.1320	0.008	2.1263

Appendix

Table 7: Optimal LR and loss vs. model depth at a fixed training token of 10.4B, comparing with and without Depth-

\mu

P. Depth-

\mu

P transfers the optimal LR across parameter size.

Batch Size	Fitted $\eta^{*}$	Fitted Min Loss
256K	0.00504	2.4711
512K	0.00706	2.4697
1M	0.01056	2.4700
2M	0.01562	2.4741

Appendix

Table 8: Optimal LR vs. batch size for dense models. The minimum loss is remarkably stable (within 0.004 nats), indicating all tested batch sizes are below the critical batch size.

$\gamma$	$\eta{=}0.004$	$\eta{=}0.008$	$\eta{=}0.01$	$\eta{=}0.012$	$\eta{=}0.02$	Best Loss
$10^{-3}$	2.427	2.350	2.340	2.334	2.349	2.334
$10^{-2}$	2.431	2.354	2.340	2.336	2.346	2.336
$10^{-1}$	2.427	2.350	2.337	2.332	2.346	2.332

Appendix

Table 9: Validation loss vs. LR across auxiliary loss weights. The optimal LR (

\eta^{*}=0.012

) and achievable loss are stable across a

100\times

range of

\gamma

Sparsity ( $S$ )	Fitted $\eta^{*}$	Fitted Min Loss
1	0.0163	2.4766
2	0.0162	2.4236
4	0.0145	2.3705
8	0.0139	2.3262
16	0.0124	2.2861
32	0.0115	2.2529

Appendix

Table 10: Optimal LR and loss vs. MoE sparsity. The LR varies mildly (0.012–0.016) across a

32\times

range.

Top- $k$	w/o SqrtGate $\eta^{*}$	w/o SqrtGate Loss	w/ SqrtGate $\eta^{*}$	w/ SqrtGate Loss
2	0.0140	2.4306	0.0139	2.4131
4	0.0132	2.3263	0.0139	2.3262
8	0.0137	2.3220	0.0135	2.3156
16	0.0126	2.3178	0.0129	2.3111
32	0.0127	2.3186	0.0131	2.3096
64	0.0122	2.3244	0.0128	2.3154

Appendix

Table 11: Optimal LR and loss across top-

k

values, with and without SqrtGate.

Depth	FLOPs	Muon	MuonH+HyperP	MuonH
8	$2.14\times 10^{19}$	2.4777	2.4804	2.4845
12	$1.49\times 10^{20}$	2.2257	2.2192	2.2099
16	$6.59\times 10^{20}$	2.0671	2.0526	2.0500
20	$2.19\times 10^{21}$	1.9591	1.9311	1.9558
24	$5.96\times 10^{21}$	1.8785	1.8365	1.9015

Appendix

Table 12: Validation loss vs. FLOPs. MuonH + HyperP increasingly outperforms both alternatives at larger scale.

Depth	FLOPs	MuonH+HyperP	MuonH
8	$2.14\times 10^{19}$	$0.99\times$	$0.96\times$
12	$1.49\times 10^{20}$	$1.04\times$	$1.19\times$
16	$6.59\times 10^{20}$	$1.16\times$	$1.17\times$
20	$2.19\times 10^{21}$	$1.35\times$	$0.99\times$
24	$5.96\times 10^{21}$	$1.58\times$	$0.70\times$

Appendix

Table 13: Compute efficiency leverage over Muon. MuonH + HyperP’s advantage grows monotonically with scale.

	LR Relative Error (%)					Loss Relative Error (%)
Tokens	$n{=}3$	$n{=}4$	$n{=}5$	$n{=}6$	$n{=}7$	$n{=}3$	$n{=}4$	$n{=}5$	$n{=}6$	$n{=}7$
10.4B	5.87	5.08	4.09	2.88	1.55	0.07	0.05	0.04	0.03	0.01
20.8B	3.68	2.01	1.46	1.00	0.53	0.04	0.02	0.01	0.01	0.01
41.6B	4.27	2.55	1.58	0.95	0.46	0.03	0.02	0.01	0.01	0.01
83.2B	5.43	3.06	1.88	1.12	0.54	0.05	0.02	0.02	0.01	0.01
166.4B	8.07	1.97	0.88	0.48	0.28	0.14	0.02	0.01	0.01	0.00

Appendix

Table 14: Mean relative error (%) of optimal LR and optimal loss estimates as a function of the number of sweep points

n

Method	Fitted $\eta^{*}$	Min Loss
GA QK-Norm	0.0158	2.4727
QK-Norm	0.0151	2.4823
Baseline	0.0149	2.4960

Appendix

Table 15: QK-Norm ablation at

d{=}8

. GA QK-Norm achieves the best loss while maintaining a similar optimal LR.

Method	Fitted $\eta^{*}$	Min Loss	$\Delta$ vs. Best
SharedExp + SqrtGate	0.0135	2.3154	—
SqrtGate	0.0135	2.3210	$+0.006$
SharedExp	0.0137	2.3215	$+0.006$

Appendix

Table 16: MoE architecture ablation at

d{=}8

S{=}8

k{=}8

, 10.4B tokens. SqrtGate and the shared expert provide complementary gains.

Language Model Scaling Laws

Language model scaling laws describe how model size, training data, compute, and inference effort jointly shape loss, capabilities, and efficiency.

See all 138 papers

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
May 20, 2026Dayal Singh Kalra, Maissam BarkeshliLanguage Model Scaling Laws Hyperparameter Transfer
Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park +1
Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
Mar 16, 2026Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang +3LMO Methods Scaling Laws
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · ETH Zürich, Zürich, Switzerland · Mila, Quebec AI Institute & Université de Montréal, Quebec, Canada +1
Hyperparameter Scaling Laws Across MoE Sparsity
Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling Laws Sparse Mixture-of-Experts
Ling Team, Ant Group

May 20, 2026J/K move · Enter open · S save

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

Dayal Singh Kalra, Maissam Barkeshli

Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park+1

Language Model Scaling Laws Hyperparameter Transfer LLM Training

Open paper AskPDF arXiv

Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameterization, such as Maximal Update (

μ

P), that renders optimal hyperparameters approximately scale invariant. In this paper, we first develop a framework to quantify hyperparameter transfer through three metrics: (1) the quality of the scaling law fit, (2) the robustness to extrapolation errors, and (3) the asymptotic loss penalty due to choice of parameterization. Next, we investigate through a comprehensive series of ablations why

μ

P appears to offer high-quality learning rate transfer relative to standard parameterization (SP), as existing theory is inadequate. We find that the overwhelming benefit of

μ

P relative to SP when training with AdamW arises simply from maximizing the learning rate of the embedding layer. In SP, the embedding layer learning rate acts as a bottleneck that induces training instabilities; increasing it by a factor of width to match

μ

P dramatically smooths out training while improving hyperparameter transfer. We also find that weight decay improves the scaling law fits, while, in the fixed token-per-parameter setting, it hurts the robustness of the extrapolation.

Abstract

Language Model Scaling Laws

Explore similar work

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

Deriving Hyperparameter Scaling Laws via Modern Optimization Theory

Hyperparameter Scaling Laws Across MoE Sparsity