Fantastic Pretraining Optimizers and Where to Find Them

Kaiyue Wen; David Hall; Tengyu Ma; Percy Liang

cs.LGSep 2, 2025

Fantastic Pretraining Optimizers and Where to Find Them

Authors: Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang

Organizations: Stanford University

Abstract

AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.

Figures & tables

Panel of Figure 1 — Figure 1: Top Left : The commonly used AdamW baseline for optimizer design is under-tuned. Up to a 2 $\times$ speedup is achievable by tuning a single hyperparameter (learning rate) in the GPT-3 recipe Brown et al. (2020) for a 100M model (adopted in Liu et al. (2024a) ; Wen et al. (2024) ; Yuan et al. (2025) ; Liang et al. (2025) ; Wang et al. (2025) ), highlighting the importance of proper hyperparameter optimization. Top Right : Fixing hyperparameters across optimizers does not guarantee fair comparison. Shared hyperparameters such as learning rate and weight decay are commonly set to a constant in previous studies. However, even conceptually similar optimizers may correspond to very different optimal hyperparameters. Bottom Left : Speedup decays with model size. While some optimizers show high (1.3-1.4 $\times$ ) speedup over AdamW on models under 1B parameters, the speedup decays with model size to only 1.1 $\times$ for 1.2B parameters. Bottom Right : Matrix-based optimizers consistently outperform scalar-based optimizers. The loss curves for three scalar-based optimizers (AdamW, Nesterov AdamW, Mars) and three matrix-based optimizers (Kron, Soap, Muon) trained with different Chinchilla ratios of data are shown. Matrix-based optimizers achieve a consistent speedup over scalar-based optimizers. Furthermore, the three matrix-based optimizers converge to a similar loss in an overtrained setting.

Baseline
Optimizer	References	Algorithm
AdamW	Kingma and Ba (2017) ; Loshchilov and Hutter (2019)	Algorithm 1
Variance-reduced AdamW Variants
NadamW	Dozat (2016)	Algorithm 2
Mars	Yuan et al. (2025)	Algorithm 5
Cautious	Liang et al. (2025) ; Wang et al. (2024)	Algorithm 7

Table 1: Optimizers under study

Model	Params	Seq Len	Hidden Dim	Inter Dim	# Layers	# Heads
Llama-130M	130M	4096	512	2048	32	8
Llama-300M	300M	4096	768	3072	32	12
Llama-520M	520M	4096	1024	4096	32	16
Llama-1.2B	1.2B	4096	1536	6144	32	24

Table 2: Detailed architecture hyperparameters for each model size we studied.

Stage	LR	WD	Warmup	Max Grad Norm	Batch	Val. Loss
Init	0.008	0.1	1000	1	256	3.298
Round 1	0.008	0.1	2000	1	256	3.282
Round 2	0.008	0.1	2000	1	128	3.263
Best	0.008	0.1	2000	2	128	3.263

Table 3: Illustrative coordinate-descent steps for AdamW on the 130M 1× Chinchilla regime. Changed hyperparameter values are highlighted in red; We omitted some unchanged hyperparameters (

\beta_{1}=0.9,\beta_{2}=0.98,\epsilon=10^{-10}

Optimizer	Scaling-Sensitive Hyperparameters
AdamW	learning rate, warmup, weight decay, batch size
Nesterov AdamW	learning rate, warmup
Lion	learning rate, beta2
Adam-mini	learning rate, weight decay, warmup
Cautious	learning rate, batch size
Mars	learning rate, warmup, beta1

Table 4: Scaling-sensitive hyperparameters identified in Phase I ( Section 3.2 ). To reduce the memory required by Soap, we apply the parameter blocking as in Anil et al. (2021)

Panel of Figure 2 — Figure 2: Main Results For Phase I & II. Top: We plot the validation loss on C4/EN for the experiments in Phase I and Phase II. Every point corresponds to the optimal loss achieved at the corresponding Chinchilla ratio for each optimizer. Bottom: we plot the HellaSwag performance corresponding to the selected run for a subset of optimizers: the AdamW baseline, the top 2 most performant scalar-based optimizers, and the top 3 most performant matrix-based optimizers. Analysis is deferred to Section 4.1 .

Figure 3: Speedup of different optimizers across scale. We estimate the speedup of different optimizers by fitting a scaling law for AdamW and then map the loss of different optimizers to the corresponding equivalent data budget. We observe that (i) The highest speedup is capped at 1.4

\times

; (ii) matrix-based optimizers consistently outperform scalar-based optimizers and show an increasing speedup with data budget.

Figure 4: Case Studies. Left: Validation loss scaling on 1.2B model for AdamW, NAdamW, Muon and Soap. Muon and Soap still offer significant speedup over AdamW but no longer significantly speed up over NAdamW. Mid: Estimated speedup ratio with the same methodology Figure 3 , we observe that Muon and Soap’s speedup decays with model size to only

1.1\times

. Last: Experiment with 300M

16\times

Chinchilla setting, Soap outperforms Muon when data-to-model ratio further increases.

Optimizer	LAMBADA	OpenBook	Wino	PIQA	BoolQ	WSC273	Hella	ARC-C.	ARC-E	COPA	Avg
8x Chinchilla (193B)
AdamW	67.16	41.40	64.96	76.12	68.59	82.78	67.56	43.43	74.49	85.00	67.15
NAdamW	67.84	40.20	64.80	77.15	68.10	83.52	67.40	43.34	73.61	81.00	66.70
Muon	67.53	39.80	67.09	77.09	68.81	80.95	67.67	43.34	73.53	84.00	66.98

Table 5: Benchmark performance of 1.2B models with different optimizers and Chinchilla scaling.

Panel of Figure 5 — Figure 5: Necessity of Careful Tuning. Left: 2 $\times$ the optimal learning rate diminishes Soap’s loss improvement over Mars on 520M 8x experiment; Mid: Variation of loss when only one hyperparameter differs from optimal learning rate and runs converge to within $0.02$ of optimal. The order of optimizers may flip arbitrarily if rigorous tuning is missing. Right: Changing a single hyperparameter like weight decay may lead to misleading faster loss improvement but plateaus later.

Figure 6: Common Phenomena Across Optimizers. Left : Learning rate used for different optimizers. Middle Left : Parameter norm of all optimizers shows a similar trend of increment and decrease, closely aligning the increasing and decaying of learning rate schedule. Middle Right : Gradient norm increases during learning rate decay. However, this increase does not lead to a loss increase. Right : The training loss and evaluation loss follows the same trend for all optimizers.

Appendix figures & tables191 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Algorithm 2 Nesterov AdamW

Panel of Figure 15 — Algorithm 4 Sophia-H

Panel of Figure 17 — Algorithm 6 Adam-mini

Panel of Figure 18 — Algorithm 7 Cautious

Panel of Figure 21 — Algorithm 10 PSGD Kron

Figure 7: Sophia Experiments. Left: Loss curve of Sophia and AdamW in

1\times

Chinchilla setting. Right: Loss curve of Sophia and AdamW for 130M model size.

Figure 8: More Case Studies. Experiment with 130M

16\times

Chinchilla setting, SOAP outperforms Muon in the overtraining setting.

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.327	0.377	0.402	0.451
OPENBOOKQA	0.274	0.304	0.298	0.302
WINOGRANDE	0.521	0.512	0.526	0.522
PIQA	0.625	0.652	0.662	0.678
BOOLQ	0.616	0.504	0.569	0.557

Appendix

Table 6: Evaluation Performance for Mars, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.450	0.493	0.514	0.538
OPENBOOKQA	0.300	0.312	0.328	0.328
WINOGRANDE	0.534	0.557	0.549	0.561
PIQA	0.680	0.686	0.695	0.720
BOOLQ	0.480	0.533	0.612	0.608

Appendix

Table 7: Evaluation Performance for Mars, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.513	0.548	0.576	0.601
OPENBOOKQA	0.330	0.336	0.366	0.378
WINOGRANDE	0.548	0.559	0.586	0.601
PIQA	0.704	0.717	0.725	0.735
BOOLQ	0.559	0.598	0.625	0.631

Appendix

Table 8: Evaluation Performance for Mars, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.358	0.396	0.431	0.456
OPENBOOKQA	0.286	0.288	0.318	0.300
WINOGRANDE	0.515	0.518	0.530	0.541
PIQA	0.640	0.660	0.670	0.678
BOOLQ	0.571	0.504	0.583	0.452

Appendix

Table 9: Evaluation Performance for Muon, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.459	0.492	0.529	0.550
OPENBOOKQA	0.302	0.310	0.330	0.346
WINOGRANDE	0.504	0.534	0.548	0.570
PIQA	0.679	0.689	0.695	0.714
BOOLQ	0.482	0.555	0.599	0.570

Appendix

Table 10: Evaluation Performance for Muon, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.520	0.559	0.597	0.611
OPENBOOKQA	0.328	0.340	0.342	0.368
WINOGRANDE	0.560	0.567	0.594	0.590
PIQA	0.713	0.718	0.730	0.736
BOOLQ	0.590	0.555	0.633	0.613

Appendix

Table 11: Evaluation Performance for Muon, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.316	0.380	0.417	0.450
OPENBOOKQA	0.280	0.296	0.312	0.292
WINOGRANDE	0.503	0.518	0.519	0.521
PIQA	0.629	0.649	0.663	0.680
BOOLQ	0.603	0.507	0.419	0.489

Appendix

Table 12: Evaluation Performance for Lion, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.428	0.471	0.509	0.535
OPENBOOKQA	0.306	0.320	0.326	0.346
WINOGRANDE	0.506	0.530	0.532	0.564
PIQA	0.676	0.686	0.699	0.713
BOOLQ	0.486	0.490	0.609	0.593

Appendix

Table 13: Evaluation Performance for Lion, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.507	0.543	0.569	0.597
OPENBOOKQA	0.336	0.334	0.356	0.358
WINOGRANDE	0.560	0.548	0.581	0.590
PIQA	0.699	0.713	0.726	0.740
BOOLQ	0.600	0.546	0.612	0.631

Appendix

Table 14: Evaluation Performance for Lion, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.325	0.383	0.413	0.446
OPENBOOKQA	0.284	0.286	0.308	0.322
WINOGRANDE	0.515	0.507	0.546	0.507
PIQA	0.633	0.652	0.667	0.682
BOOLQ	0.591	0.569	0.459	0.557

Appendix

Table 15: Evaluation Performance for NAdamW, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.437	0.477	0.514	0.543
OPENBOOKQA	0.310	0.332	0.316	0.348
WINOGRANDE	0.520	0.536	0.549	0.564
PIQA	0.681	0.693	0.708	0.712
BOOLQ	0.539	0.564	0.520	0.612

Appendix

Table 16: Evaluation Performance for NAdamW, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.500	0.551	0.577	0.599
OPENBOOKQA	0.332	0.350	0.358	0.342
WINOGRANDE	0.553	0.571	0.594	0.597
PIQA	0.695	0.719	0.731	0.738
BOOLQ	0.613	0.606	0.639	0.618

Appendix

Table 17: Evaluation Performance for NAdamW, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.341	0.387	0.426	0.451
OPENBOOKQA	0.268	0.278	0.296	0.314
WINOGRANDE	0.520	0.522	0.515	0.543
PIQA	0.636	0.662	0.662	0.683
BOOLQ	0.533	0.530	0.575	0.557

Appendix

Table 18: Evaluation Performance for Kron, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.429	0.485	0.516	0.548
OPENBOOKQA	0.310	0.314	0.314	0.326
WINOGRANDE	0.527	0.534	0.542	0.571
PIQA	0.675	0.690	0.706	0.709
BOOLQ	0.487	0.475	0.610	0.611

Appendix

Table 19: Evaluation Performance for Kron, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.512	0.549	0.569	0.603
OPENBOOKQA	0.338	0.332	0.350	0.352
WINOGRANDE	0.536	0.567	0.575	0.601
PIQA	0.721	0.713	0.724	0.739
BOOLQ	0.529	0.560	0.537	0.559

Appendix

Table 20: Evaluation Performance for Kron, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.355	0.382	0.427	0.446
OPENBOOKQA	0.282	0.300	0.288	0.314
WINOGRANDE	0.515	0.502	0.502	0.535
PIQA	0.641	0.655	0.675	0.684
BOOLQ	0.574	0.518	0.532	0.474

Appendix

Table 21: Evaluation Performance for Scion, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.445	0.489	0.511	0.540
OPENBOOKQA	0.306	0.308	0.326	0.358
WINOGRANDE	0.531	0.550	0.555	0.566
PIQA	0.684	0.689	0.701	0.712
BOOLQ	0.576	0.610	0.574	0.585

Appendix

Table 22: Evaluation Performance for Scion, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.521	0.552	0.575	0.601
OPENBOOKQA	0.338	0.362	0.366	0.374
WINOGRANDE	0.569	0.551	0.605	0.586
PIQA	0.704	0.717	0.736	0.743
BOOLQ	0.575	0.615	0.620	0.641

Appendix

Table 23: Evaluation Performance for Scion, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.314	0.385	0.420	0.458
OPENBOOKQA	0.298	0.280	0.308	0.318
WINOGRANDE	0.515	0.520	0.519	0.508
PIQA	0.638	0.652	0.664	0.669
BOOLQ	0.580	0.512	0.512	0.550

Appendix

Table 24: Evaluation Performance for Cautious, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.437	0.479	0.510	0.544
OPENBOOKQA	0.316	0.314	0.326	0.328
WINOGRANDE	0.515	0.533	0.530	0.576
PIQA	0.677	0.699	0.707	0.711
BOOLQ	0.547	0.492	0.584	0.574

Appendix

Table 25: Evaluation Performance for Cautious, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.509	0.549	0.578	0.604
OPENBOOKQA	0.318	0.338	0.342	0.364
WINOGRANDE	0.548	0.575	0.571	0.605
PIQA	0.705	0.715	0.733	0.739
BOOLQ	0.601	0.611	0.582	0.606

Appendix

Table 26: Evaluation Performance for Cautious, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.344	0.404	0.434	0.458
OPENBOOKQA	0.280	0.270	0.300	0.318
WINOGRANDE	0.511	0.510	0.523	0.517
PIQA	0.643	0.652	0.666	0.676
BOOLQ	0.569	0.491	0.519	0.559

Appendix

Table 27: Evaluation Performance for SOAP, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.001	0.492	0.518	0.541
OPENBOOKQA	0.256	0.324	0.324	0.334
WINOGRANDE	0.506	0.534	0.546	0.555
PIQA	0.546	0.689	0.706	0.711
BOOLQ	0.402	0.593	0.565	0.593

Appendix

Table 28: Evaluation Performance for SOAP, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.513	0.555	0.582	0.606
OPENBOOKQA	0.308	0.312	0.368	0.350
WINOGRANDE	0.548	0.562	0.594	0.583
PIQA	0.695	0.714	0.729	0.739
BOOLQ	0.563	0.623	0.596	0.643

Appendix

Table 29: Evaluation Performance for SOAP, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.322	0.380	0.410	0.442
OPENBOOKQA	0.306	0.290	0.298	0.292
WINOGRANDE	0.506	0.519	0.515	0.530
PIQA	0.632	0.663	0.665	0.674
BOOLQ	0.568	0.415	0.553	0.576

Appendix

Table 30: Evaluation Performance for AdamW, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.435	0.484	0.500	0.537
OPENBOOKQA	0.300	0.312	0.328	0.318
WINOGRANDE	0.519	0.530	0.542	0.552
PIQA	0.673	0.691	0.697	0.713
BOOLQ	0.454	0.496	0.508	0.602

Appendix

Table 31: Evaluation Performance for AdamW, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.500	0.540	0.570	0.591
OPENBOOKQA	0.320	0.336	0.342	0.356
WINOGRANDE	0.539	0.586	0.583	0.601
PIQA	0.705	0.719	0.732	0.739
BOOLQ	0.555	0.615	0.596	0.609

Appendix

Table 32: Evaluation Performance for AdamW, Model Size = 520m

DATA SIZE	2B	5B	10B	21B
PERFORMANCE METRIC
LAMBADA OPENAI	0.317	0.369	0.412	0.444
OPENBOOKQA	0.286	0.282	0.282	0.302
WINOGRANDE	0.525	0.500	0.515	0.528
PIQA	0.640	0.655	0.662	0.670
BOOLQ	0.506	0.496	0.445	0.583

Appendix

Table 33: Evaluation Performance for Adam-Mini, Model Size = 130m

DATA SIZE	6B	12B	24B	48B
PERFORMANCE METRIC
LAMBADA OPENAI	0.429	0.468	0.509	0.535
OPENBOOKQA	0.300	0.318	0.318	0.350
WINOGRANDE	0.502	0.528	0.545	0.586
PIQA	0.688	0.693	0.683	0.701
BOOLQ	0.480	0.546	0.609	0.573

Appendix

Table 34: Evaluation Performance for Adam-Mini, Model Size = 300m

DATA SIZE	10B	21B	42B	85B
PERFORMANCE METRIC
LAMBADA OPENAI	0.500	0.538	0.576	0.604
OPENBOOKQA	0.306	0.326	0.330	0.368
WINOGRANDE	0.534	0.561	0.571	0.594
PIQA	0.707	0.721	0.723	0.733
BOOLQ	0.543	0.626	0.619	0.536

Appendix

Table 35: Evaluation Performance for Adam-Mini, Model Size = 520m

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-20	0.008	1	0	128	2000	0.1	3.529	0
0.95	–	–	–	–	–	–	–	–	3.539	1
0.98	–	–	–	–	–	–	–	–	3.882	2
–	0.9	–	–	–	–	–	–	–	3.545	3
–	0.95	–	–	–	–	–	–	–	3.535	4
–	–	1e-25	–	–	–	–	–	–	3.529	5

Appendix

Table 36: Hyperparameter ablation for AdamW on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-20	0.008	1	0	128	2000	0.1	3.409	0
0.95	–	–	–	–	–	–	–	–	3.417	1
0.98	–	–	–	–	–	–	–	–	7.557	2
–	0.9	–	–	–	–	–	–	–	3.423	3
–	0.95	–	–	–	–	–	–	–	3.413	4
–	–	1e-25	–	–	–	–	–	–	3.409	5

Appendix

Table 37: Hyperparameter ablation for AdamW on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-20	0.008	1	0	128	2000	0.1	3.322	0
0.95	–	–	–	–	–	–	–	–	3.330	1
0.98	–	–	–	–	–	–	–	–	3.416	2
–	0.9	–	–	–	–	–	–	–	3.338	3
–	0.95	–	–	–	–	–	–	–	3.329	4
–	–	1e-25	–	–	–	–	–	–	3.322	5

Appendix

Table 38: Hyperparameter ablation for AdamW on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-20	0.008	1	0	256	1000	0.1	3.262	0
0.95	–	–	–	–	–	–	–	–	3.273	1
0.98	–	–	–	–	–	–	–	–	3.430	2
–	0.9	–	–	–	–	–	–	–	3.272	3
–	0.95	–	–	–	–	–	–	–	3.266	4
–	–	1e-25	–	–	–	–	–	–	3.262	5

Appendix

Table 39: Hyperparameter ablation for AdamW on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	1	0	128	2000	0.1	3.264	0
0.95	–	–	–	–	–	–	–	–	3.271	1
0.98	–	–	–	–	–	–	–	–	7.351	2
–	0.9	–	–	–	–	–	–	–	3.280	3
–	0.95	–	–	–	–	–	–	–	3.269	4
–	–	1e-25	–	–	–	–	–	–	3.265	5

Appendix

Table 40: Hyperparameter ablation for AdamW on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	256	1000	0.2	3.110	0
0.95	–	–	–	–	–	–	–	–	3.112	1
0.98	–	–	–	–	–	–	–	–	3.229	2
–	0.9	–	–	–	–	–	–	–	3.116	3
–	0.95	–	–	–	–	–	–	–	3.111	4
–	–	1e-25	–	–	–	–	–	–	3.116	5

Appendix

Table 41: Hyperparameter ablation for AdamW on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-15	0.008	1	0	128	2000	0.1	3.535	0
0.8	–	–	–	–	–	–	–	–	6.698	1
0.9	–	–	–	–	–	–	–	–	3.549	2
0.98	–	–	–	–	–	–	–	–	3.534	3
–	0.9	–	–	–	–	–	–	–	3.551	4
–	0.95	–	–	–	–	–	–	–	3.543	5

Appendix

Table 42: Hyperparameter ablation for Cautious on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-15	0.008	2	0	128	2000	0.1	3.403	0
0.8	–	–	–	–	–	–	–	–	7.417	1
0.9	–	–	–	–	–	–	–	–	3.427	2
0.95	–	–	–	–	–	–	–	–	3.413	3
–	0.9	–	–	–	–	–	–	–	>10	4
–	0.95	–	–	–	–	–	–	–	3.408	5

Appendix

Table 43: Hyperparameter ablation for Cautious on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-15	0.008	2	0	256	2000	0.1	3.253	0
0.8	–	–	–	–	–	–	–	–	7.395	1
0.9	–	–	–	–	–	–	–	–	3.271	2
0.95	–	–	–	–	–	–	–	–	3.259	3
–	0.9	–	–	–	–	–	–	–	>10	4
–	0.95	–	–	–	–	–	–	–	3.253	5

Appendix

Table 44: Hyperparameter ablation for Cautious on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	2	0	128	2000	0.1	3.260	0
0.9	–	–	–	–	–	–	–	–	3.286	1
0.95	–	–	–	–	–	–	–	–	3.271	2
–	0.9	–	–	–	–	–	–	–	>10	3
–	0.95	–	–	–	–	–	–	–	>10	4
–	–	1e-25	–	–	–	–	–	–	3.260	5

Appendix

Table 45: Hyperparameter ablation for Cautious on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	1	0	256	2000	0.1	3.100	0
0.8	–	–	–	–	–	–	–	–	>10	1
0.9	–	–	–	–	–	–	–	–	7.264	2
0.95	–	–	–	–	–	–	–	–	3.108	3
–	0.9	–	–	–	–	–	–	–	>10	4
–	0.95	–	–	–	–	–	–	–	3.105	5

Appendix

Table 46: Hyperparameter ablation for Cautious on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.002	1	0	128	2000	0.7	3.552	0
0.8	–	–	–	–	–	–	–	3.575	1
0.95	–	–	–	–	–	–	–	3.557	2
0.98	–	–	–	–	–	–	–	7.644	3
–	0.9	–	–	–	–	–	–	3.550	4
–	0.98	–	–	–	–	–	–	3.630	5

Appendix

Table 47: Hyperparameter ablation for Lion on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	128	2000	0.7	3.414	0
0.8	–	–	–	–	–	–	–	3.436	1
0.95	–	–	–	–	–	–	–	3.418	2
0.98	–	–	–	–	–	–	–	7.313	3
–	0.9	–	–	–	–	–	–	3.433	4
–	0.98	–	–	–	–	–	–	3.409	5

Appendix

Table 48: Hyperparameter ablation for Lion on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	128	2000	0.7	3.331	0
0.8	–	–	–	–	–	–	–	3.345	1
0.95	–	–	–	–	–	–	–	3.335	2
0.98	–	–	–	–	–	–	–	7.330	3
–	0.9	–	–	–	–	–	–	3.345	4
–	0.98	–	–	–	–	–	–	3.340	5

Appendix

Table 49: Hyperparameter ablation for Lion on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	0.001	1	0	128	2000	0.7	3.252	0
0.8	–	–	–	–	–	–	–	3.287	1
0.95	–	–	–	–	–	–	–	3.264	2
0.98	–	–	–	–	–	–	–	3.286	3
–	0.9	–	–	–	–	–	–	3.277	4
–	0.95	–	–	–	–	–	–	3.263	5

Appendix

Table 50: Hyperparameter ablation for Lion on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	128	2000	0.6	3.268	0
0.8	–	–	–	–	–	–	–	3.283	1
0.95	–	–	–	–	–	–	–	3.271	2
0.98	–	–	–	–	–	–	–	7.690	3
–	0.9	–	–	–	–	–	–	3.283	4
–	0.98	–	–	–	–	–	–	3.271	5

Appendix

Table 51: Hyperparameter ablation for Lion on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	128	2000	0.7	3.108	0
0.8	–	–	–	–	–	–	–	3.117	1
0.95	–	–	–	–	–	–	–	3.121	2
0.98	–	–	–	–	–	–	–	7.577	3
–	0.9	–	–	–	–	–	–	3.119	4
–	0.98	–	–	–	–	–	–	3.208	5

Appendix

Table 52: Hyperparameter ablation for Lion on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	1e-25	0.025	0.016	1	0	128	2000	0.1	3.537	0
0.8	–	–	–	–	–	–	–	–	–	3.568	1
0.95	–	–	–	–	–	–	–	–	–	3.548	2
0.98	–	–	–	–	–	–	–	–	–	3.586	3
–	0.9	–	–	–	–	–	–	–	–	3.548	4
–	0.98	–	–	–	–	–	–	–	–	3.536	5

Appendix

Table 53: Hyperparameter ablation for Mars on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-25	0.025	0.008	1	0	128	2000	0.1	3.396	0
0.8	–	–	–	–	–	–	–	–	–	3.421	1
0.9	–	–	–	–	–	–	–	–	–	3.402	2
0.98	–	–	–	–	–	–	–	–	–	3.401	3
–	0.9	–	–	–	–	–	–	–	–	3.409	4
–	0.95	–	–	–	–	–	–	–	–	3.400	5

Appendix

Table 54: Hyperparameter ablation for Mars on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-25	0.025	0.008	1	0	128	2000	0.1	3.323	0
0.8	–	–	–	–	–	–	–	–	–	3.336	1
0.95	–	–	–	–	–	–	–	–	–	3.326	2
0.98	–	–	–	–	–	–	–	–	–	3.350	3
–	0.9	–	–	–	–	–	–	–	–	3.337	4
–	0.95	–	–	–	–	–	–	–	–	3.330	5

Appendix

Table 55: Hyperparameter ablation for Mars on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.99	1e-25	0.025	0.008	1	0	128	2000	0.1	3.247	0
0.8	–	–	–	–	–	–	–	–	–	3.272	1
0.9	–	–	–	–	–	–	–	–	–	3.252	2
0.95	–	–	–	–	–	–	–	–	–	3.256	3
–	0.9	–	–	–	–	–	–	–	–	NaN	4
–	0.95	–	–	–	–	–	–	–	–	3.257	5

Appendix

Table 56: Hyperparameter ablation for Mars on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.05	0.008	1	0	128	1000	0.1	3.249	0
0.8	–	–	–	–	–	–	–	–	–	7.899	1
0.9	–	–	–	–	–	–	–	–	–	3.259	2
0.95	–	–	–	–	–	–	–	–	–	3.247	3
–	0.9	–	–	–	–	–	–	–	–	NaN	4
–	0.95	–	–	–	–	–	–	–	–	3.256	5

Appendix

Table 57: Hyperparameter ablation for Mars on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.95	1e-25	0.025	0.008	1	0	256	2000	0.1	3.101	0
0.8	–	–	–	–	–	–	–	–	–	>10	1
0.9	–	–	–	–	–	–	–	–	–	3.108	2
0.98	–	–	–	–	–	–	–	–	–	3.107	3
–	0.9	–	–	–	–	–	–	–	–	3.108	4
–	0.98	–	–	–	–	–	–	–	–	3.099	5

Appendix

Table 58: Hyperparameter ablation for Mars on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-25	0.008	1	0	128	2000	0.1	3.531	0
0.8	–	–	–	–	–	–	–	–	4.764	1
0.9	–	–	–	–	–	–	–	–	3.552	2
0.98	–	–	–	–	–	–	–	–	3.585	3
–	0.9	–	–	–	–	–	–	–	3.552	4
–	0.95	–	–	–	–	–	–	–	3.535	5

Appendix

Table 59: Hyperparameter ablation for NAdamW on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	128	2000	0.1	3.394	0
0.8	–	–	–	–	–	–	–	–	3.452	1
0.9	–	–	–	–	–	–	–	–	3.408	2
0.98	–	–	–	–	–	–	–	–	3.402	3
–	0.9	–	–	–	–	–	–	–	3.403	4
–	0.95	–	–	–	–	–	–	–	3.399	5

Appendix

Table 60: Hyperparameter ablation for NAdamW on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	128	2000	0.1	3.319	0
0.8	–	–	–	–	–	–	–	–	7.085	1
0.9	–	–	–	–	–	–	–	–	3.331	2
0.98	–	–	–	–	–	–	–	–	3.349	3
–	0.9	–	–	–	–	–	–	–	3.332	4
–	0.95	–	–	–	–	–	–	–	3.327	5

Appendix

Table 61: Hyperparameter ablation for NAdamW on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	128	2000	0.1	3.251	0
0.8	–	–	–	–	–	–	–	–	3.282	1
0.9	–	–	–	–	–	–	–	–	3.257	2
0.98	–	–	–	–	–	–	–	–	3.253	3
–	0.9	–	–	–	–	–	–	–	3.260	4
–	0.95	–	–	–	–	–	–	–	3.255	5

Appendix

Table 62: Hyperparameter ablation for NAdamW on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	128	2000	0.1	3.248	0
0.8	–	–	–	–	–	–	–	–	7.542	1
0.9	–	–	–	–	–	–	–	–	7.054	2
0.98	–	–	–	–	–	–	–	–	3.254	3
–	0.9	–	–	–	–	–	–	–	3.263	4
–	0.95	–	–	–	–	–	–	–	3.256	5

Appendix

Table 63: Hyperparameter ablation for NAdamW on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-15	0.008	1	0	128	2000	0.1	3.542	0
0.95	–	–	–	–	–	–	–	–	3.560	1
0.98	–	–	–	–	–	–	–	–	7.733	2
–	0.9	–	–	–	–	–	–	–	3.554	3
–	0.95	–	–	–	–	–	–	–	3.545	4
–	–	1e-25	–	–	–	–	–	–	3.546	5

Appendix

Table 64: Hyperparameter ablation for Adam-Mini on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-20	0.008	2	0	128	2000	0.1	3.416	0
0.95	–	–	–	–	–	–	–	–	3.429	1
0.98	–	–	–	–	–	–	–	–	7.520	2
–	0.9	–	–	–	–	–	–	–	3.422	3
–	0.95	–	–	–	–	–	–	–	3.419	4
–	–	1e-25	–	–	–	–	–	–	3.416	5

Appendix

Table 65: Hyperparameter ablation for Adam-Mini on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	1	0	128	2000	0.1	3.328	0
0.95	–	–	–	–	–	–	–	–	3.360	1
0.98	–	–	–	–	–	–	–	–	7.771	2
–	0.9	–	–	–	–	–	–	–	3.337	3
–	0.95	–	–	–	–	–	–	–	3.331	4
–	–	1e-25	–	–	–	–	–	–	3.331	5

Appendix

Table 66: Hyperparameter ablation for Adam-Mini on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	1	0	128	2000	0.1	3.266	0
0.95	–	–	–	–	–	–	–	–	3.290	1
0.98	–	–	–	–	–	–	–	–	7.552	2
–	0.9	–	–	–	–	–	–	–	3.281	3
–	0.95	–	–	–	–	–	–	–	3.267	4
–	–	1e-25	–	–	–	–	–	–	3.267	5

Appendix

Table 67: Hyperparameter ablation for Adam-Mini on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-25	0.004	2	0	128	2000	0.2	3.272	0
0.95	–	–	–	–	–	–	–	–	3.276	1
0.98	–	–	–	–	–	–	–	–	8.024	2
–	0.9	–	–	–	–	–	–	–	3.279	3
–	0.95	–	–	–	–	–	–	–	3.277	4
–	–	1e-25	–	–	–	–	–	–	3.272	5

Appendix

Table 68: Hyperparameter ablation for Adam-Mini on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	0	0	128	4000	0.1	3.112	0
0.95	–	–	–	–	–	–	–	–	3.113	1
0.98	–	–	–	–	–	–	–	–	7.459	2
–	0.9	–	–	–	–	–	–	–	3.120	3
–	0.95	–	–	–	–	–	–	–	3.115	4
–	–	1e-25	–	–	–	–	–	–	3.115	5

Appendix

Table 69: Hyperparameter ablation for Adam-Mini on 520m on 1x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.002	1	0	True	True	1	0.2	0.05	128	2000	1000	0.7	3.492	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.500	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	3.497	2
–	128	–	–	–	–	–	–	–	–	–	–	–	–	3.492	3
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.494	4
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.528	5

Appendix

Table 70: Hyperparameter ablation for Kron on 130m on 1x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.002	1	0	True	True	1	0.2	0.05	128	2000	1000	0.5	3.389	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.393	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	3.391	2
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.390	3
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.409	4
–	–	0.001	–	–	–	–	–	–	–	–	–	–	–	3.391	5

Appendix

Table 71: Hyperparameter ablation for Kron on 130m on 2x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.002	1	0	True	True	1	0.2	0.05	128	2000	1000	0.5	3.307	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.316	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	3.310	2
–	128	–	–	–	–	–	–	–	–	–	–	–	–	3.308	3
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.313	4
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.316	5

Appendix

Table 72: Hyperparameter ablation for Kron on 130m on 4x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.001	1	0	True	True	1	0.2	0.1	128	2000	1000	0.5	3.239	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.243	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	3.239	2
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.239	3
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.245	4
–	–	0.002	–	–	–	–	–	–	–	–	–	–	–	3.252	5

Appendix

Table 73: Hyperparameter ablation for Kron on 130m on 8x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.001	1	0	True	True	1	0.2	0.1	128	2000	1000	0.5	3.244	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.248	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	3.244	2
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.243	3
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.263	4
–	–	0.002	–	–	–	–	–	–	–	–	–	–	–	3.245	5

Appendix

Table 74: Hyperparameter ablation for Kron on 300m on 1x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.001	1	0	True	True	1	0.2	0.1	128	2000	1000	0.5	3.084	0
0.9	–	–	–	–	–	–	–	–	–	–	–	–	–	3.088	1
0.98	–	–	–	–	–	–	–	–	–	–	–	–	–	6.663	2
–	512	–	–	–	–	–	–	–	–	–	–	–	–	3.087	3
–	–	0.0005	–	–	–	–	–	–	–	–	–	–	–	3.095	4
–	–	0.002	–	–	–	–	–	–	–	–	–	–	–	6.412	5

Appendix

Table 75: Hyperparameter ablation for Kron on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	256	1e-15	0.016	1	0	True	1	0.95	128	1000	0.1	3.483	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	4.868	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	4.547	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.496	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.491	4
–	0.99	–	–	–	–	–	–	–	–	–	–	–	3.482	5

Appendix

Table 76: Hyperparameter ablation for Soap on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	256	1e-15	0.016	1	0	False	1	0.98	128	500	0.1	3.376	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	5.163	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.397	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.393	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.384	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	3.380	5

Appendix

Table 77: Hyperparameter ablation for Soap on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	256	1e-15	0.008	1	0	False	1	0.98	128	500	0.1	3.295	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	3.328	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.302	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.312	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.303	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	3.298	5

Appendix

Table 78: Hyperparameter ablation for Soap on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-15	0.008	1	0	True	10	0.98	256	1000	0.1	3.239	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	3.298	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.250	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.251	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.244	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	3.241	5

Appendix

Table 79: Hyperparameter ablation for Soap on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.9	128	1000	0.1	3.231	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	4.544	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.251	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.247	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.240	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	3.234	5

Appendix

Table 80: Hyperparameter ablation for Soap on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.95	128	1000	0.1	3.079	0
0.8	–	–	–	–	–	–	–	–	–	–	–	–	4.630	1
0.9	–	–	–	–	–	–	–	–	–	–	–	–	4.316	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	3.097	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	3.090	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	3.085	5

Appendix

Table 81: Hyperparameter ablation for Soap on 520m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0032	0.8	0.98	0.8	1e-15	0.016	linear	1	0	0.95	1e-05	128	0	0.1	3.464	0
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.474	1
0.0048	–	–	–	–	–	–	–	–	–	–	–	–	–	3.463	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.468	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.467	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.470	5

Appendix

Table 82: Hyperparameter ablation for Muon on 130m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	0.8	1e-15	0.008	linear	1	0	0.98	1e-05	128	0	0.1	3.369	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	–	–	3.385	1
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.373	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.372	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.373	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.375	5

Appendix

Table 83: Hyperparameter ablation for Muon on 130m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	0.8	1e-15	0.008	linear	1	0	0.95	1e-05	128	0	0.1	3.296	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	–	–	3.309	1
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.300	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.297	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.299	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.299	5

Appendix

Table 84: Hyperparameter ablation for Muon on 130m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	1	1e-15	0.008	linear	1	0	0.98	1e-05	128	0	0.1	3.240	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	–	–	3.249	1
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.243	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.240	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.240	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.241	5

Appendix

Table 85: Hyperparameter ablation for Muon on 130m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	0.8	1e-15	0.008	linear	1	0	0.98	1e-05	128	0	0.1	3.224	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	–	–	3.232	1
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.225	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.226	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.226	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.227	5

Appendix

Table 86: Hyperparameter ablation for Muon on 300m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	1	1e-25	0.008	linear	1	0	0.98	1e-05	128	0	0.1	3.073	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	–	–	3.076	1
0.0016	–	–	–	–	–	–	–	–	–	–	–	–	–	3.072	2
–	0.9	–	–	–	–	–	–	–	–	–	–	–	–	3.073	3
–	0.95	–	–	–	–	–	–	–	–	–	–	–	–	3.074	4
–	0.98	–	–	–	–	–	–	–	–	–	–	–	–	3.073	5

Appendix

Table 87: Hyperparameter ablation for Muon on 520m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0032	0.95	0.8	0.016	linear	1	0	0.95	1e-15	128	0	0.1	3.477	0
0.0016	–	–	–	–	–	–	–	–	–	–	–	3.477	1
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.481	2
0.0064	–	–	–	–	–	–	–	–	–	–	–	3.485	3
0.008	–	–	–	–	–	–	–	–	–	–	–	3.488	4
0.0096	–	–	–	–	–	–	–	–	–	–	–	3.490	5

Appendix

Table 88: Hyperparameter ablation for Scion on 130m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0032	0.95	1	0.016	linear	1	0	0.9	1e-15	128	0	0.1	3.379	0
0.0016	–	–	–	–	–	–	–	–	–	–	–	3.378	1
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.382	2
0.0064	–	–	–	–	–	–	–	–	–	–	–	3.384	3
0.008	–	–	–	–	–	–	–	–	–	–	–	3.386	4
0.0096	–	–	–	–	–	–	–	–	–	–	–	3.388	5

Appendix

Table 89: Hyperparameter ablation for Scion on 130m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0016	0.98	1	0.008	linear	2	0	0.9	1e-15	128	0	0.1	3.302	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	3.302	1
0.0024	–	–	–	–	–	–	–	–	–	–	–	3.306	2
0.0032	–	–	–	–	–	–	–	–	–	–	–	3.307	3
0.004	–	–	–	–	–	–	–	–	–	–	–	3.306	4
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.307	5

Appendix

Table 90: Hyperparameter ablation for Scion on 130m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0016	0.98	1	0.008	linear	2	0	0.9	1e-05	128	0	0.1	3.246	0
0.0008	–	–	–	–	–	–	–	–	–	–	–	3.245	1
0.0024	–	–	–	–	–	–	–	–	–	–	–	3.250	2
0.0032	–	–	–	–	–	–	–	–	–	–	–	3.249	3
0.004	–	–	–	–	–	–	–	–	–	–	–	3.251	4
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.252	5

Appendix

Table 91: Hyperparameter ablation for Scion on 130m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0008	0.98	0.8	0.008	linear	2	0	0.9	1e-05	128	0	0.1	3.232	0
0.0016	–	–	–	–	–	–	–	–	–	–	–	3.237	1
0.0024	–	–	–	–	–	–	–	–	–	–	–	3.242	2
0.0032	–	–	–	–	–	–	–	–	–	–	–	3.242	3
0.004	–	–	–	–	–	–	–	–	–	–	–	3.241	4
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.243	5

Appendix

Table 92: Hyperparameter ablation for Scion on 300m on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0008	0.98	1	0.008	linear	2	0	0.9	1e-05	128	0	0.1	3.080	0
0.0016	–	–	–	–	–	–	–	–	–	–	–	3.089	1
0.0024	–	–	–	–	–	–	–	–	–	–	–	3.090	2
0.0032	–	–	–	–	–	–	–	–	–	–	–	3.090	3
0.004	–	–	–	–	–	–	–	–	–	–	–	3.090	4
0.0048	–	–	–	–	–	–	–	–	–	–	–	3.091	5

Appendix

Table 93: Hyperparameter ablation for Scion on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.9	1e-07	0.0125	0.004	128	4000	0	3.544	0
0.8	–	–	–	–	–	–	–	3.581	1
0.9	–	–	–	–	–	–	–	3.546	2
0.98	–	–	–	–	–	–	–	3.624	3
–	0.95	–	–	–	–	–	–	3.546	1
–	0.98	–	–	–	–	–	–	3.553	2

Appendix

Table 94: Hyperparameter ablation for Sophia on 130m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.9	1e-07	0.0125	0.004	128	4000	0.1	3.414	0
0.8	–	–	–	–	–	–	–	3.439	1
0.9	–	–	–	–	–	–	–	3.416	2
0.98	–	–	–	–	–	–	–	3.469	3
–	0.95	–	–	–	–	–	–	3.414	1
–	0.98	–	–	–	–	–	–	3.418	2

Appendix

Table 95: Hyperparameter ablation for Sophia on 130m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	1e-07	0.0125	0.004	128	4000	0.2	3.330	0
0.8	–	–	–	–	–	–	–	3.351	1
0.9	–	–	–	–	–	–	–	3.332	2
0.98	–	–	–	–	–	–	–	3.371	3
–	0.9	–	–	–	–	–	–	3.328	1
–	0.95	–	–	–	–	–	–	3.328	2

Appendix

Table 96: Hyperparameter ablation for Sophia on 130m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.95	1e-07	0.0125	0.002	128	4000	0.2	3.259	0
0.8	–	–	–	–	–	–	–	3.291	1
0.9	–	–	–	–	–	–	–	3.265	2
–	0.9	–	–	–	–	–	–	3.259	1
–	0.98	–	–	–	–	–	–	3.260	2
–	0.99	–	–	–	–	–	–	3.260	3

Appendix

Table 97: Hyperparameter ablation for Sophia on 130m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.9	1e-07	0.0125	0.004	128	4000	0.1	3.267	0
0.8	–	–	–	–	–	–	–	3.288	1
0.95	–	–	–	–	–	–	–	7.390	2
0.98	–	–	–	–	–	–	–	7.525	3
–	0.95	–	–	–	–	–	–	3.270	1
–	0.98	–	–	–	–	–	–	3.275	2

Appendix

Table 98: Hyperparameter ablation for Sophia on 300m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.9	1e-07	0.0125	0.002	128	4000	0.3	3.106	0
0.8	–	–	–	–	–	–	–	3.125	1
0.9	–	–	–	–	–	–	–	3.109	2
0.98	–	–	–	–	–	–	–	3.133	3
–	0.95	–	–	–	–	–	–	3.111	1
–	0.98	–	–	–	–	–	–	3.111	2

Appendix

Table 99: Hyperparameter ablation for Sophia on 520m on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-15	0.008	1	0	128	2000	0.1	3.166	0
–	–	–	0.004	–	–	–	–	–	3.167	1
–	–	–	–	–	–	256	–	–	3.170	2
–	–	–	–	–	–	–	–	0.2	3.183	3

Appendix

Table 100: Hyperparameter ablation for AdamW on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	2	0	256	2000	0.1	3.094	0
–	–	–	0.004	–	–	–	–	–	3.101	1
–	–	–	–	–	–	128	–	–	3.103	2
–	–	–	–	–	–	–	–	0.2	3.103	3

Appendix

Table 101: Hyperparameter ablation for AdamW on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	2	0	256	2000	0.1	3.043	0
–	–	–	0.004	–	–	–	–	–	3.042	1
–	–	–	–	–	–	128	–	–	3.057	2
–	–	–	–	–	–	–	–	0.2	3.059	3

Appendix

Table 102: Hyperparameter ablation for AdamW on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	256	1000	0.2	3.023	0
–	–	–	–	–	–	128	–	–	6.654	1
–	–	–	–	–	–	–	–	0.1	3.025	2

Appendix

Table 103: Hyperparameter ablation for AdamW on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	256	1000	0.1	2.958	0
–	–	–	0.008	–	–	–	–	–	7.075	1
–	–	–	–	–	–	128	–	–	7.139	2
–	–	–	–	–	–	–	–	0.2	2.962	3

Appendix

Table 104: Hyperparameter ablation for AdamW on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	256	1000	0.1	2.913	0
–	–	–	0.008	–	–	–	–	–	7.183	1
–	–	–	–	–	–	128	–	–	6.932	2
–	–	–	–	–	–	–	–	0.2	2.921	3

Appendix

Table 105: Hyperparameter ablation for AdamW on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	1	0	256	2000	0.1	3.165	0
–	–	–	0.004	–	–	–	–	–	3.175	1
–	–	–	–	–	–	128	–	–	3.171	2

Appendix

Table 106: Hyperparameter ablation for Cautious on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	1	0	256	2000	0.1	3.094	0
–	–	–	0.004	–	–	–	–	–	3.098	1
–	–	–	–	–	–	128	–	–	3.114	2

Appendix

Table 107: Hyperparameter ablation for Cautious on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	1	0	256	2000	0.1	3.043	0
–	–	–	0.004	–	–	–	–	–	3.041	1
–	–	–	–	–	–	128	–	–	3.071	2

Appendix

Table 108: Hyperparameter ablation for Cautious on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.008	1	0	256	2000	0.1	3.017	0
–	–	–	0.004	–	–	–	–	–	3.019	1
–	–	–	–	–	–	128	–	–	>10	2

Appendix

Table 109: Hyperparameter ablation for Cautious on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.004	1	0	256	2000	0.1	2.956	0
–	–	–	0.008	–	–	–	–	–	2.959	1
–	–	–	–	–	–	128	–	–	2.971	2

Appendix

Table 110: Hyperparameter ablation for Cautious on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-25	0.004	1	0	256	2000	0.1	2.910	0
–	–	–	0.008	–	–	–	–	–	2.919	1
–	–	–	–	–	–	128	–	–	2.931	2

Appendix

Table 111: Hyperparameter ablation for Cautious on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	0.001	1	0	128	2000	0.7	3.170	0
–	0.9	–	–	–	–	–	–	3.189	1
–	0.95	–	–	–	–	–	–	3.175	2
–	–	0.0005	–	–	–	–	–	3.172	3
–	–	–	–	–	256	–	–	3.183	4

Appendix

Table 112: Hyperparameter ablation for Lion on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	0.001	1	0	256	2000	0.7	3.100	0
–	0.9	–	–	–	–	–	–	3.114	1
–	0.95	–	–	–	–	–	–	3.103	2
–	–	0.0005	–	–	–	–	–	3.105	3
–	–	–	–	–	128	–	–	3.104	4

Appendix

Table 113: Hyperparameter ablation for Lion on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	0.001	1	0	256	2000	0.7	3.046	0
–	0.9	–	–	–	–	–	–	3.058	1
–	0.95	–	–	–	–	–	–	3.050	2
–	–	0.0005	–	–	–	–	–	3.043	3
–	–	–	–	–	128	–	–	3.061	4

Appendix

Table 114: Hyperparameter ablation for Lion on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	128	2000	0.6	3.029	0
–	0.9	–	–	–	–	–	–	3.045	1
–	0.98	–	–	–	–	–	–	7.779	2
–	–	0.0005	–	–	–	–	–	3.028	3
–	–	–	–	–	256	–	–	3.030	4

Appendix

Table 115: Hyperparameter ablation for Lion on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.001	1	0	256	2000	0.6	2.965	0
–	0.9	–	–	–	–	–	–	2.971	1
–	0.98	–	–	–	–	–	–	2.965	2
–	–	0.0005	–	–	–	–	–	2.966	3
–	–	–	–	–	128	–	–	2.975	4

Appendix

Table 116: Hyperparameter ablation for Lion on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.95	0.0005	1	0	256	2000	0.6	2.915	0
–	0.9	–	–	–	–	–	–	2.922	1
–	0.98	–	–	–	–	–	–	2.915	2
–	–	0.001	–	–	–	–	–	2.920	3
–	–	–	–	–	128	–	–	2.922	4

Appendix

Table 117: Hyperparameter ablation for Lion on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.99	1e-25	0.05	0.008	1	0	128	1000	0.1	3.158	0
0.9	–	–	–	–	–	–	–	–	–	3.174	1
0.95	–	–	–	–	–	–	–	–	–	3.156	2
–	–	–	–	0.004	–	–	–	–	–	3.168	3
–	–	–	–	–	–	–	256	–	–	3.171	4
–	–	–	–	–	–	–	–	2000	–	3.159	5

Appendix

Table 118: Hyperparameter ablation for Mars on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.99	1e-25	0.05	0.008	1	0	128	1000	0.1	3.097	0
0.9	–	–	–	–	–	–	–	–	–	3.111	1
0.95	–	–	–	–	–	–	–	–	–	3.096	2
–	–	–	–	0.004	–	–	–	–	–	3.095	3
–	–	–	–	–	–	–	256	–	–	3.098	4
–	–	–	–	–	–	–	–	2000	–	3.097	5

Appendix

Table 119: Hyperparameter ablation for Mars on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.99	1e-25	0.05	0.008	1	0	256	1000	0.1	3.040	0
0.9	–	–	–	–	–	–	–	–	–	3.049	1
0.95	–	–	–	–	–	–	–	–	–	3.038	2
–	–	–	–	0.004	–	–	–	–	–	3.046	3
–	–	–	–	–	–	–	128	–	–	3.050	4
–	–	–	–	–	–	–	–	2000	–	3.049	5

Appendix

Table 120: Hyperparameter ablation for Mars on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-25	0.025	0.008	1	0	256	2000	0.1	3.015	0
0.9	–	–	–	–	–	–	–	–	–	3.019	1
0.98	–	–	–	–	–	–	–	–	–	3.014	2
–	–	–	–	0.004	–	–	–	–	–	3.019	3
–	–	–	–	–	–	–	128	–	–	3.025	4
–	–	–	–	–	–	–	–	1000	–	3.023	5

Appendix

Table 121: Hyperparameter ablation for Mars on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-25	0.025	0.008	1	0	256	2000	0.1	2.955	0
0.9	–	–	–	–	–	–	–	–	–	2.960	1
0.98	–	–	–	–	–	–	–	–	–	2.953	2
–	–	–	–	0.004	–	–	–	–	–	2.953	3
–	–	–	–	–	–	–	128	–	–	2.974	4
–	–	–	–	–	–	–	–	1000	–	2.964	5

Appendix

Table 122: Hyperparameter ablation for Mars on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\gamma$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-25	0.025	0.004	1	0	256	2000	0.1	2.906	0
0.9	–	–	–	–	–	–	–	–	–	2.908	1
0.98	–	–	–	–	–	–	–	–	–	2.906	2
–	–	–	–	0.008	–	–	–	–	–	2.917	3
–	–	–	–	–	–	–	128	–	–	2.916	4
–	–	–	–	–	–	–	–	1000	–	2.906	5

Appendix

Table 123: Hyperparameter ablation for Mars on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	128	2000	0.1	3.160	0
–	–	–	0.004	–	–	–	–	–	3.160	1
–	–	–	–	–	–	256	–	–	3.165	2

Appendix

Table 124: Hyperparameter ablation for NAdamW on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	256	2000	0.1	3.090	0
–	–	–	0.004	–	–	–	–	–	3.097	1
–	–	–	–	–	–	128	–	–	3.098	2

Appendix

Table 125: Hyperparameter ablation for NAdamW on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	256	2000	0.1	3.039	0
–	–	–	0.004	–	–	–	–	–	3.039	1
–	–	–	–	–	–	128	–	–	3.055	2

Appendix

Table 126: Hyperparameter ablation for NAdamW on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	128	4000	0.1	3.013	0
–	–	–	0.008	–	–	–	–	–	3.023	1
–	–	–	–	–	–	256	–	–	3.020	2

Appendix

Table 127: Hyperparameter ablation for NAdamW on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	128	4000	0.1	2.955	0
–	–	–	0.008	–	–	–	–	–	2.971	1
–	–	–	–	–	–	256	–	–	2.954	2

Appendix

Table 128: Hyperparameter ablation for NAdamW on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	256	4000	0.1	2.907	0
–	–	–	0.008	–	–	–	–	–	2.910	1
–	–	–	–	–	–	128	–	–	2.913	2

Appendix

Table 129: Hyperparameter ablation for NAdamW on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-25	0.004	2	0	128	2000	0.2	3.178	0
–	–	–	0.002	–	–	–	–	–	3.180	1
–	–	–	–	–	–	256	–	–	6.960	2
–	–	–	–	–	–	–	4000	–	3.183	3
–	–	–	–	–	–	–	–	0.1	3.179	4

Appendix

Table 130: Hyperparameter ablation for Adam-Mini on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-25	0.004	2	0	128	2000	0.1	3.103	0
–	–	–	0.002	–	–	–	–	–	3.111	1
–	–	–	–	–	–	256	–	–	3.109	2
–	–	–	–	–	–	–	4000	–	3.104	3
–	–	–	–	–	–	–	–	0.2	3.111	4

Appendix

Table 131: Hyperparameter ablation for Adam-Mini on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-25	0.002	2	0	128	2000	0.2	3.049	0
–	–	–	0.004	–	–	–	–	–	3.064	1
–	–	–	–	–	–	256	–	–	3.052	2
–	–	–	–	–	–	–	4000	–	3.050	3
–	–	–	–	–	–	–	–	0.1	3.051	4

Appendix

Table 132: Hyperparameter ablation for Adam-Mini on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	128	4000	0.1	3.027	0
–	–	–	0.002	–	–	–	–	–	3.031	1
–	–	–	–	–	–	256	–	–	3.032	2
–	–	–	–	–	–	–	2000	–	7.359	3
–	–	–	–	–	–	–	–	0.2	3.037	4

Appendix

Table 133: Hyperparameter ablation for Adam-Mini on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	128	4000	0.1	2.966	0
–	–	–	0.002	–	–	–	–	–	2.963	1
–	–	–	–	–	–	256	–	–	2.963	2
–	–	–	–	–	–	–	2000	–	7.529	3
–	–	–	–	–	–	–	–	0.2	2.981	4

Appendix

Table 134: Hyperparameter ablation for Adam-Mini on 520m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	1	0	256	4000	0.1	2.912	0
–	–	–	0.002	–	–	–	–	–	2.918	1
–	–	–	–	–	–	128	–	–	2.921	2
–	–	–	–	–	–	–	2000	–	7.449	3
–	–	–	–	–	–	–	–	0.2	3.025	4

Appendix

Table 135: Hyperparameter ablation for Adam-Mini on 520m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	0.8	1e-15	0.008	linear	1	0	0.98	1e-05	128	0	0.1	3.143	0
–	–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.144	1
–	–	–	–	–	–	–	–	–	–	–	256	–	–	3.145	2

Appendix

Table 136: Hyperparameter ablation for Muon on 300m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	0.8	1e-15	0.004	linear	1	0	0.98	1e-05	128	0	0.1	3.079	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	3.088	1
–	–	–	–	–	–	–	–	–	–	–	256	–	–	3.083	2

Appendix

Table 137: Hyperparameter ablation for Muon on 300m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	0.8	1e-15	0.008	linear	1	0	0.98	1e-05	256	0	0.1	3.032	0
–	–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.029	1
–	–	–	–	–	–	–	–	–	–	–	128	–	–	3.049	2

Appendix

Table 138: Hyperparameter ablation for Muon on 300m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-25	0.004	linear	2	0	0.98	1e-05	128	0	0.1	3.002	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	3.008	1
–	–	–	–	–	–	–	–	–	–	–	256	–	–	3.009	2

Appendix

Table 139: Hyperparameter ablation for Muon on 520m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	1	1e-25	0.008	linear	2	0	0.98	1e-05	256	0	0.1	2.945	0
–	–	–	–	–	0.004	–	–	–	–	–	–	–	–	2.944	1
–	–	–	–	–	–	–	–	–	–	–	128	–	–	2.963	2

Appendix

Table 140: Hyperparameter ablation for Muon on 520m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0024	0.8	0.98	1	1e-25	0.008	linear	2	0	0.98	1e-05	256	0	0.1	2.906	0
–	–	–	–	–	0.004	–	–	–	–	–	–	–	–	2.900	1
–	–	–	–	–	–	–	–	–	–	–	128	–	–	2.930	2

Appendix

Table 141: Hyperparameter ablation for Muon on 520m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0008	0.98	0.8	0.008	linear	2	0	0.95	1e-05	128	0	0.1	3.152	0
–	–	–	0.004	–	–	–	–	–	–	–	–	3.153	1
–	–	–	–	–	–	–	–	–	256	–	–	3.154	2

Appendix

Table 142: Hyperparameter ablation for Scion on 300m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0008	0.98	1	0.008	linear	2	0	0.95	1e-05	256	0	0.1	3.086	0
–	–	–	0.004	–	–	–	–	–	–	–	–	3.099	1
–	–	–	–	–	–	–	–	–	128	–	–	3.090	2

Appendix

Table 143: Hyperparameter ablation for Scion on 300m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0004	0.98	0.8	0.004	linear	2	0	0.95	1e-05	128	0	0.1	3.039	0
–	–	–	0.008	–	–	–	–	–	–	–	–	3.057	1
–	–	–	–	–	–	–	–	–	256	–	–	3.037	2

Appendix

Table 144: Hyperparameter ablation for Scion on 300m on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0004	0.98	1	0.004	linear	2	0	0.95	1e-05	128	0	0.1	3.007	0
–	–	–	0.008	–	–	–	–	–	–	–	–	3.015	1
–	–	–	–	–	–	–	–	–	256	–	–	3.020	2

Appendix

Table 145: Hyperparameter ablation for Scion on 520m on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0008	0.98	1	0.008	linear	2	0	0.95	1e-05	256	0	0.1	2.952	0
–	–	–	0.004	–	–	–	–	–	–	–	–	2.952	1
–	–	–	–	–	–	–	–	–	128	–	–	2.970	2

Appendix

Table 146: Hyperparameter ablation for Scion on 520m on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\mathrm{Decay(WSD)}$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{scion}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0004	0.98	1	0.004	linear	2	0	0.95	1e-05	256	0	0.1	2.904	0
–	–	–	0.008	–	–	–	–	–	–	–	–	2.913	1
–	–	–	–	–	–	–	–	–	128	–	–	2.913	2

Appendix

Table 147: Hyperparameter ablation for Scion on 520m on 8x Chinchilla Data

\beta_{1}

\mathrm{blocksize}

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{NormGrad}

\mathrm{Blocking}

Init_{pc}

\eta_{pc}

p_{pc}

\mathrm{BSZ}

Step_{pc}

\mathrm{warmup}

\lambda

Loss

Link

0.95

256

0.001

True

0.2

0.1

128

2000

1000

0.7

3.151

–

0.0005

–

3.157

Appendix

Table 148: Hyperparameter ablation for Kron on 300m on 2x Chinchilla Data

\beta_{1}

\mathrm{blocksize}

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{NormGrad}

\mathrm{Blocking}

Init_{pc}

\eta_{pc}

p_{pc}

\mathrm{BSZ}

Step_{pc}

\mathrm{warmup}

\lambda

Loss

Link

0.95

256

0.0005

True

0.2

0.1

128

2000

1000

0.7

3.083

–

0.001

–

3.090

Appendix

Table 149: Hyperparameter ablation for Kron on 300m on 4x Chinchilla Data

\beta_{1}

\mathrm{blocksize}

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{NormGrad}

\mathrm{Blocking}

Init_{pc}

\eta_{pc}

p_{pc}

\mathrm{BSZ}

Step_{pc}

\mathrm{warmup}

\lambda

Loss

Link

0.95

256

0.0005

True

0.2

0.1

128

2000

1000

0.7

3.031

–

0.001

–

3.074

Appendix

Table 150: Hyperparameter ablation for Kron on 300m on 8x Chinchilla Data

\beta_{1}

\mathrm{blocksize}

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{NormGrad}

\mathrm{Blocking}

Init_{pc}

\eta_{pc}

p_{pc}

\mathrm{BSZ}

Step_{pc}

\mathrm{warmup}

\lambda

Loss

Link

0.95

256

0.0005

True

0.2

0.1

128

2000

1000

0.5

3.009

–

0.001

–

3.009

Appendix

Table 151: Hyperparameter ablation for Kron on 520m on 2x Chinchilla Data

\beta_{1}

\mathrm{blocksize}

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{NormGrad}

\mathrm{Blocking}

Init_{pc}

\eta_{pc}

p_{pc}

\mathrm{BSZ}

Step_{pc}

\mathrm{warmup}

\lambda

Loss

Link

0.95

256

0.0005

True

0.2

0.1

128

2000

1000

0.5

2.946

–

0.001

–

2.950

Appendix

Table 152: Hyperparameter ablation for Kron on 520m on 4x Chinchilla Data

$\beta_{1}$	$\mathrm{blocksize}$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{NormGrad}$	$\mathrm{Blocking}$	$Init_{pc}$	$\eta_{pc}$	$p_{pc}$	$\mathrm{BSZ}$	$Step_{pc}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	256	0.0005	1	0	True	True	1	0.2	0.1	128	2000	1000	0.5	2.900	0
–	–	0.001	–	–	–	–	–	–	–	–	–	–	–	2.909	1
–	–	–	–	–	–	–	–	–	–	256	–	–	–	2.902	2

Appendix

Table 153: Hyperparameter ablation for Kron on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.9	128	1000	0.1	3.147	0
–	–	128	–	–	–	–	–	–	–	–	–	–	3.154	1
–	–	256	–	–	–	–	–	–	–	–	–	–	3.150	2
–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.147	3
–	–	–	–	–	–	–	–	–	–	256	–	–	3.153	4

Appendix

Table 154: Hyperparameter ablation for Soap on 300m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.9	256	1000	0.1	3.084	0
–	–	128	–	–	–	–	–	–	–	–	–	–	3.086	1
–	–	256	–	–	–	–	–	–	–	–	–	–	3.084	2
–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.086	3
–	–	–	–	–	–	–	–	–	–	128	–	–	3.091	4

Appendix

Table 155: Hyperparameter ablation for Soap on 300m on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.9	256	1000	0.1	3.030	0
–	–	128	–	–	–	–	–	–	–	–	–	–	3.034	1
–	–	256	–	–	–	–	–	–	–	–	–	–	3.032	2
–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.031	3
–	–	–	–	–	–	–	–	–	–	128	–	–	3.043	4

Appendix

Table 156: Hyperparameter ablation for Soap on 300m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.95	256	1000	0.1	3.004	0
–	–	128	–	–	–	–	–	–	–	–	–	–	3.013	1
–	–	256	–	–	–	–	–	–	–	–	–	–	3.010	2
–	–	–	–	0.004	–	–	–	–	–	–	–	–	3.008	3
–	–	–	–	–	–	–	–	–	–	128	–	–	3.011	4

Appendix

Table 157: Hyperparameter ablation for Soap on 520m on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.95	256	1000	0.1	2.944	0
–	–	128	–	–	–	–	–	–	–	–	–	–	2.948	1
–	–	256	–	–	–	–	–	–	–	–	–	–	2.945	2
–	–	–	–	0.008	–	–	–	–	–	–	–	–	2.949	3
–	–	–	–	–	–	–	–	–	–	128	–	–	2.946	4

Appendix

Table 158: Hyperparameter ablation for Soap on 520m on 4x Chinchilla Data

\beta_{1}

\beta_{2}

\mathrm{blocksize}

\epsilon

\eta

g_{\mathrm{norm}}

\eta_{min}

\mathrm{Blocking}

f_{pc}

\beta_{shampoo}

\mathrm{BSZ}

\mathrm{warmup}

\lambda

Loss

Link

0.95

0.99

512

1e-10

0.004

True

0.95

256

1000

0.1

2.899

–

0.008

–

2.906

Appendix

Table 159: Hyperparameter ablation for Soap on 520m on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.002	2	0	256	2000	0.2	2.905	0
0.95	–	–	–	–	–	–	–	–	2.905	1
0.98	–	–	–	–	–	–	–	–	2.909	2
–	0.9	–	–	–	–	–	–	–	2.914	3
–	0.95	–	–	–	–	–	–	–	2.909	4
–	–	1e-25	–	–	–	–	–	–	2.907	5

Appendix

Table 160: Hyperparameter ablation for AdamW on 1.2b on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.002	1	0	256	1000	0.2	2.836	0

Appendix

Table 161: Hyperparameter ablation for AdamW on 1.2b on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.002	1	0	256	1000	0.2	2.787	0

Appendix

Table 162: Hyperparameter ablation for AdamW on 1.2b on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.002	1	0	256	1000	0.2	2.752	0

Appendix

Table 163: Hyperparameter ablation for AdamW on 1.2b on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	256	4000	0.1	2.902	0

Appendix

Table 164: Hyperparameter ablation for NAdamW on 1.2b on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	256	4000	0.1	2.833	0

Appendix

Table 165: Hyperparameter ablation for NAdamW on 1.2b on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	256	4000	0.1	2.785	0

Appendix

Table 166: Hyperparameter ablation for NAdamW on 1.2b on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.98	0.98	1e-10	0.004	1	0	256	4000	0.1	2.749	0

Appendix

Table 167: Hyperparameter ablation for NAdamW on 1.2b on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.9	256	1000	0.1	2.940	0

Appendix

Table 168: Hyperparameter ablation for Soap on 1.2b on 1x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.9	256	1000	0.1	2.829	0

Appendix

Table 169: Hyperparameter ablation for Soap on 1.2b on 2x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.9	256	1000	0.1	2.783	0

Appendix

Table 170: Hyperparameter ablation for Soap on 1.2b on 4x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.9	256	1000	0.1	2.749	0

Appendix

Table 171: Hyperparameter ablation for Soap on 1.2b on 8x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-15	0.004	linear	2	0	0.98	1e-05	256	0	0.1	2.891	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	2.886	1

Appendix

Table 172: Hyperparameter ablation for Muon on 1.2b on 1x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-15	0.004	linear	2	0	0.98	1e-05	256	0	0.1	2.827	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	2.833	1

Appendix

Table 173: Hyperparameter ablation for Muon on 1.2b on 2x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-15	0.004	linear	2	0	0.98	1e-05	256	0	0.1	2.780	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	2.793	1

Appendix

Table 174: Hyperparameter ablation for Muon on 1.2b on 4x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-15	0.004	linear	2	0	0.98	1e-05	256	0	0.1	2.748	0

Appendix

Table 175: Hyperparameter ablation for Muon on 1.2b on 8x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.008	1	0	256	1000	0.1	3.207	0

Appendix

Table 176: Hyperparameter ablation for AdamW on 130m on 16x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.9	0.98	1e-10	0.004	2	0	256	2000	0.1	3.001	0

Appendix

Table 177: Hyperparameter ablation for AdamW on 300m on 16x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.008	1	0	256	2000	0.1	3.200	0

Appendix

Table 178: Hyperparameter ablation for NAdamW on 130m on 16x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.98	1e-10	0.004	1	0	256	2000	0.1	2.998	0

Appendix

Table 179: Hyperparameter ablation for NAdamW on 300m on 16x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.008	1	0	True	10	0.98	256	1000	0.1	3.191	0

Appendix

Table 180: Hyperparameter ablation for Soap on 130m on 16x Chinchilla Data

$\beta_{1}$	$\beta_{2}$	$\mathrm{blocksize}$	$\epsilon$	$\eta$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{Blocking}$	$f_{pc}$	$\beta_{shampoo}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.95	0.99	512	1e-10	0.004	1	0	True	10	0.9	256	1000	0.1	2.990	0

Appendix

Table 181: Hyperparameter ablation for Soap on 300m on 16x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	1	1e-25	0.004	linear	1	0	0.98	1e-05	128	0	0.1	3.192	0
–	–	–	–	–	0.008	–	–	–	–	–	–	–	–	3.202	1

Appendix

Table 182: Hyperparameter ablation for Muon on 130m on 16x Chinchilla Data

$\eta_{adam}$	$\beta_{1}$	$\beta_{2}$	$\mathrm{Decay(WSD)}$	$\epsilon$	$\eta$	$\mathrm{Schedule}$	$g_{\mathrm{norm}}$	$\eta_{min}$	$\mathrm{\beta_{muon}}$	$\epsilon_{muon}$	$\mathrm{BSZ}$	$\mathrm{warmup}$	$\lambda$	Loss	Link
0.0012	0.8	0.98	0.8	1e-15	0.004	linear	1	0	0.98	1e-05	256	0	0.1	2.991	0

Appendix

Table 183: Hyperparameter ablation for Muon on 300m on 16x Chinchilla Data

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 208 papers

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
May 9, 2026Aditya RanganathDeep Learning Optimization LLM Input Optimization
Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
Stanford University. · Princeton University. · Tsinghua University.
The Best Optimizer Depends on Batch Size
Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning Optimization Hyperparameter Transfer
Princeton University · Stanford University · University of California, San Diego

May 9, 2026J/K move · Enter open · S save

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers

Aditya Ranganath

Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA

Deep Learning Optimization LLM Input Optimization Memory-Efficient Optimization Matrix Optimization LLM Training

Open paper AskPDF arXiv

Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design for large language models through a systems-and-optimization lens. We organize the literature into classical first-order optimizers, adaptive optimizers, memory-efficient variants, second-order and curvature-aware methods, sign-based and discovered optimizers, low-rank and projection-based methods, and matrix-based optimizers such as Muon. We also discuss benchmarking methodology, including hyperparameter fairness, scale dependence, wall-clock efficiency, token efficiency, memory overhead, and downstream evaluation. We argue that optimizer research for LLMs is entering a new phase: moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons that jointly evaluate convergence, stability, memory, and implementation complexity.

Abstract

Deep Learning Optimization

Explore similar work

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

The Best Optimizer Depends on Batch Size