Matrix-structured optimizers can improve language-model pretraining, but it is unclear whether these gains transfer to short-horizon KL distillation from an instruction-tuned initialization. We compare AdamW and diagonal AdaGrad with two tractable Shampoo reductions, SOAP, and All-2D Muon across four dense Qwen student-teacher pairs under a common forward-KL objective and fixed training budget. Across scales, none of the evaluated matrix-structured recipes consistently improves the quality versus trainer-time trade-off over the elementwise references: AdamW generally attains the lowest validation KL, while AdaGrad can be slightly faster at larger scale. SOAP often tracks AdamW in KL but incurs substantial trainer-time overhead, and the two Shampoo reductions reverse ordering across model pairs, showing that conclusions depend on the feasible recipe. These results suggest that pretraining optimizer gains do not transfer automatically to KL distillation and should be re-established for the target objective and systems regime.
Figures & tables
Figure 1: Minimum validation KL within 1683 steps: three-seed means of per-seed minima, with one-standard-deviation error bars. Lower is better. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; Qwen2.5 SOAP was not extended beyond one matched 561-step run.
Figure 2: Validation KL over training (three-seed means). Dashed lines mark effective epochs of 552 updates; training ends at step 1683 . Vertical scales differ. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; Qwen2.5 SOAP was not extended beyond one matched 561-step run.
Figure 3: Trainer time normalized per pair and seed to AdamW ( 1.0 ); bars show three-seed means with one-standard-deviation error bars. Qwen3 pairs use 72 GPUs and Qwen2.5 uses 144; ratios are within-pair. The matched 561-step Qwen2.5 SOAP run took 4.31× AdamW trainer time and was not extended.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Validation-KL trajectories for grafted right-only Shampoo. Replicated Scalable Shampoo and parameter-owner Scalable Shampoo are three-seed means; Meta Distributed Shampoo is a single-seed control with additional recipe differences.
Figure 5: Validation-KL trajectories for unblocked SOAP (three-seed mean) and Meta blockwise SOAP (single-seed control). The comparison includes backend and algorithmic differences, including Meta’s matrix blocking and delayed basis use.
Figure 6: Validation-KL trajectories for All-2D Muon (three-seed mean) and the single-seed Reference and Meta controls. The controls apply Muon only to hidden-layer matrices; All-2D also includes embeddings and the output head.
Method
Qwen3 0.6B←1.7B
Qwen3 4B←8B
Qwen3 14B←32B
Qwen2.5 32B←72B
Learning rate
AdamW
3×10−5
3×10−5
3×10−5
3×10−5
AdaGrad
3×10−4
3×10−4
3×10−4
1.5×10−4
Scalar-row-mean quarter-power Shampoo
7×10−3
5×10−4
5×10−4
5×10−4
Right-only half-power Shampoo
10−2
3×10−4
3×10−4
3×10−4
SOAP
3×10−5
3×10−5
10−4
10−4†
Appendix
Table 1: Pair-specific learning rates and, for each Shampoo recipe, the first preconditioner step and refresh interval (in updates; reported as first/refresh).
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
Knowledge distillation trains a student language model to match a teacher's predictions on the training text. We ask whether learning the teacher's behavior beyond the training text can further improve the student. We investigate this using three input perturbations: additive token noise, blending two documents, and blending neighboring tokens. Specifically, for an early part of pretraining we perturb the inputs and train the student to match the teacher on them, and we spend the rest of the same token budget on ordinary distillation. When pretraining from scratch, the best method improves on standard distillation at both scales we tried, by 0.025 nats at 135M parameters and by 0.020 nats at 0.5B (mean of three seeds). At 135M this is 1.45 times the gain that distillation itself provides over next-token training. We also observe promising downstream transfer: at 0.5B, the best variant improves average zero-shot accuracy by about one percentage point across seven tasks. It also retains its validation-loss advantage after low-bit weight quantization. We also test a data-limited setting in which training data must be reused, and find that the gains persist. Finally, ablations suggest that, for additive noise, perturbing the teacher's inputs contributes most of the gain, while perturbing only the student's inputs yields much less.
AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.
Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.