Do Pretraining Optimizer Gains Transfer to KL Distillation?
Organizations: NVIDIA
Abstract
Matrix-structured optimizers can improve language-model pretraining, but it is unclear whether these gains transfer to short-horizon KL distillation from an instruction-tuned initialization. We compare AdamW and diagonal AdaGrad with two tractable Shampoo reductions, SOAP, and All-2D Muon across four dense Qwen student-teacher pairs under a common forward-KL objective and fixed training budget. Across scales, none of the evaluated matrix-structured recipes consistently improves the quality versus trainer-time trade-off over the elementwise references: AdamW generally attains the lowest validation KL, while AdaGrad can be slightly faster at larger scale. SOAP often tracks AdamW in KL but incurs substantial trainer-time overhead, and the two Shampoo reductions reverse ordering across model pairs, showing that conclusions depend on the feasible recipe. These results suggest that pretraining optimizer gains do not transfer automatically to KL distillation and should be re-established for the target objective and systems regime.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Qwen3 | Qwen3 | Qwen3 | Qwen2.5 |
|---|---|---|---|---|
| Learning rate | ||||
| AdamW | ||||
| AdaGrad | ||||
| Scalar-row-mean quarter-power Shampoo | ||||
| Right-only half-power Shampoo | ||||
| SOAP | ||||
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 561 papers