Perturbing Inputs Improves Pretraining Distillation
Organizations: Princeton University · Microsoft Research
Abstract
Knowledge distillation trains a student language model to match a teacher's predictions on the training text. We ask whether learning the teacher's behavior beyond the training text can further improve the student. We investigate this using three input perturbations: additive token noise, blending two documents, and blending neighboring tokens. Specifically, for an early part of pretraining we perturb the inputs and train the student to match the teacher on them, and we spend the rest of the same token budget on ordinary distillation. When pretraining from scratch, the best method improves on standard distillation at both scales we tried, by 0.025 nats at 135M parameters and by 0.020 nats at 0.5B (mean of three seeds). At 135M this is 1.45 times the gain that distillation itself provides over next-token training. We also observe promising downstream transfer: at 0.5B, the best variant improves average zero-shot accuracy by about one percentage point across seven tasks. It also retains its validation-loss advantage after low-bit weight quantization. We also test a data-limited setting in which training data must be reused, and find that the gains persist. Finally, ablations suggest that, for additive noise, perturbing the teacher's inputs contributes most of the gain, while perturbing only the student's inputs yields much less.
Figures & tables
| full | int4, group 128 | int3, group 128 | |||||
|---|---|---|---|---|---|---|---|
| precision | RTN | GPTQ | AWQ | RTN | GPTQ | AWQ | |
| standard KD | 2.6782 | 0.205 | 0.028 | 0.019 | 6.226 | 0.160 | 0.099 |
| noise | 2.6642 | 0.317 | 0.027 | 0.019 | 5.623 | 0.117 | 0.097 |
| parallel | 2.6660 | 0.161 | 0.024 | 0.020 | 1.480 | 0.123 | 0.099 |
| adjacent | 2.6579 | 0.113 | 0.024 | 0.019 | 1.203 | 0.120 | 0.098 |
| HellaSwag | ARC-E | ARC-C | PIQA | Wino. | LAMBADA | SciQ | Avg. | Avg. | |
| 0.5B | |||||||||
| standard KD | 45.0 | 57.9 | 29.0 | 68.0 | 52.7 | 37.5 | 77.3 | 52.5 | – |
| noise | 45.7 | 56.8 | 30.5 | 68.8 | 52.8 | 39.1 | 81.3 | 53.6 | |
| parallel | 45.5 | 58.1 | 30.3 | 68.3 | 54.1 | 36.7 | 77.5 | 52.9 | |
| adjacent | 45.9 | 58.0 | 30.6 | 69.7 | 53.4 | 36.3 | 80.9 | 53.5 | |
| 135M | |||||||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| 135M | 0.5B | |
| Student architecture | SmolLM2-135M | Qwen2.5-0.5B |
| Teacher (frozen) | SmolLM2-1.7B | Qwen2.5-1.5B |
| Vocabulary (shared) | 49,152 | 151,936 |
| Data | FineWeb-Edu, 58.6B-token pool | FineWeb-Edu, 65B-token pool |
| Steps batch length | (4.2B tokens) | (10B tokens) |
| Optimizer | AdamW, , weight decay 0.1 | same |
| 1 epoch, wd | 16 epochs, weight decay | |||||
|---|---|---|---|---|---|---|
| 0.1 | 1 | 0.1 | 0.3 | 1 | 3 | |
| Next-token prediction | 3.1542 | – | 3.3365 | – | 3.2297 | – |
| + noise | 3.1516 | – | 3.3250 | – | 3.2214 | – |
| Standard distillation | 3.1370 | 3.1318 | 3.2016 | 3.1812 | 3.1682 / 3.1693 | 3.2198 |
| + noise | 3.1177 | 3.1120 | 3.1693 | 3.1510 | 3.1448 / 3.1490 | – |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 561 papers