Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models
Organizations: Fudan University · The Hong Kong Polytechnic University · Sichuan University · Alibaba Group ·
Abstract
Continual pre-training (CPT) adapts large language models (LLMs) to specialized domains, yet its standard next-token prediction (NTP) objective treats every corpus token as a fixed target, regardless of its utility to the current model. In a controlled study on the same 1B-token mathematical corpus, we observe sharply different outcomes across base models: NTP improves Llama-3.2-3B's performance on mathematical benchmarks but substantially degrades Qwen3-1.7B-Base's performance, indicating that, under a fixed training recipe, the same training sequences can have model-dependent effects. To address this problem, we introduce On-Policy Mid-Training Distillation (OPMD), which reformulates mid-training from offline text imitation into active, query-driven knowledge acquisition. At selected anchor positions, OPMD samples trajectories from the evolving student policy and applies dense token-level reverse-KL rewards from a frozen teacher, training the model on states it actually visits rather than only on ground-truth prefixes. This on-policy objective reduces the training-generation mismatch of teacher-forced NTP. On MATH-500 and Minerva, OPMD improves the Qwen student over its base performance and outperforms the evaluated NTP baseline, which degrades on both benchmarks. OPMD integrates into CPT pipelines without added inference-time overhead and offers a model-aware alternative to passive domain adaptation.
Figures & tables
| Methods | MATH-500 | Minerva | ||
| pass@16 | avg@16 | pass@16 | avg@16 | |
| Qwen3-1.7B-Base (Student) | 76.6 | 18.5 | 30.5 | 4.7 |
| Qwen3-4B-Base (Teacher 1) | 84.8 | 21.3 | 34.9 | 5.6 |
| Qwen3-14B-Base (Teacher 2) | 94.2 | 68.6 | 51.5 | 24.0 |
| Student NTP | \mathbf{18.2}^{{\color[rgb]{0,0.5,0}\downarrow 58.4}} | \mathbf{1.5}^{{\color[rgb]{0,0.5,0}\downarrow 17.0}} | \mathbf{5.5}^{{\color[rgb]{0,0.5,0}\downarrow 25.0}} | \mathbf{0.4}^{{\color[rgb]{0,0.5,0}\downarrow 4.3}} |
| Teacher 1 OPMD | \mathbf{82.8}^{{\color[rgb]{0.8281,0.1836,0.1836}\uparrow 6.2}} | \mathbf{30.5}^{{\color[rgb]{0.8281,0.1836,0.1836}\uparrow 12.0}} | \mathbf{34.9}^{{\color[rgb]{0.8281,0.1836,0.1836}\uparrow 4.4}} | \mathbf{7.0}^{{\color[rgb]{0.8281,0.1836,0.1836}\uparrow 2.3}} |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Checkpoint | MATH-500 | Minerva | ||
| pass@16 | avg@16 | pass@16 | avg@16 | |
| Teacher: Qwen3-4B-Base | ||||
| Student (initial) | 76.6 | 18.5 | 30.5 | 4.7 |
| Steps 100 | 80.6 | 27.5 | 33.8 | 6.7 |
| Steps 200 | 80.8 | 29.0 | 34.2 | 6.3 |
| Steps 300 | 81.8 | 29.0 | 35.3 | 6.9 |
| Model and checkpoint | MATH-500 | Minerva | ||
| pass@16 | avg@16 | pass@16 | avg@16 | |
| Qwen3-1.7B-Base (initial) | 76.6 | 18.5 | 30.5 | 4.7 |
| Qwen3-1.7B NTP, step 400 | 30.2 | 2.5 | 9.6 | 0.7 |
| Qwen3-1.7B NTP, step 800 | 17.8 | 1.5 | 8.8 | 0.7 |
| Qwen3-1.7B NTP, step 1200 | 17.6 | 1.3 | 6.6 | 0.4 |
| Qwen3-1.7B NTP, step 1600 | 20.0 | 1.5 | 8.1 | 0.6 |
| Checkpoint | MATH-500 | Minerva | ||
| pass@16 | avg@16 | pass@16 | avg@16 | |
| High-Entropy | ||||
| Student (initial) | 76.6 | 18.5 | 30.5 | 4.7 |
| Steps 100 | 81.0 | 29.4 | 33.5 | 7.1 |
| Steps 200 | 82.4 | 30.3 | 33.8 | 7.1 |
| Steps 300 | 81.2 | 30.4 | 37.5 | 7.7 |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 556 papers