Speculative Self-Distillation enables Efficient Knowledge Internalization
Organizations: Stanford University · ETH Zürich
Abstract
Many real-world applications require language models to internalize knowledge that was absent from pretraining, to incorporate continually updated facts, or to learn new domain skills. Self-distillation has emerged as a natural approach for this task. A model conditioned on the source material acts as the teacher, and an unconditioned student is trained to reproduce the teacher's behavior closed-book. We propose Speculative Self-Distillation (SSD), an efficient per-token mixed-policy method for self-distillation. The student generates each rollout by default, while the teacher intervenes only at positions where its next-token distribution substantially diverges from the student's. Unlike off-policy distillation with teacher-generated prefixes, this design keeps training rollouts close to the student's test-time behavior, while avoiding the low-signal states that fully on-policy rollouts drift into. In effect, SSD lets the teacher steer the rollout exactly where the privileged context changes the prediction. On three closed-book QA benchmarks measuring fact recall, knowledge updates, and skill acquisition, SSD nearly matches on-policy distillation while using 38-69% fewer supervised tokens and consistently outperforms off-policy distillation on the held-out test set. On a forgetting stress test, SSD retains the most general capability while also achieving the best in-domain accuracy and using less supervision than the on-policy baseline. Overall, SSD establishes a new accuracy-efficiency-forgetting Pareto frontier for self-distillation.
Figures & tables
| On-policy | SSD | ||||
| Model | Acc. | Tokens (M) | Acc. | Tokens (M) | Fewer tokens |
| Qwen3-4B | 13.19 | 5.88 | 55% | ||
| Qwen2.5-7B | 2.84 | 2.20 | 23% | ||
| Qwen3-14B | 73.2 | 2.16 | 75.5 | 1.26 | 42% |
| Method | Hella. | Human | IFEval | MMLU | TQA1 | Wino. | Avg.6 | SciQA | Tokens (M) | |
| Base | 62.1 | 70.1 | 72.9 | 71.8 | 47.9 | 71.1 | 65.98 | – | 34.2 2.5 | – |
| Off-policy (SFT) | 61.0 | 64.6 | 47.0 | 69.8 | 40.9 | 72.8 | 59.34 | 65.0 2.4 | 1.82 | |
| Off-policy (FKL) | 59.6 | 64.0 | 48.4 | 69.4 | 41.1 | 73.4 | 59.33 | 65.3 3.0 | 1.82 | |
| On-policy | 59.6 | 73.8 | 49.5 | 69.9 | 43.1 | 73.2 | 61.51 | 71.0 0.8 | 2.84 | |
| SSD | 61.0 | 70.7 | 66.8 | 71.1 | 44.7 | 73.2 | 64.59 | 73.5 1.7 | 2.20 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Train | Test |
| Company Memo | 1133 | 100 |
| Wikipedia (Cyclone Ditwah) | 1129 | 100 |
| Science Q&A | 2674 | 507 |
| Hyperparameter | Off-policy SD | On-policy SD | Speculative SD |
| Base model | Qwen3-4B-Instruct-2507 | Qwen3-4B-Instruct-2507 | Qwen3-4B-Instruct-2507 |
| Loss | CE, forward-KL | forward-KL | forward-KL |
| Optimizer | AdamW | AdamW | AdamW |
| LR scheduler | Constant w. warmup (cosine on SciQA) | Constant w. warmup (cosine on SciQA) | Constant w. warmup (cosine on SciQA) |
| Warmup ratio | |||
| Learning rate |
| Experiment | Maximum new tokens | Evaluation decoding |
| Company Memo and Wikipedia | 512 | Greedy |
| Science Q&A: main Qwen3-4B runs | 1024 | Greedy |
| Science Q&A: Qwen3-14B scaling | 2048 | Greedy |
| Retrieval: base, BM25, and dense retrieval (4B and 14B) | 16384 | Greedy |
| Retrieval: SSD , Qwen3-4B | 1024 | Greedy |
| Retrieval: SSD , Qwen3-14B | 2048 | Greedy |
| Hyperparameter | Value |
| Epochs | 2 |
| Effective batch size | 16 |
| Learning rate schedule | cosine, 1.5% warmup |
| Gradient clipping | 1.0 |
| Precision | bf16 |
| Learning rate | , except SSD uses |
| Task | Metric | Scoring |
| HellaSwag | acc, not length-normalized | loglikelihood, deterministic |
| MMLU | acc | loglikelihood, deterministic |
| TruthfulQA | mc1 | loglikelihood, deterministic |
| Winogrande | acc | loglikelihood, deterministic |
| HumanEval | pass@1 | generative, greedy |
| IFEval | inst_level_loose_acc | generative, greedy |
| Method | Generation | Loss calculation | Backward | Update + sync | Other | Total |
| SSD | 30.25 | 3.16 | 6.37 | 0.50 | 1.10 | 41.38 |
| On-policy | 28.86 | 5.64 | 5.64 | 0.23 | 1.16 | 41.54 |
| Off-policy FKL | 0.00 | 4.01 | 6.26 | 0.19 | 0.80 | 11.25 |
| Off-policy SFT | 0.00 | 2.47 | 5.41 | 0.15 | 0.67 | 8.71 |
| Model | Inference method | Accuracy | Mean generated tokens |
| Qwen3-4B | Closed-book base | 49.9 | 2864 |
| Qwen3-4B | Top-4 BM25 demonstrations | 65.9 | 2045 |
| Qwen3-4B | Top-4 dense demonstrations | 61.3 | 2146 |
| Qwen3-4B | SSD closed-book | 66.3 | 508 |
| Qwen3-14B | Closed-book base | 45.2 | 373 |
| Qwen3-14B | Top-4 BM25 demonstrations | 58.4 | 510 |
| Switching rule | Setting | Teacher-token frac. | Accuracy | Tokens (M) |
| JSD | 0.7% | 4.47 | ||
| Top- inclusion | 9.3% | 4.51 | ||
| Entropy | 11.5% | 5.18 | ||
| Reverse KL, | 1.5% | 4.02 | ||
| Forward KL, | 3.4% | 4.07 |
| Strategy | Acc. | Mean resp. | Teacher frac. |
| length | (per-row) | ||
| Student only | 12.8% | 879 | 0 0.0% |
| SSD , | 34.12% | 850 | 0 1.3% |
| SSD , | 51.87% | 745 | 0 2.7% |
| SSD , | 66.07% | 588 | 0 7.0% |
| SSD , | 73.96% | 523 | 15.7% |
| Strategy | Teacher tokens | Teacher frac. | Length | Extracted |
| Student only | 0 | 0.0% | 1024 | no tag |
| SSD , | 11 | 1.1% | 1024 | no tag |
| SSD , | 4 | 0.4% | 1024 | no tag |
| SSD , | 9 | 0.9% | 1024 | no tag |
| SSD , | 75 | 9.0% | 837 | B |
| Teacher only | 549 | 100.0% | 549 | B |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 556 papers