How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
Organizations: Aalto University · Nanyang Technological University · Independent · Beijing Institute of Technology ·
Abstract
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Held-out KL line | Fail | Budget bracket (nats/token) |
|---|---|---|
| up to 6.35% | 6 | not ordered: 0.055 fails, 0.097 and 0.184 pass |
| 6.35–23.28% | 5 | [0.184, 0.39], as in Section 5.1 |
| 23.28–23.69% | 4 | 0.39 splits: one of its two runs fails |
| 23.69–53.92% | 3 | [0.39, 0.56] |
| 53.92–59.66% | 1–2 | not ordered: 0.56 fails, 0.68 passes |
| above 59.66% | 0 | no run fails |
| Experiment and question | What was run | Reported in |
|---|---|---|
| Budget. How much mismatch does OPD tolerate? | Standard run; weight noise at four levels, two run twice; four Model Optimizer engines; vLLM’s FP8 cache, with and without the first layer’s KV cache in BF16; staleness of 1, 2, 4 and 8 versions; a colocated asynchronous run | Section 5.1 |
| Admission. Can we tell before training which engines fit? | Offline estimates; the 1.5B and Llama-3.2-3B students with the standard engine and three and two FP8 engines | Sections 5.3 , 5.2 |
| Speed. What does the budget buy? | Speed and memory benchmarks; eight-step runs at 1,024 and 1,536 sequences per step; four 30-step runs without the draft head; the combined run and time to quality | Section 5.4 |
| GRPO and REINFORCE. Do they break where OPD holds? | Four of OPD’s engines at two learning rates (the second set not read, by the authors’ decision); score centering; under staleness and on the smaller students: running, not reported | Section 5.5 |
| Mechanism. Why does OPD tolerate mismatch? | Gradient probe; error at later checkpoints; score centering; five offset runs; two truncated-importance-sampling runs; corrections at fixed prefixes (undetermined); prefix shift | Section 6 , App. D |
| Qwen3 pair. Does the budget hold for a Qwen3 student? | Qwen3-4B with a Qwen3-30B-A3B teacher: training loss at step 50 still 0.854 of its start, above the 0.80 limit; stopped | This table; not retried |
| Engine | Weights / activations / cache |
|---|---|
| Trained | |
| vLLM FP8 cache | BF16 / BF16 / FP8, scale 1 |
| first layer’s KV cache in BF16 | BF16 / BF16 / FP8, scale 1 |
| Model Optimizer FP8 | FP8 / FP8 / FP8 |
| first and last layers in BF16 | FP8 / FP8 / FP8 |
| NVFP4, emulated | FP4 / FP4 / FP8 |
| In FP8 | Offline estimate | Fixed prefixes |
|---|---|---|
| Nothing (standard engine) | 0.0006 | – |
| Values | 0.0005 | 0.02% |
| Keys | 0.51 [0.45, 0.57] | 99.95% |
| Keys and values | 0.62 | 100% |
| vLLM’s first-layer cache, real | 0.55 | 1.03 [0.96, 1.10] a |
| Every layer but the first, real | 0.0037 | 99.9% b |
| Probe mismatch (nats/token) | 0.0086 | 0.044 |
|---|---|---|
| Token mean of | ||
| 0.079 | 0.267 | |
| Angle between and (degrees) | 4.2 | 12.2 |
| Mismatch | Held-out KL vs. the standard run (%) [95% interval] | Accuracy | Primary rule | ||
| Rollout engine | level | Step 100 | Step 200 | (points) | (100 / 200) |
| Standard engine (reference) | 0.0008 | (KL 0.0573) | (KL 0.0481) | (60.93%) | – |
| Asynchronous rollouts, standard engine | |||||
| Colocated, at most 2 versions old | 0.0009 | [ , ] | [ , ] | pass / pass | |
| Staleness of 1 model version | 0.0009 | [ , ] | [ , ] | – | pass / pass |
| Staleness of 2 model versions | 0.0011 | [ , ] | [ , ] | – | pass / pass |