How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
Authors: Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai, Youhe Jiang, Eiko Yoneki, Tuo Shi
Organizations: Aalto University · Nanyang Technological University · Independent · Beijing Institute of Technology · University of Cambridge · Shenzhen University of Advanced Technology
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least 1.93× as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality 1.62× sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd
Figures & tables
Figure 1: OPD tolerates the mismatch of quantized rollouts, up to a limit. Training accuracy (the fraction of the rollout engine’s responses whose final answer is correct, smoothed over 21 steps) for OPD on the 7B student, one training run per line.
Figure 2: OPD workflow and mismatch budgets. Top: 1 the rollout engine generates responses; 2 the teacher scores sampled tokens; 3 the trainer computes advantages and updates the student; and 4 updated weights are synchronized to the engine. Training runs locate the mismatch budget (§ 5.1 ); each dot represents a passing (blue) or failing (red) run. Bottom: the admission check estimates a new engine’s mismatch before training (§ 3.3 ).
Figure 3: On the 7B student, runs pass up to 0.184 nats/token and fail at 0.39 and above. One point per run; hollow markers are second runs, nudged right. Bars in (a) show 95% intervals; shading marks the held-out KL verdicts. Panel (b) includes only runs evaluated for accuracy. Values: App. Table 6 .
Figure 4: Mismatch estimates separate the tested passing and failing configurations. Points show trained engines, their estimated mismatch, and final held-out KL; error bars show 95% intervals. For the 1.5B and Llama points, estimates come from five-step training runs, not offline checks. Dashed: the pilot admission boundary L/α≈0.27 nats/token. Hollow: an engine with weights up to eight versions old, shown at its recipe’s estimate.
Figure 5: On the 7B student, an FP8 KV cache dominates mismatch, and one layer carries it. W8A16 means 8-bit weights and 16-bit activations. Dashed: admission boundary L/α≈0.27 ; engines below it are admitted. Recipes: App. Table 3 .
Figure 6: Asynchronous rollouts improve throughput until the trainer no longer waits. One run per bar on the 7B student. Rollout weights can be up to k versions older than the trainer’s; purple bars additionally use Model Optimizer’s FP8.
Figure 7: At a memory-bound rollout batch, FP8 benefits from faster generation and saved KV-cache memory. The 7B student with 1,536 responses per step, 24 times the training batch. The lighter Model Optimizer FP8 bar is not decomposed. Values: App. Table 11 .
Figure 8: REINFORCE broke under all three rollout configurations that OPD tolerates, and GRPO under two by step 200. Training accuracy (for GRPO and REINFORCE, mean 0/1 reward), smoothed over 11 steps. Each method uses different prompts, so compare it only with its own dashed standard-engine baseline.
Figure 9: Two ways mismatch adds error to OPD’s update. Boxes are tokens; the rollout engine’s response (purple) leaves the standard engine’s (blue) at the fork. Bottom : the size of T and S at the untrained student, to scale.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Held-out KL line
Fail
Budget bracket (nats/token)
up to 6.35%
6
not ordered: 0.055 fails, 0.097 and 0.184 pass
6.35–23.28%
5
[0.184, 0.39], as in Section 5.1
23.28–23.69%
4
0.39 splits: one of its two runs fails
23.69–53.92%
3
[0.39, 0.56]
53.92–59.66%
1–2
not ordered: 0.56 fails, 0.68 passes
above 59.66%
0
no run fails
Appendix
Table 1: The budget on our 7B student for other held-out KL lines (step 200, same interval condition; from Table 6 ). Fail: runs that fail. Not ordered: a failing level lies below a passing one.
Experiment and question
What was run
Reported in
Budget. How much mismatch does OPD tolerate?
Standard run; weight noise at four levels, two run twice; four Model Optimizer engines; vLLM’s FP8 cache, with and without the first layer’s KV cache in BF16; staleness of 1, 2, 4 and 8 versions; a colocated asynchronous run
Section 5.1
Admission. Can we tell before training which engines fit?
Offline estimates; the 1.5B and Llama-3.2-3B students with the standard engine and three and two FP8 engines
Sections 5.3 , 5.2
Speed. What does the budget buy?
Speed and memory benchmarks; eight-step runs at 1,024 and 1,536 sequences per step; four 30-step runs without the draft head; the combined run and time to quality
Section 5.4
GRPO and REINFORCE. Do they break where OPD holds?
Four of OPD’s engines at two learning rates (the second set not read, by the authors’ decision); score centering; under staleness and on the smaller students: running, not reported
Section 5.5
Mechanism. Why does OPD tolerate mismatch?
Gradient probe; error at later checkpoints; score centering; five offset runs; two truncated-importance-sampling runs; corrections at fixed prefixes (undetermined); prefix shift
Section 6 , App. D
Qwen3 pair. Does the budget hold for a Qwen3 student?
Qwen3-4B with a Qwen3-30B-A3B teacher: training loss at step 50 still 0.854 of its start, above the 0.80 limit; stopped
This table; not retried
Appendix
Table 2: Every experiment of the study.
Engine
Weights / activations / cache
Trained
vLLM FP8 cache
BF16 / BF16 / FP8, scale 1
first layer’s KV cache in BF16
BF16 / BF16 / FP8, scale 1
Model Optimizer FP8
FP8 / FP8 / FP8
first and last layers in BF16
FP8 / FP8 / FP8
NVFP4, emulated
FP4 / FP4 / FP8
Appendix
Table 3: Quantized rollout engines. All run on vLLM 0.19.1’s own kernels unless marked emulated. FP8 caches use calibrated scales unless marked scale 1. † After a one-line fix to vLLM 0.19.1’s W4A8 scale conversion.
In FP8
Offline estimate
Fixed prefixes
Nothing (standard engine)
0.0006
–
Values
0.0005
0.02%
Keys
0.51 [0.45, 0.57]
99.95%
Keys and values
0.62
100%
vLLM’s first-layer cache, real
0.55
1.03 [0.96, 1.10] a
Every layer but the first, real
0.0037
99.9% b
Appendix
Table 4: The first layer’s FP8 cache, one tensor at a time (untrained 7B student; one measurement per row; emulated at scale 1 unless marked real). Offline estimate in nats/token, with 95% intervals over prompts. Fixed prefixes: KL relative to keys and values. a Ratio of emulated to real. b Share of the every-layer cache’s KL that a BF16 first layer removes.
σ=0.0665
σ=0.15
Probe mismatch (nats/token)
0.0086
0.044
Token mean of Eq[A]
−0.0705
−0.0904
rbias
0.079
0.267
Angle between gq and gKL (degrees)
4.2
12.2
⟨gbias,n^⟩
5.83×10−5
1.49×10−4
Appendix
Table 5: Gradient probe, weight-noise engines. Probe mismatch: KL(q∥p) at the untrained student on its own responses; in training these engines average 0.055 and 0.097 (Table 6 ).
Mismatch
Held-out KL vs. the standard run (%) [95% interval]
Accuracy
Primary rule
Rollout engine
level D
Step 100
Step 200
(points)
(100 / 200)
Standard engine (reference)
0.0008
(KL 0.0573)
(KL 0.0481)
(60.93%)
–
Asynchronous rollouts, standard engine
Colocated, at most 2 versions old
0.0009
−0.95 [ −2.85 , +0.95 ]
−0.16 [ −2.61 , +2.31 ]
−0.93
pass / pass
Staleness of 1 model version
0.0009
−1.93 [ −4.27 , +0.39 ]
−2.72 [ −5.21 , −0.30 ]
–
pass / pass
Staleness of 2 model versions
0.0011
−1.35 [ −3.59 , +0.89 ]
−0.82 [ −3.50 , +1.89 ]
–
pass / pass
Appendix
Table 6: Every OPD run on the mismatch axis (one 200-step run per row). D : mismatch in training, mean over steps 1–200 (nats/token), which names noise levels in the main text (0.097: σ=0.15 ). Accuracy: composite change at step 200. Intervals are over problems within a run.
Table 7: The admission check’s record (one 200-step OPD run per row). Estimate : offline, before training. Training : mean mismatch over steps 1–100. Ratio : training over estimate, predicted between 1/1.5 and 1.5 ( ∗ outside). Held-out KL : against the standard run at the same step (first six rows at step 100: the mean of three 100-step standard runs; otherwise the pair’s 200-step standard run). Accuracy : change at step 200 ( † fails the accuracy rule, App. B.1 ). Predicted : the check’s verdict, fixed before the run (none: no prediction). Emulated: quality only. a A forecast, not a fixed prediction. b The estimate omits staleness; the prediction used the recipe’s admission and the mismatch of staleness alone. c No offline estimate exists; the mean mismatch of a 5-step test run. d Not shown to fail: the smallest detectable rise is 14 to 20%, and the interval reaches +27.3% . e Recorded before the run without an estimate; not one of the eight.
Table 9: Real-kernel rollout engines at three batch sizes (one run per cell). Left: our training batch (64 sequences per step), steps 3–200, with the EAGLE-3 draft head (Table 11 : without it). Right: 8-step runs without it at 1,024 and (bottom block) 1,536 sequences per step, where the BF16 cache fills, steps 3–8. Budget: the quality verdict at 64 sequences per step. Tokens/h: generated tokens per hour. Peaks are lower bounds. † Replica cache capacities differ by one 16-token block.
On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Sirui Chen, Yunzhe Qi, Haobo Xu +4
University of Illinois Urbana-Champaign · Stanford University
On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.
Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.