Streaming On-Policy Distillation: Decoupling Three Clocks for Stage-Wise Supervision Consumption
Organizations: Independent Researcher, China
Abstract
In on-policy distillation (OPD), response completion, supervision delivery, and parameter updates need not share the same granularity. We develop streaming OPD by decoupling these three clocks: a response repeatedly delivers newly generated segments while generation continues, retaining the full causal prefix at each delivery and computing loss only on new positions. A consumption protocol then organizes supervision budgets, position selection, staleness handling, and update scheduling, allowing control over quantity, coverage, and version lag across training stages. In GSM8K experiments distilling Qwen3-4B into Qwen3-0.6B with VERL, supervised-budget matching reduces the early gap between best test accuracies over seven common candidates from 4.55 to 2.42 points. Synchronous continuation from the same checkpoint reaches 73.39% best test accuracy over seven common candidates at u=384,...,576, versus 74.30% synchronous and 71.95% budget-matched (differences -0.91 and -2.35 points); an unpaired extended record reaches 76.12%. Freshness filtering and disagreement-overlap selection improve local peaks but do not establish sustained advantages. A system prototype shows lower data-supply waiting times without establishing an equal-update speedup. Quality recovery and supply-side benefits are assessed separately; end-to-end gains must account for continuation cost.
Figures & tables
| Method | Data used directly for training | Later content from the same response |
|---|---|---|
| AsyncOPD [ 1 ] | Completed samples are teacher-scored and batched | In-flight prefixes are retained and continued; delivery follows completion |
| TOPD [ 2 ] | The first positions are generated and distilled | The response is not continued beyond truncation for training |
| ADWIN [ 3 ] | Current-window prefixes support synchronous updates | Some prefixes continue in the background for window calibration |
| This work | New, prefix-preserving segments are repeatedly delivered and batched by budget | Later segments remain candidate supervision through the final segment |
| Delivery | Existing context (no loss) | New segment (supervised) | Segment mask |
|---|---|---|---|
| Segment 1 | — | There are 3 bags. | |
| Segment 2 | There are 3 bags. | Each holds 4 apples. | |
| Segment 3 | There are 3 bags. Each holds 4 apples. | So there are 12 apples. <EOS> |
| Variable | Direct change | Question tested | Intervention |
|---|---|---|---|
| Budget | Supervision per update and its variability | Does matching supervision quantity narrow the gap? | Raw-token budget; supervised-budget matching |
| Selection | Positions retained within a budget | Are high disagreement or compatibility more useful? | High-disagreement selection; disagreement–overlap selection; uniform random selection |
| Staleness rule | Retained positions and relative weights | Does removing older-version supervision help? | Freshness filtering |
| Schedule | Data consumption and its joint distribution | Can synchronous continuation narrow the residual? | Synchronous continuation with optimizer reset |
| Dimension | Observations | Tested intervention | Main finding |
|---|---|---|---|
| Supervision quantity | Segmenting reduces per-update supervision and increases its variability | Raw-token budget; supervised-budget matching | Early best-score gap decreases from 4.55 to 2.42 percentage points |
| Update scale | Both supervision quantity and cross-update variability change | Reduced learning rate | Early best test accuracy 71.49%, 0.07 percentage points above supervised-budget matching |
| Position coverage | Larger gaps on medium-length and long responses; a lower share of supervision in terminal segments | Tail-quota selection for positions that have arrived | Unpaired five-point best 68.01%; no baseline difference without a step map |
| Version lag | Stale positions retain disagreement, with similar local probability coverage | Freshness filtering | Best 72.78% versus 71.95% for supervised-budget matching, but wins only the first of three points |
| Position selection | Many candidate positions have little local disagreement | High-disagreement selection; disagreement–overlap selection; uniform random selection; teacher-support mass reweighting | Random best 71.80% versus high-disagreement 69.45%; disagreement–overlap best 72.55% |
| Delivery timing | Tail-segment availability and consumption order may affect updates | Completeness gating; terminal-first scheduling; stale-segment replay; regeneration with current weights | At the sole candidate per run, mean improvements for gating and regeneration, but neither reaches the synchronous anchor |
| Metric (summary value) | Naive streaming | Synchronous baseline |
|---|---|---|
| Supervised tokens per update | Approx. 46–48K | Approx. 74.8K |
| Coefficient of variation of supervision counts across updates | 0.43–0.65 | 0.045 |
| Share of supervised positions in terminal segments | Approx. 40% | Approx. 80% |
| Method | Best test accuracy (%) | Difference between best scores vs. baseline (pp) |
|---|---|---|
| Synchronous baseline | 73.84 | |
| Naive streaming | 69.29 | |
| Raw-token budget | 70.96 | |
| Supervised-budget matching | 71.42 | |
| Reduced learning rate | 71.49 |
| Method | Intervention rule | Best test accuracy (%) | Difference between best scores vs. baseline (pp) |
|---|---|---|---|
| Synchronous baseline | Synchronous consumption | 74.30 | |
| Supervised-budget matching | Budget measured in supervised positions | 71.95 | |
| High-disagreement selection | Retain the top 50% by per-position FKL | 69.45 | |
| Disagreement–overlap selection | Rank by FKL top- overlap | 72.55 | |
| Uniform random selection | Retain 50% uniformly at random | 71.80 | |
| Freshness filtering | Mask positions with | 72.78 |
| Metric | Summary value | Supported interpretation |
|---|---|---|
| Per-update median KL | Approx. 0.0015–0.002 | Many positions have little disagreement |
| Per-update 10th percentile of mass | Approx. 0.997 | High local probability coverage |
| Mean KL of stale/fresh positions | 0.14 / 0.12 | Residual disagreement remains in the stale group |
| Mass of stale/fresh positions | Approx. 0.997 / 0.997 | Similar local probability coverage |
| Method | Mean accuracy (%) | Difference vs. synchronous baseline (pp) |
|---|---|---|
| Synchronous baseline | 66.79 | |
| Naive streaming | 59.06 | |
| Completeness gating | 61.79 | |
| Terminal-first scheduling | 53.15 | |
| Stale-segment replay | 55.76 | |
| Regeneration with current weights | 62.70 |
| Metric | Reporting basis | Complete-response delivery | Sealed-segment delivery |
|---|---|---|---|
| Supply wait for a 128-sample batch | Range/approximation | 47–218 s | s |
| Data-supply wait | p95, target: 128 updates | 5.06 s | 0.34 s |
| Trainer idle fraction | Peak summary | 57% | |
| Lag-1 tokens/batch | Count/reported upper bound | 18,073 | |
| Fresh-token fraction | Value/reported range | 0.74 | 0.87–0.95 |
| Runtime | Target: 64 updates | min | min |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Best (%) | Mean (%) | Mean (pp) | |
| Early: seven candidates | ||||
| Synchronous baseline | 192 | 73.84 | 71.55 | |
| Naive streaming | 192 | 69.29 | 67.74 | |
| Raw-token budget | 160 | 70.96 | 68.86 | |
| Supervised-budget matching | 128 | 71.42 | 69.65 | |
| Reduced learning rate | 160 | 71.49 | 69.90 | |
| Method | Best (%) | Mean (%) | Mean (pp) | |
| Late start: three candidates | ||||
| Synchronous baseline | 384 | 74.30 | 73.82 | |
| Supervised-budget matching | 416 | 71.95 | 71.34 | |
| High-disagreement selection | 448 | 69.45 | 66.74 | |
| Disagreement–overlap selection | 416 | 72.55 | 71.29 | |
| Uniform random selection | 384 | 71.80 | 71.55 | |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 562 papers