In on-policy distillation (OPD), response completion, supervision delivery, and parameter updates need not share the same granularity. We develop streaming OPD by decoupling these three clocks: a response repeatedly delivers newly generated segments while generation continues, retaining the full causal prefix at each delivery and computing loss only on new positions. A consumption protocol then organizes supervision budgets, position selection, staleness handling, and update scheduling, allowing control over quantity, coverage, and version lag across training stages. In GSM8K experiments distilling Qwen3-4B into Qwen3-0.6B with VERL, supervised-budget matching reduces the early gap between best test accuracies over seven common candidates from 4.55 to 2.42 points. Synchronous continuation from the same checkpoint reaches 73.39% best test accuracy over seven common candidates at u=384,...,576, versus 74.30% synchronous and 71.95% budget-matched (differences -0.91 and -2.35 points); an unpaired extended record reaches 76.12%. Freshness filtering and disagreement-overlap selection improve local peaks but do not establish sustained advantages. A system prototype shows lower data-supply waiting times without establishing an equal-update speedup. Quality recovery and supply-side benefits are assessed separately; end-to-end gains must account for continuation cost.
Figures & tables
Method
Data used directly for training
Later content from the same response
AsyncOPD [ 1 ]
Completed samples are teacher-scored and batched
In-flight prefixes are retained and continued; delivery follows completion
TOPD [ 2 ]
The first H positions are generated and distilled
The response is not continued beyond truncation for training
ADWIN [ 3 ]
Current-window prefixes support synchronous updates
Some prefixes continue in the background for window calibration
This work
New, prefix-preserving segments are repeatedly delivered and batched by budget
Later segments remain candidate supervision through the final segment
Table 1: Mechanistic distinctions from neighboring methods. The comparison concerns the organization of directly trained data and the use of later content from the same response, not a performance ranking under matched configurations.
Delivery
Existing context (no loss)
New segment (supervised)
Segment mask
Segment 1
—
There are 3 bags.
1
Segment 2
There are 3 bags.
Each holds 4 apples.
0,1
Segment 3
There are 3 bags. Each holds 4 apples.
So there are 12 apples. <EOS>
0,0,1
Table 2: Segment supervision within one response. The English fragments illustrate the flow rather than actual tokenization boundaries; every row retains the prompt. Each segment can enter the queue independently but need not occupy a separate parameter update.
Figure 1: Three clocks and the training loop. (a) An illustrative schedule, not measured durations, fixed sealing boundaries, or a guarantee that every segment enters an update before response completion. The final segment also enters the pool for later batching. Dashed arrows indicate that a segment may join supervision from other responses in an update; scoring time is omitted. (b) The corresponding data and parameter flows. Full prefixes and new-segment masks are detailed in Table 2 .
Variable
Direct change
Question tested
Intervention
Budget B
Supervision per update and its variability
Does matching supervision quantity narrow the gap?
Raw-token budget; supervised-budget matching
Selection s
Positions retained within a budget
Are high disagreement or compatibility more useful?
High-disagreement selection; disagreement–overlap selection; uniform random selection
Staleness rule ρ
Retained positions and relative weights
Does removing older-version supervision help?
Freshness filtering
Schedule σ
Data consumption and its joint distribution
Can synchronous continuation narrow the residual?
Synchronous continuation with optimizer reset
Table 3: Principal consumption-protocol interventions. Each intervention can affect several statistical dimensions and must be interpreted using actual supervision counts and composition. Descriptive names associate results with configurations; archival identifiers are mapped in Appendix A.1 .
Dimension
Observations
Tested intervention
Main finding
Supervision quantity
Segmenting reduces per-update supervision and increases its variability
Raw-token budget; supervised-budget matching
Early best-score gap decreases from 4.55 to 2.42 percentage points
Update scale
Both supervision quantity and cross-update variability change
Reduced learning rate
Early best test accuracy 71.49%, 0.07 percentage points above supervised-budget matching
Position coverage
Larger gaps on medium-length and long responses; a lower share of supervision in terminal segments
Tail-quota selection for positions that have arrived
Unpaired five-point best 68.01%; no baseline difference without a step map
Version lag
Stale positions retain disagreement, with similar local probability coverage
Freshness filtering
Best 72.78% versus 71.95% for supervised-budget matching, but wins only the first of three points
Position selection
Many candidate positions have little local disagreement
High-disagreement selection; disagreement–overlap selection; uniform random selection; teacher-support mass reweighting
Random best 71.80% versus high-disagreement 69.45%; disagreement–overlap best 72.55%
Delivery timing
Tail-segment availability and consumption order may affect updates
Completeness gating; terminal-first scheduling; stale-segment replay; regeneration with current weights
At the sole candidate per run, mean improvements for gating and regeneration, but neither reaches the synchronous anchor
Table 4: Evidence overview for Sections 4 and 5 . Each row uses its own configuration and candidate window; best scores are selected independently within common candidates where mappings are available. Results are neither ranked across groups nor added together as causal contributions.
Figure 2: Early budget and learning-rate comparisons. Shading covers the seven common candidates u=32,64,…,224 , corresponding to Table 6 ; open circles highlight the earliest maximum for each method within this set. Solid lines connect measured values without smoothing. Supervised-budget matching and the reduced learning rate narrow naive streaming’s best-score gap, but their subsequent trajectories differ.
Metric (summary value)
Naive streaming
Synchronous baseline
Supervised tokens per update
Approx. 46–48K
Approx. 74.8K
Coefficient of variation of supervision counts across updates
0.43–0.65
0.045
Share of supervised positions in terminal segments
Approx. 40%
Approx. 80%
Table 5: Consumption-side audit, retaining source precision. Supervised-token counts are approximate range/value summaries; coefficients of variation are reported range/value summaries; terminal-segment shares are approximate values. Intervals are source-reported ranges, not extrema over a common window or confidence intervals. These are count and composition audits, not accuracy means. Definitions are given in Appendix A.5 .
Method
Best test accuracy (%)
Difference between best scores vs. baseline (pp)
Synchronous baseline
73.84
0.00
Naive streaming
69.29
−4.55
Raw-token budget
70.96
−2.88
Supervised-budget matching
71.42
−2.42
Reduced learning rate
71.49
−2.35
Table 6: Early comparison within common candidates u=32,64,96,128,160,192,224 . Each method independently selects its best test accuracy from one trajectory; differences compare best scores, not same-update pairs. Selected updates and process means are in Table 11 . Reduced learning rate retains naive streaming’s sample-count batching and uses 0.58 times the default learning rate, without a token budget.
Figure 3: Long-horizon curves over twelve common candidates u=256,288,…,576 plus 596 . Open circles highlight each method’s earliest maximum; top annotations report best scores and differences between them. Solid lines connect measured values without smoothing. Horizontal dashed lines show process means, not confidence intervals. Supervised-budget matching reaches 72.93% versus the baseline’s 74.30%, a best-score gap of 1.37 percentage points despite exceeding it at individual checkpoints.
Method
Intervention rule
Best test accuracy (%)
Difference between best scores vs. baseline (pp)
Synchronous baseline
Synchronous consumption
74.30
0.00
Supervised-budget matching
Budget measured in supervised positions
71.95
−2.35
High-disagreement selection
Retain the top 50% by per-position FKL
69.45
−4.85
Disagreement–overlap selection
Rank by FKL × top- k overlap
72.55
−1.75
Uniform random selection
Retain 50% uniformly at random
71.80
−2.50
Freshness filtering
Mask positions with lag>0
72.78
−1.52
Table 7: Late-start GSM8K comparison within the three common candidates u=384,416,448 . Each method independently selects its best test accuracy from one trajectory, not three seeds. Differences compare best scores with the synchronous baseline, not same-update pairs. Selected updates and process means are in Table 12 ; full trajectories and branch records remain in Appendix A.2 .
Metric
Summary value
Supported interpretation
Per-update median KL
Approx. 0.0015–0.002
Many positions have little disagreement
Per-update 10th percentile of mass
Approx. 0.997
High local probability coverage
Mean KL of stale/fresh positions
0.14 / 0.12
Residual disagreement remains in the stale group
Mass of stale/fresh positions
Approx. 0.997 / 0.997
Similar local probability coverage
Table 8: Audit of uniform random selection’s early candidate pool, retaining source precision. The first row is a reported range of per-update KL medians, and the second an approximate summary of per-update mass 10th percentiles, over the first 8 updates. The last two report grouped mean KL and approximate mass summaries for fresh and stale positions at approximately update 8. These are local-distribution audits, not accuracy means. Reduction conventions are given in Appendix A.5 .
Figure 4: Late-start GSM8K comparisons at the three common checkpoints u=384,416,448 : (a) high-disagreement selection and uniform random selection; (b) disagreement–overlap selection and teacher-support mass reweighting; (c) freshness filtering. All panels use the same x- and y-axis ranges and repeat the same synchronous-baseline and supervised-budget-matching curves. Lines connect measured points only, without interpolated checkpoints or smoothing; open circles highlight each method’s earliest maximum within these candidates. Table 7 reports these best scores and their differences; full mapped trajectories appear only as supplementary evidence in Figure 7 .
Method
Mean accuracy (%)
Difference vs. synchronous baseline (pp)
Synchronous baseline
66.79
0.00
Naive streaming
59.06
−7.73
Completeness gating
61.79
−5.00
Terminal-first scheduling
53.15
−13.64
Stale-segment replay
55.76
−11.03
Regeneration with current weights
62.70
−4.09
Table 9: Independent short-horizon scheduling screen. Values are run means at u=16 ; differences use the synchronous anchor of 66.79%. Run-level records and aggregation rules appear in Appendix A.3 .
Figure 5: Synchronous-continuation curves over seven common candidates from u=384 to 576 at 32-update intervals. The vertical line marks the switch and optimizer reset at u=352 ; shading includes the first two post-switch evaluations. Open circles highlight each method’s earliest maximum. Synchronous continuation reaches 73.39%, supervised-budget matching 71.95%, and the synchronous baseline 74.30%; the respective best-score differences from the baseline are −0.91 and −2.35 percentage points. Lines connect measured values without smoothing; horizontal dashed lines show seven-point process means, not confidence intervals.
Figure 6: Two common-candidate comparisons of consumption-side interventions. (a) Supervised-budget matching reduces the early best-score gap from 4.55 to 2.42 percentage points over seven common candidates. (b) Synchronous continuation with an optimizer reset narrows the best-score gap from 2.35 to 0.91 percentage points over u=384,416,…,576 . Each method selects its maximum independently; these are not same-update paired differences. The panels use different candidate windows and do not provide a cross-window decomposition of causal contributions.
Metric
Reporting basis
Complete-response delivery
Sealed-segment delivery
Supply wait for a 128-sample batch
Range/approximation
47–218 s
≈0.2 s
Data-supply wait
p95, target: 128 updates
5.06 s
0.34 s
Trainer idle fraction
Peak summary
57%
≈10%
Lag-1 tokens/batch
Count/reported upper bound
18,073
≤743
Fresh-token fraction
Value/reported range
0.74
0.87–0.95
Runtime
Target: 64 updates
≈55 min
≈35 min
Table 10: Separate prototype measurements, retaining source precision. Each row uses its stated statistical basis rather than a common accuracy window: range/approximation, p95, peak summary, count/upper bound, value/range, or approximate runtime. ≈ indicates an approximation, intervals are reported ranges, and ≤ denotes a reported upper bound. Target update counts do not resolve the recorded completion-count discrepancy; these measurements do not establish equal-update speedup. Details appear in Appendix A.5 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Full trajectories in method-specific panels with shared axis scales. Shading marks the primary candidate windows: twelve points from u=256 to 596 for naive streaming, seven from u=384 to 576 for synchronous continuation, and u=384,416,448 for the other interventions. Each panel includes the synchronous baseline and supervised-budget matching as references. Faded curves show supplementary records; dashed curves denote separate continuation branches. Only measured points with known update mappings are shown, without adding points or joining branches. Records outside the candidate windows are excluded from primary checkpoint selection.
Figure 8: Supplementary extended synchronous continuation. Cumulative stage updates r and total updates satisfy u=352+r , with the restart at r=224 . The open circle highlights the best test accuracy of 76.12% at r=480 ( u=832 ) within all eleven candidates r=224,256,…,544 . The dashed line indicates their process mean of 74.06%, including the first-four-point plateau; it is not a confidence interval. Connecting lines are not smoothed. This unpaired figure is not spliced with Figure 5 and has no contemporaneous synchronous reference.
Method
u∗
Best (%)
Mean (%)
Mean Δ (pp)
Early: seven candidates
Synchronous baseline
192
73.84
71.55
0.00
Naive streaming
192
69.29
67.74
−3.80
Raw-token budget
160
70.96
68.86
−2.69
Supervised-budget matching
128
71.42
69.65
−1.90
Reduced learning rate
160
71.49
69.90
−1.65
Appendix
Table 11: Checkpoint-selection audit for the early and long-horizon comparisons. Early: seven candidates u=32,64,…,224 , excluding u=256 . Long horizon: twelve candidates u=256,288,…,576 plus 596 . Each method has one trajectory per group; means describe checkpoints, not seeds. Mean Δ uses the synchronous mean within the same group.
Method
u∗
Best (%)
Mean (%)
Mean Δ (pp)
Late start: three candidates
Synchronous baseline
384
74.30
73.82
0.00
Supervised-budget matching
416
71.95
71.34
−2.48
High-disagreement selection
448
69.45
66.74
−7.08
Disagreement–overlap selection
416
72.55
71.29
−2.53
Uniform random selection
384
71.80
71.55
−2.27
Appendix
Table 12: Checkpoint-selection audit for late-start interventions and synchronous continuation. Late start: three candidates u=384,416,448 . Continuation: seven candidates u=384,416,…,576 . Each method has one trajectory per group; means describe checkpoints, not seeds. Mean Δ uses the synchronous mean within the same group.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA
On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.