Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons
Organizations: Renmin University of China
Abstract
On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.
Figures & tables
| Discrepancy reduction | |||||
| Supervised block | Avg@4 Acc. (%) | Eval. | Eval. | Eval. | Eval. |
| 56.46 | 92.7 | 63.9 | 53.5 | 46.5 | |
| 54.38 | 66.8 | 92.2 | 65.3 | 59.9 | |
| 55.42 | 60.5 | 70.4 | 92.0 | 65.8 | |
| 52.60 | 54.2 | 66.1 | 67.2 | 91.7 | |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Sampled trajectory | Excerpt | Qualitative observation |
| Early | ID 1, | “The problem reminds me of generating functions where you can encode conditions on the exponents … Because the condition is is even, it splits into two cases: either all of are even, or exactly two of them are odd.” | Shared task-specific strategy and correct parity decomposition. |
| Early | ID 4, | “Case 2: Exactly two of are odd … For Case 2: For the case where exactly one is odd, we have , , or variations.” | A correct decomposition immediately branches into an invalid one-odd case. |
| Middle | ID 20, | “Hmmm. Hmmm confusing. Wait no, maybe I should model separately … How can we relate to ? … Wait, hold on … Wait, no.” | Still reorganizing the parity argument. |
| Middle | ID 41, | “ , which is correct … Thus, the answer is . Therefore, , , so .” | Already completes the numerical consistency check at a comparable depth. |
| Late | ID 12, | “Recall that even part odd part … probably get … is .” | Reaches the correct final result after extended verification. |
| Late | ID 44, | “ , … Thus, the sum is … .” | Reaches a self-consistent but incorrect branch and final answer. |
| Token Reduction | ||||
| 0 | 7,168 | 0.0% | 0.00% | 100.00% |
| 0.5 | 4,114 | 42.6% | 29.29% | 37.80% |
| 1.0 | 2,655 | 63.0% | 50.00% | 14.29% |
| 1.5 | 1,928 | 73.1% | 64.64% | 5.40% |
| 5.0 | 1,062 | 85.2% | 96.88% | 0.006% |
| Domain | Updates | EOS term. | ||||
| Math | 0 | 279 | 7,168 | 7,168 | 5,764 | 54.34% |
| Math | 0.5 | 279 | 4,114 | 4,169 | 3,569 | 25.25% |
| Math | 1.0 | 279 | 2,655 | 2,859 | 2,546 | 13.88% |
| Math | 1.5 | 279 | 1,928 | 2,184 | 1,999 | 8.70% |
| Math | 5.0 | 279 | 1,062 | 1,057 | 1,055 | 0.58% |
| Code | 0 | 384 | 7,168 | 7,168 | 6,452 | 23.67% |
| Token cost | Wall time | AMC23 | AIME25 | LCB v6 | Multi-IF mean | Multi-IF turn 3 | |
| 0 | 100.0% | 100.0% | 85.83 | 36.67 | 25.33 | 37.89 | 26.79 |
| 0.5 | 61.2% | 59.9% | 83.00 | 32.22 | 25.33 | 39.26 | 27.81 |
| 1.5 | 36.4% | 35.6% | 82.50 | 35.56 | 23.05 | 38.73 | 26.96 |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 562 papers