On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose Step-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.
Figures & tables
Figure 1: Overview of Step-Level On-Policy Distillation (SOPD). The left and center panels compare On-Policy Distillation (OPD), On-Policy Expert Correction (OEC), and SOPD. The right panel illustrates the packed attention mask used for efficient training. We pack the input prompt, the complete student trajectory, and its multiple step-level teacher targets into a single sequence, enabling these step-level supervision instances to be processed in parallel within a single forward pass to ensure training efficiency. Each teacher target can attend only to the input prompt, its corresponding student prefix, and its own preceding causal teacher tokens, while future student steps and other teacher targets remain masked.
Valid Seen
Valid Unseen
Hard
Method
SR ↑
Rounds ↓
SR ↑
Rounds ↓
SR ↑
Rounds ↓
Qwen2.5-7B-RL (Teacher)
85.71
10.61
76.87
13.06
6.61
27.31
Qwen2.5-3B Zero-Shot
7.86
28.73
2.24
29.63
0.83
29.88
Supervised fine-tuning (SFT)
32.14
22.85
25.37
24.16
4.96
29.12
Vanilla OPD
65.72
14.73
60.45
16.21
10.74
28.64
TCOD (B2F)
77.86 ↑ 12.14
12.57 ↓ 2.16
70.90 ↑ 10.45
14.56 ↓ 1.65
13.22 ↑ 2.48
28.16 ↓ 0.48
Table 1: ALFWorld results with a 3B student and 7B RL teacher. SR denotes success rate (%). TCOD supplies all rows except SOPD ( Wang et al., 2026 ) . Arrows show the absolute change from Vanilla OPD; bold marks the best student result in each column.
Figure 2: ALFWorld full-split evaluation over normalized Explorer progress. Seen and Unseen success improve over training while interaction length decreases overall. The curves summarize full-split evaluations at the saved checkpoints.
Figure 3: ALFWorld training diagnostics following TCOD’s multi-panel analysis. (a) Explorer-batch task success; (b) mean environment rounds; (c) mean tokens per generated teacher turn target; and (d) step-balanced SFT loss in the Trainer. The horizontal axis normalizes each signal to the endpoint of its corresponding component.
Method
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg4
Teacher
58.0
54.6
32.5
38.9
46.0
+ continued RL (100 updates)
60.9
55.6
32.8
38.4
46.9
Student
21.5
21.9
10.0
8.0
15.4
OPD
61.9
57.0
32.5
39.6
47.7
SOPD
71.8
67.4
38.9
52.7
57.7
Table 2: Comparison of OPD and SOPD using the same-size teacher–student pair. Teacher denotes Qwen3-4B-Non-Thinking-RL-Math, and Student denotes the initial Qwen3-4B non-thinking model. The first three rows are reference results reported by G-OPD ( Yang et al., 2026 ) . Values are mean accuracy (%); Avg4 averages the four benchmarks, and bold marks the best student method in each column.
Figure 4: SOPD step structure over mathematical training. (a) Mean natural steps per AIME24 evaluation response; (b) mean student tokens per natural step; and (c) mean supervised teacher tokens per retained step during training. All boundaries use the semantic-step rule and 1,024-token safety cap.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training batch size
128
Micro-batch size
128
Rollouts per prompt
8
Maximum prompt length
2,048
Maximum response length
16,384
Temperature
1.0
Appendix
Table 3: GRPO hyperparameters used to train the mathematical domain teacher, following G-OPD ( Yang et al., 2026 ) .
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA
On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision. Although OPD operates on trajectories, its learning signal remains token-level: it identifies deviations through high-loss tokens and repairs them through local reverse-KL correction. We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories. About 30% of high-loss tokens fall into the low-divergence regime, indicating that many are surface-form mismatches rather than real reasoning forks. Moreover, even truly divergent tokens are difficult to repair with isolated token-level supervision, since reasoning failures often unfold as short-horizon distributional drift. We propose Trajectory-aware OPD (TOPD), which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens. Experiments show that suppressing non-divergent high-loss tokens improves standard OPD from 47.8% to 48.2% average accuracy, while TOPD further improves performance to 52.2%, with gains on AIME24 from 60.0% to 63.3% and AIME25 from 46.7% to 53.3%.