On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.
Figures & tables
Figure 1: TISD changes where self-distillation is evaluated. Standard OPSD (e.g., SDPO) distills teacher targets along the original student trajectory, including the suffix that follows the student’s sampled action. TISD instead forces the teacher-preferred token at the peak-divergence position, lets the student regenerate the suffix, and distills the resulting trajectory. This exposes successor prefixes that are absent from the original rollout.
Figure 2
Figure 4: Training collapse of distillation on fully teacher-regenerated rollouts (TRD) on SciKnowEval L3 biology (Qwen3-8B).
Figure 5: Token-level KL-divergence from original and token-changed regenerated suffixes. Results aggregate failed rollouts from LiveCodeBench v6 of the base model (Qwen3-8B).
Method
Qwen3-8B
Qwen3-14B
Olmo3-7B-Instruct
Average
Avg
Pass
Avg
Pass
Avg
Pass
Avg
Pass
Base
27.5
36.1
28.6
35.1
32.1
42.7
29.4
38.0
+ GRPO
40.7
44.0
44.1
48.1
39.3
48.1
41.4
46.7
+ SDPO
48.1
48.6
48.9
48.9
50.8
53.4
49.2
50.3
+ TRD
39.3
41.2
48.1
48.1
38.0
40.5
41.8
43.3
+ TISD (ours)
49.2
49.6
51.1
51.1
51.0
53.4
50.4
51.4
Table 1: Best Avg@4 and corresponding Pass@4 (%) within 80-step training on LiveCodeBench v6 with rich environment feedback. Bold and underlining mark the highest and second-highest trained-method values in each column.
Method
Biology
Chemistry
Materials
Physics
Average
Steps
Hours
Steps
Hours
Steps
Hours
Steps
Hours
Steps
Hours
Base
30.6
30.6
41.6
41.6
59.0
59.0
59.0
59.0
47.6
47.6
+ GRPO
59.6
63.8
73.9
75.2
78.5
80.1
73.6
73.6
71.4
73.2
+ GRPO (on-policy)
51.5
51.5
66.8
66.8
73.4
73.4
69.4
69.4
65.2
65.2
+ SDPO
57.4
61.1
79.5
81.1
77.5
78.1
76.3
76.3
72.7
74.2
+ TISD (ours)
60.7
63.9
80.4
80.8
74.7
75.1
78.2
78.3
73.5
74.5
Table 2: Best Avg@128 (%) on SciKnowEval L3 without rich environment feedback (Qwen3-8B). Steps and Hours denote training budgets of 200 steps and 10 hours, respectively. Average is the unweighted mean across the four disciplines. Bold and underlining mark the highest and second-highest means in each column among the reported methods.
Figure 6: Mean test-case pass rate (Avg@4, %) across training data scales, through 80 training steps in the rich-feedback setting (Qwen3-8B). LCB-Large combines v1–v6, and LCB-Small contains v6. TISD first exceeds GRPO’s maximum on LCB-Large with a 33% speedup and reaches SDPO’s maximum on LCB-Small with a 56% speedup.
Figure 8
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: TISD pseudocode. Each record contains a prompt x , response y , binary reward, and privileged context c . peak_divergence_pos selects the eligible position with the largest teacher–student divergence. Loss and optimization settings appear in Appendix D .
Figure 10: CDF of the reverse-KL concentration ratio KL(t∗)/∑tKL(t) on 202 failed Qwen3-1.7B LiveCodeBench v6 rollouts.
Role
1.7B
4B
8B
Student
Algorithmic
36.1%
11.6%
11.6%
Naming
11.4%
3.9%
5.5%
Formatting
7.4%
13.3%
12.7%
Non-code text
45.0%
71.3%
70.2%
Teacher
Algorithmic
38.6%
10.5%
14.4%
Naming
9.9%
4.4%
3.9%
Appendix
Table 3: GPT-5.4 semantic annotations of student and teacher tokens at t∗ on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Entries are percentages within each model scale.
Figure 11: Student-to-teacher semantic-role transitions at t∗ on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Each row shows the distribution of teacher-token roles conditioned on the corresponding student-token role. The dominant diagonal shows that disagreement usually remains within a semantic category.
Peak divergence
First divergence
Random
Semantic role (student token at t∗ )
Algorithmic
11.6%
0.6%
20.4%
Naming
5.5%
0.0%
3.9%
Formatting
12.7%
15.5%
11.0%
Non-code text
70.2%
84.0%
64.6%
Execution-based success rate at t∗
Appendix
Table 4: Effect of branch-position selection on 181 failed Qwen3-8B LiveCodeBench v6 rollouts. Semantic roles refer to the student token at the selected position and are assigned by GPT-5.4. Execution success is measured after either teacher continuation or student regeneration from the forced teacher or original token. First divergence denotes the first position where the student and teacher argmax tokens differ.
Step 0 (202)
Step 20 (183)
Step 40 (192)
Step 60 (176)
Step 80 (145)
Pass rate
0.211
0.285
0.250
0.312
0.434
Semantic role (student token at t∗ )
Algorithmic
36.1%
35.5%
36.5%
35.8%
26.2%
Naming
11.4%
15.3%
12.0%
14.8%
9.7%
Formatting
7.4%
8.2%
13.5%
11.4%
15.9%
Non-code text
45.0%
41.0%
38.0%
38.1%
48.3%
Appendix
Table 5: Evolution of rollout pass rate and semantic roles at the peak-divergence position during Qwen3-1.7B SDPO training on LiveCodeBench v6, using 256 rollouts per update. Pass rate is computed over all rollouts, while GPT-5.4 semantic-role percentages are computed over failed rollouts at each update; the corresponding failure counts are shown in parentheses.
Fail (129)
Succ (92)
Semantic role (student token at t∗ )
Algorithmic
7.8%
15.2%
Naming
3.1%
10.9%
Formatting
7.8%
10.9%
Non-code text
81.4%
63.0%
Execution-based success rate at t∗
Appendix
Table 6: Reverse-KL diagnostic on 221 non-truncated OLMo3-7B-Instruct LiveCodeBench v6 rollouts. Entries report semantic-role distributions at t∗ and execution success after continuation from t∗ , separately for failed and successful original rollouts.
Biology (148)
Chemistry (143)
Physics (128)
Materials (134)
Pooled (553)
Execution-based success rate at t∗
Teacher continuation
96.6%
93.0%
93.8%
90.3%
93.5%
Student cont. (teacher token)
31.1%
41.3%
40.6%
45.5%
39.4%
Student cont. (original token)
22.3%
25.9%
30.5%
19.4%
24.4%
Appendix
Table 7: Verifier-based intervention on failed Qwen3-8B SciKnowEval L3 rollouts at SDPO update 0. The teacher is conditioned on the first successful same-problem rollout in source order, excluding the target rollout, and t∗ is the valid response position with maximum reverse KL. Each domain contains 32 problems with eight rollouts per problem. Numbers in parentheses denote failed-rollout counts.
1.7B
4B
8B
Problem groups (out of 32)
All-fail
23 (71.9%)
19 (59.4%)
19 (59.4%)
All-success
5 (15.6%)
6 (18.8%)
5 (15.6%)
Mixed
4 (12.5%)
7 (21.9%)
8 (25.0%)
Failed rollouts
Total
202
181
181
Appendix
Table 8: Coverage of group-relative outcome supervision for Qwen3-{1.7B, 4B, 8B} on 32 LiveCodeBench v6 problems at update 0, with eight rollouts per problem. Counts and percentages partition problem groups by verifier outcome and failed rollouts by whether their normalized GRPO advantage is zero. All-fail and all-success groups receive zero normalized advantage.
1.7B
4B
8B
By semantic role (correct / total)
Algorithmic
9/73 (12.3%)
1/21 (4.8%)
0/21 (0.0%)
Naming
4/23 (17.4%)
0/7 (0.0%)
2/10 (20.0%)
Formatting
0/15 (0.0%)
1/24 (4.2%)
2/23 (8.7%)
Non-code text
3/91 (3.3%)
17/129 (13.2%)
20/127 (15.7%)
Appendix
Table 9: Verifier success of teacher continuations from t∗ , stratified by GPT-5.4 semantic role, on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Each entry reports successful continuations over failed rollouts assigned to the corresponding role.
Student succeeds
Student fails
Teacher succeeds
17
7
Teacher fails
1
156
Appendix
Table 10: Paired continuation outcomes on 181 failed Qwen3-8B LiveCodeBench v6 rollouts. Both conditions preserve the original prefix, force the teacher’s argmax token at peak divergence, and differ only in whether the teacher or student generates the suffix. Success requires passing all private tests.
Figure 12: Example of trajectory redirection on a failed Qwen3-8B LiveCodeBench response. The peak-divergence token is the naming token j , which carries 12.3% of response KL. Forcing the teacher-preferred k and regenerating with the student changes the algorithm from iterating over (i,k) to iterating over (i,j) , producing an integer k=2j−i and a correct solution.
Figure 13: Repetitive meta-commentary in a TRD student validation response. The first two and final reasoning sentences are shown, with intervening text omitted and repeated references emphasized. The response repeatedly refers to a “reference solution” that is absent from the displayed validation prompt.
Parameters
GRPO
SDPO
TRD
TISD
General
Model
Qwen3-8B, Qwen3-14B, Olmo3-7B-Instruct
Thinking (Qwen3)
False
False
False
False
Data
Max. prompt length
2048
2048
2048
2048
Max. response length
8192
8192
8192
8192
Appendix
Table 11: Hyperparameters for learning with rich environment feedback on LiveCodeBench.
Parameters
GRPO
SDPO
TRD
TISD
General
Model
Qwen3-8B
Qwen3-8B
Qwen3-8B
Qwen3-8B
Thinking
False
False
False
False
Data
Max. prompt length
2048
2048
2048
2048
Max. response length
8192
8192
8192
8192
Appendix
Table 12: Hyperparameters for learning without rich environment feedback on SciKnowEval L3. Paired GRPO values denote off-policy / on-policy settings.
Method
TISD
SDPO
GRPO
TRD
Average time (min/step)
16.60
15.05
16.69
16.36
Appendix
Table 13: Average wall-clock time per training step (minutes) for Qwen3-8B on LiveCodeBench v6 using four NVIDIA B200 GPUs.
On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student's response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged. PAST preserves the student's next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A 2×2 factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.
Yangyang Feng, Zhuoyan Feng, Junlan Chen
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University
On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches. BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks. Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld and 31% on WebShop relative to seven independent trajectories. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD (n=7) across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.
Dongyang Chen, Shudong Liu, Shubo Li +4
Tsinghua University · SenseTime Research · Peking University
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP