TISD: On-Policy Self-Distillation with Trajectory Intervention

Taeckyung Lee; Rinat Amankos; Jeonghye Kim; Hyungjun Yoon; Woogyeol Jin; Sung-Ju Lee

cs.AISep 25, 2026

TISD: On-Policy Self-Distillation with Trajectory Intervention

Authors: Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee

Organizations: KAIST

Abstract

On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.

Figures & tables

Figure 1: TISD changes where self-distillation is evaluated. Standard OPSD (e.g., SDPO) distills teacher targets along the original student trajectory, including the suffix that follows the student’s sampled action. TISD instead forces the teacher-preferred token at the peak-divergence position, lets the student regenerate the suffix, and distills the resulting trajectory. This exposes successor prefixes that are absent from the original rollout.

Panel of Figure 4 — Figure 4: Training collapse of distillation on fully teacher-regenerated rollouts (TRD) on SciKnowEval L3 biology (Qwen3-8B).

Panel of Figure 5 — Figure 5: Token-level KL-divergence from original and token-changed regenerated suffixes. Results aggregate failed rollouts from LiveCodeBench v6 of the base model (Qwen3-8B).

Method	Qwen3-8B		Qwen3-14B		Olmo3-7B-Instruct		Average
	Avg	Pass	Avg	Pass	Avg	Pass	Avg	Pass
Base	27.5	36.1	28.6	35.1	32.1	42.7	29.4	38.0
+ GRPO	40.7	44.0	44.1	48.1	39.3	48.1	41.4	46.7
+ SDPO	48.1	48.6	48.9	48.9	50.8	53.4	49.2	50.3
+ TRD	39.3	41.2	48.1	48.1	38.0	40.5	41.8	43.3
+ TISD (ours)	49.2	49.6	51.1	51.1	51.0	53.4	50.4	51.4

Table 1: Best Avg@4 and corresponding Pass@4 (%) within 80-step training on LiveCodeBench v6 with rich environment feedback. Bold and underlining mark the highest and second-highest trained-method values in each column.

Method	Biology		Chemistry		Materials		Physics		Average
	Steps	Hours	Steps	Hours	Steps	Hours	Steps	Hours	Steps	Hours
Base	30.6	30.6	41.6	41.6	59.0	59.0	59.0	59.0	47.6	47.6
+ GRPO	59.6	63.8	73.9	75.2	78.5	80.1	73.6	73.6	71.4	73.2
+ GRPO (on-policy)	51.5	51.5	66.8	66.8	73.4	73.4	69.4	69.4	65.2	65.2
+ SDPO	57.4	61.1	79.5	81.1	77.5	78.1	76.3	76.3	72.7	74.2
+ TISD (ours)	60.7	63.9	80.4	80.8	74.7	75.1	78.2	78.3	73.5	74.5

Table 2: Best Avg@128 (%) on SciKnowEval L3 without rich environment feedback (Qwen3-8B). Steps and Hours denote training budgets of 200 steps and 10 hours, respectively. Average is the unweighted mean across the four disciplines. Bold and underlining mark the highest and second-highest means in each column among the reported methods.

Figure 6: Mean test-case pass rate (Avg@4, %) across training data scales, through 80 training steps in the rich-feedback setting (Qwen3-8B). LCB-Large combines v1–v6, and LCB-Small contains v6. TISD first exceeds GRPO’s maximum on LCB-Large with a

33\%

speedup and reaches SDPO’s maximum on LCB-Small with a

56\%

speedup.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: TISD pseudocode. Each record contains a prompt x , response y , binary reward, and privileged context c . peak_divergence_pos selects the eligible position with the largest teacher–student divergence. Loss and optimization settings appear in Appendix D .

Figure 10: CDF of the reverse-KL concentration ratio

\mathrm{KL}(t^{*})/\sum_{t}\mathrm{KL}(t)

202

failed Qwen3-1.7B LiveCodeBench v6 rollouts.

	Role	1.7B	4B	8B
Student	Algorithmic	36.1%	11.6%	11.6%
	Naming	11.4%	3.9%	5.5%
	Formatting	7.4%	13.3%	12.7%
	Non-code text	45.0%	71.3%	70.2%
Teacher	Algorithmic	38.6%	10.5%	14.4%
	Naming	9.9%	4.4%	3.9%

Appendix

Table 3: GPT-5.4 semantic annotations of student and teacher tokens at

t^{*}

on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Entries are percentages within each model scale.

Figure 11: Student-to-teacher semantic-role transitions at $t^{*}$ on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Each row shows the distribution of teacher-token roles conditioned on the corresponding student-token role. The dominant diagonal shows that disagreement usually remains within a semantic category.

		Peak divergence	First divergence	Random
Semantic role (student token at $t^{*}$ )
	Algorithmic	11.6%	0.6%	20.4%
	Naming	5.5%	0.0%	3.9%
	Formatting	12.7%	15.5%	11.0%
	Non-code text	70.2%	84.0%	64.6%
Execution-based success rate at $t^{*}$

Appendix

Table 4: Effect of branch-position selection on

181

failed Qwen3-8B LiveCodeBench v6 rollouts. Semantic roles refer to the student token at the selected position and are assigned by GPT-5.4. Execution success is measured after either teacher continuation or student regeneration from the forced teacher or original token. First divergence denotes the first position where the student and teacher argmax tokens differ.

		Step 0 (202)	Step 20 (183)	Step 40 (192)	Step 60 (176)	Step 80 (145)
Pass rate		0.211	0.285	0.250	0.312	0.434
Semantic role (student token at $t^{*}$ )
	Algorithmic	36.1%	35.5%	36.5%	35.8%	26.2%
	Naming	11.4%	15.3%	12.0%	14.8%	9.7%
	Formatting	7.4%	8.2%	13.5%	11.4%	15.9%
	Non-code text	45.0%	41.0%	38.0%	38.1%	48.3%

Appendix

Table 5: Evolution of rollout pass rate and semantic roles at the peak-divergence position during Qwen3-1.7B SDPO training on LiveCodeBench v6, using

256

rollouts per update. Pass rate is computed over all rollouts, while GPT-5.4 semantic-role percentages are computed over failed rollouts at each update; the corresponding failure counts are shown in parentheses.

		Fail (129)	Succ (92)
Semantic role (student token at $t^{*}$ )
	Algorithmic	7.8%	15.2%
	Naming	3.1%	10.9%
	Formatting	7.8%	10.9%
	Non-code text	81.4%	63.0%
Execution-based success rate at $t^{*}$

Appendix

Table 6: Reverse-KL diagnostic on

221

non-truncated OLMo3-7B-Instruct LiveCodeBench v6 rollouts. Entries report semantic-role distributions at

t^{*}

and execution success after continuation from

t^{*}

, separately for failed and successful original rollouts.

		Biology (148)	Chemistry (143)	Physics (128)	Materials (134)	Pooled (553)
Execution-based success rate at $t^{*}$
	Teacher continuation	96.6%	93.0%	93.8%	90.3%	93.5%
	Student cont. (teacher token)	31.1%	41.3%	40.6%	45.5%	39.4%
	Student cont. (original token)	22.3%	25.9%	30.5%	19.4%	24.4%

Appendix

Table 7: Verifier-based intervention on failed Qwen3-8B SciKnowEval L3 rollouts at SDPO update 0. The teacher is conditioned on the first successful same-problem rollout in source order, excluding the target rollout, and

t^{*}

is the valid response position with maximum reverse KL. Each domain contains

32

problems with eight rollouts per problem. Numbers in parentheses denote failed-rollout counts.

	1.7B	4B	8B
Problem groups (out of 32)
All-fail	23 (71.9%)	19 (59.4%)	19 (59.4%)
All-success	5 (15.6%)	6 (18.8%)	5 (15.6%)
Mixed	4 (12.5%)	7 (21.9%)	8 (25.0%)
Failed rollouts
Total	202	181	181

Appendix

Table 8: Coverage of group-relative outcome supervision for Qwen3-{1.7B, 4B, 8B} on

32

LiveCodeBench v6 problems at update 0, with eight rollouts per problem. Counts and percentages partition problem groups by verifier outcome and failed rollouts by whether their normalized GRPO advantage is zero. All-fail and all-success groups receive zero normalized advantage.

		1.7B	4B	8B
By semantic role (correct / total)
	Algorithmic	9/73 (12.3%)	1/21 (4.8%)	0/21 (0.0%)
	Naming	4/23 (17.4%)	0/7 (0.0%)	2/10 (20.0%)
	Formatting	0/15 (0.0%)	1/24 (4.2%)	2/23 (8.7%)
	Non-code text	3/91 (3.3%)	17/129 (13.2%)	20/127 (15.7%)

Appendix

Table 9: Verifier success of teacher continuations from

t^{*}

, stratified by GPT-5.4 semantic role, on failed Qwen3-{1.7B, 4B, 8B} LiveCodeBench v6 rollouts. Each entry reports successful continuations over failed rollouts assigned to the corresponding role.

	Student succeeds	Student fails
Teacher succeeds	17	7
Teacher fails	1	156

Appendix

Table 10: Paired continuation outcomes on

181

failed Qwen3-8B LiveCodeBench v6 rollouts. Both conditions preserve the original prefix, force the teacher’s argmax token at peak divergence, and differ only in whether the teacher or student generates the suffix. Success requires passing all private tests.

Panel of Figure 12 — Figure 12: Example of trajectory redirection on a failed Qwen3-8B LiveCodeBench response. The peak-divergence token is the naming token j , which carries $12.3\%$ of response KL. Forcing the teacher-preferred k and regenerating with the student changes the algorithm from iterating over $(i,k)$ to iterating over $(i,j)$ , producing an integer $k=2j-i$ and a correct solution.

Panel of Figure 13 — Figure 13: Repetitive meta-commentary in a TRD student validation response. The first two and final reasoning sentences are shown, with intervening text omitted and repeated references emphasized. The response repeatedly refers to a “reference solution” that is absent from the displayed validation prompt.

Parameters	GRPO	SDPO	TRD	TISD
General
Model	Qwen3-8B, Qwen3-14B, Olmo3-7B-Instruct
Thinking (Qwen3)	False	False	False	False
Data
Max. prompt length	2048	2048	2048	2048
Max. response length	8192	8192	8192	8192

Appendix

Table 11: Hyperparameters for learning with rich environment feedback on LiveCodeBench.

Parameters	GRPO	SDPO	TRD	TISD
General
Model	Qwen3-8B	Qwen3-8B	Qwen3-8B	Qwen3-8B
Thinking	False	False	False	False
Data
Max. prompt length	2048	2048	2048	2048
Max. response length	8192	8192	8192	8192

Appendix

Table 12: Hyperparameters for learning without rich environment feedback on SciKnowEval L3. Paired GRPO values denote off-policy / on-policy settings.

Method	TISD	SDPO	GRPO	TRD
Average time (min/step)	$16.60$	$15.05$	$16.69$	$16.36$

Appendix

Table 13: Average wall-clock time per training step (minutes) for Qwen3-8B on LiveCodeBench v6 using four NVIDIA B200 GPUs.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
Aug 9, 2026Yangyang Feng, Zhuoyan Feng, Junlan ChenOn-Policy Self-Distillation Knowledge Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University
Branching On-Policy Distillation
Oct 9, 2026Dongyang Chen, Shudong Liu, Shubo Li +4On-Policy Distillation Language Model Distillation
Tsinghua University · SenseTime Research · Peking University
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
Aug 10, 2026Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1On-Policy Self-Distillation Language Model Distillation
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP

Aug 9, 2026J/K move · Enter open · S save

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation

Yangyang Feng, Zhuoyan Feng, Junlan Chen

1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University

On-Policy Self-Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student's response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged. PAST preserves the student's next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A

2\times2

factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.

Abstract

On-Policy Self-Distillation

Explore similar work

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation

Branching On-Policy Distillation

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation