On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.
Figures & tables
Figure 1: Overview of State-OPD. The teacher conditions on the interaction state rather than prior student reasoning; the student retains its full history and samples additional complete responses at states with high response entropy.
Figure 2: Removing prior student reasoning improves Qwen3-32B teacher continuation at ALFWorld states visited by Qwen3-0.6B and Qwen3-4B students and changes the reverse-KL depth profile. Continuations run for at most 30 additional turns. Paired intervals appear in Table 3 .
Figure 3: Resampling reasoning reveals more actions at the same state. (a) For each state, we draw 16 responses with fixed reasoning and 16 complete responses. Side-by-side bars show the percentage of states in each action-count bin within each condition. (b) Faint points show individual states; solid markers show equal-count-bin means with 95% trajectory-bootstrap intervals. Higher response entropy accompanies more additional actions.
Figure 4: Training dynamics on ALFWorld for a Qwen3-4B student distilled from Qwen3-32B. (a,b) Seen/Unseen averages. (c) Success-rate gap at matched supervised-turn budgets (State-OPD minus linearly interpolated OPD). (d) Mean thinking tokens per root-trajectory turn; faint lines show step values and marked lines show centered 11-step moving averages.
Method
Seen SR ↑
Unseen SR ↑
Avg. SR ↑
State-OPD (Ours)
43.39
45.71
44.55
w/o State-Centered Response Allocation
40.36
43.10
41.73
w/o State-Grounded Supervision
33.39
37.31
35.35
Vanilla OPD
30.89
33.96
32.42
Table 2: State-OPD ablation on ALFWorld. A Qwen3-4B student is distilled from Qwen3-32B; Avg. is the mean of Seen and Unseen SR.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Student
Anchor depth
Th
Ts
ΔT [95% CI]
Qwen3-0.6B
Early
5.47
17.97
+12.50[+4.41,+21.67]
Middle
2.34
15.63
+13.28[+6.45,+20.51]
Late
0.00
14.06
+14.06[+4.61,+25.78]
Qwen3-4B
Early
14.84
15.63
+0.78[−4.84,+6.90]
Middle
17.97
21.09
+3.13[−9.48,+17.74]
Late
10.16
30.47
+20.31[+5.47,+37.07]
Appendix
Table 3: Teacher continuation from matched ALFWorld anchors. Each row uses 32 anchors from 30 source episodes and four repeats per condition (128 paired continuations). Th and Ts are success rates (%); ΔT is their paired difference in percentage points with a 95% source-episode bootstrap interval.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Training steps
200
Root tasks per step
16
Baseline rollouts per task
4
Learning rate
1×10−6 (constant)
Appendix
Table 4: Training hyperparameters across environments.
Hyperparameter
Value
Generation
Temperature
0.4
Top- p
1.0
Top- k
−1
Evaluation protocol
Evaluation interval
20 training steps
Appendix
Table 5: Evaluation hyperparameters across environments.
Environment
Train
Seen evaluation
Unseen evaluation / test
ALFWorld
3,553
140
134
ScienceWorld
2,294 (17)
456 (17)
782 (13)
WebShop
4,096
–
500
Appendix
Table 6: Training and evaluation sets. Parentheses give the number of ScienceWorld task types.
Hyperparameter
Value
State-OPD (ours)
Concentration factor β
2
TCOD-F2B
Starting horizon
1 turn
Pacing factor η
2
Guided-OPD
Appendix
Table 7: Method-specific hyperparameters.
Setting
Baseline
Qwen3-1.7B
Qwen3-4B
ALFWorld Seen
Vanilla OPD
+4.82[+0.54,+9.29]
+12.50[+8.39,+16.79]
TCOD-F2B
+2.14[−2.32,+6.61]
+14.11[+9.64,+18.57]
Guided-OPD
+3.39[−0.89,+7.68]
+16.25[+11.43,+21.07]
TurnOPD
+3.93[−0.36,+8.39]
+16.61[+11.79,+21.61]
ALFWorld Unseen
Vanilla OPD
+3.54[−1.49,+8.58]
+11.75[+6.53,+17.35]
TCOD-F2B
+7.46[+2.05,+13.06]
+10.45[+5.04,+15.86]
Appendix
Table 8: Paired success-rate differences, State-OPD minus the indicated baseline, in percentage points. Brackets give 95% task-bootstrap intervals. Positive differences favor State-OPD. Each student column contains 20 comparisons.
Figure 6: State-OPD reduces persistent failed-action loops throughout ALFWorld training. For a Qwen3-4B student distilled from Qwen3-32B, each panel shows the share of failed-action starts with at least k further identical actions, each returning Nothing happens.
Figure 7: State selection and allocation sensitivity on ALFWorld. (a) Share of additional distinct actions in a 16-response probe recovered by entropy-ranked or random selection under matched budgets; error bars are 95% root-cluster bootstrap intervals. (b) Seen/Unseen SR across β for a Qwen3-4B student distilled from Qwen3-32B; each setting has one training run.
Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice. On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students, but we find that it suffers a characteristic failure mode in this setting: small student errors compound across turns and push the trajectory out of the teacher's familiar state distribution, so the teacher's supervision becomes least reliable precisely where the student needs it most. We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout and schedules the teacher's intervention probability along a curriculum that decays to zero. Strong guidance keeps early trajectories close to the teacher distribution and is then gradually withdrawn to recover the purely on-policy regime used at inference. On ALFWorld, ScienceWorld, and WebShop, distilling Qwen3 students from a Qwen3-30B-A3B teacher, Guided-OPD yields average relative gains of 21.1% in Score and 25.5% in Success Rate over vanilla OPD, with larger gains on smaller students.
Gengsheng Li, Mao Zheng, Mingyang Song +8
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4
On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning. However, in multi-turn agentic tasks, student decisions shape subsequent observations, causing early errors to accumulate across turns. The resulting trajectories can drift away from the teacher's rollout distribution, making the teacher's token-level supervision less reliable or even counterproductive for OPD training. To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD. During multi-turn interaction, STI-OPD uses teacher intervention guided by teacher-student policy discrepancy to replace the student's proposed action with a teacher-generated one to maximize the acquisition of reliable supervision. We further develop a stochastic intervention strategy, addressing the limitations of previous threshold-based or fixed-schedule approaches, that estimates policy discrepancy using KL divergence and maps it to an intervention probability. By sampling whether to intervene from this probability, STI-OPD adaptively balances teacher control with student exploration. To learn from the resulting mixed-policy trajectories, we introduce an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective. Across tool-integrated reasoning and long-horizon interaction, STI-OPD outperforms the strongest prior OPD baseline on every evaluated benchmark and student size. Ablations further show that both discrepancy-guided intervention and importance weighting contribute to these gains.
Junnan Liu, Linhao Luo, Zhijun Chen +3
Department of Data Science and AI, Faculty of Information Technology, Monash University, Australia · The Hong Kong Polytechnic University, Hong Kong SAR, China · Zhongguancun Laboratory, Beijing, P.R. China
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.