On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires identifying both where guidance is needed and whether the student can sustain the interaction afterward. We introduce Frontier-OPD, which addresses this problem through the student's learning frontier: decision situations that remain challenging but admit useful autonomous continuation after brief teacher guidance. To operationalize this frontier, Frontier-OPD organizes on-policy experience into a shared-state trajectory graph, where trajectory-balanced aggregation of teacher-student disagreement identifies distinct intervention candidates while preserving the histories needed for replay. Short teacher bridges are then assessed by comparing agreement over the bridge and the ensuing student continuation with an equally long segment of the original trajectory, providing a behavioral proxy for guidance that the current student can build on. Accepted teacher responses augment ordinary distillation data, and fresh rollouts renew the frontier estimate as the policy changes. Experiments on ALFWorld, WebShop, and ScienceWorld report higher mean autonomous performance than the compared distillation baselines across the evaluated student scales and teacher configurations, with ablations indicating complementary effects of candidate selection and guidance filtering. In one ALFWorld timing setting, the additional collection cost is offset by a shorter time to a target success rate.
Figures & tables
Figure 1: Motivation for Frontier-OPD. Repeated student–teacher disagreement is treated as evidence of a potential learning need; a teacher bridge may be useful when it enables the student to continue; their overlap is operationalized as the learning frontier used for targeted distillation. The shared results strip reports success rate (highlighting the student, teacher, and Frontier-OPD outcomes.
Figure 2: Overview of Frontier-OPD. Trajectory-balanced disagreement aggregation ranks shared-state frontier candidates, each of which supplies a concrete history for teacher intervention. Matched-window validation assesses the bridge and subsequent student behavior before accepted teacher responses enter distillation, after which fresh rollouts renew the frontier estimate.
ALFWorld
WebShop
ScienceWorld
Method
SR ↑
Score ↑
SR ↑
Score ↑
SR ↑
Qwen3-32B teacher → Qwen3-1.7B student
Student (zero-shot)
8.2
40.5
31.3
8.3
1.8
Teacher (zero-shot)
47.8
54.0
52.0
45.1
17.6
OPD
22.1±2.8
35.2±2.0
33.7±2.9
27.5±1.5
6.2±1.0
TCOD-F2B
31.1±3.0
41.3±1.9
39.3±1.5
29.8±0.8
8.5±0.9
Table 1: Performance comparison with Qwen3-32B as teacher and Qwen3-1.7B or Qwen3-4B as student. Trained-policy results are mean ± sample standard deviation over three seeds. Boldface marks the highest mean among trained students within each configuration; blue shading highlights Frontier-OPD.
Qwen3-1.7B student
Qwen3-4B student
ALFWorld
WebShop
ALFWorld
WebShop
Method
SR ↑
Score ↑
SR ↑
SR ↑
Score ↑
SR ↑
Student (zero-shot)
8.2
40.5
31.3
21.1
33.0
29.3
Teacher (zero-shot)
82.1
64.0
62.3
82.1
64.0
62.3
OPD
55.2±4.2
60.5±2.7
58.3±2.1
67.9±3.9
64.1±2.8
64.3±2.1
TCOD-F2B
57.7±2.2
60.1±2.4
59.3±2.1
66.9±0.9
63.8±2.8
64.3±2.1
Table 2: Teacher-configuration comparison with Qwen3-8B-RL teacher and Qwen3-1.7B or Qwen3-4B students. Trained-policy results are mean ± sample standard deviation over three seeds. Boldface marks the highest mean among trained students within each configuration; blue shading highlights Frontier-OPD.
Qwen3-1.7B student
Qwen3-4B student
Variant
ALFWorld
WebShop
ALFWorld
WebShop
SR ↑
Score ↑
SR ↑
SR ↑
Score ↑
SR ↑
Student (zero-shot)
8.2
40.5
31.3
21.1
33.0
29.3
Teacher (zero-shot)
82.1
64.0
62.3
82.1
64.0
62.3
w/o bridge data
62.2±3.0
58.4±1.5
58.4±1.6
70.9±1.3
61.7±1.2
62.0±1.6
Random selection
58.7±3.5
56.0±1.8
56.7±1.5
67.4±2.3
59.3±1.7
60.3±1.4
Table 3: Core component ablations with a Qwen3-8B-RL teacher and Qwen3-1.7B or Qwen3-4B students on ALFWorld and WebShop. The full configuration uses M=4 , B=4 , K=1 , H=3 , and η=0 . Trained-policy results are mean ± sample standard deviation over three seeds; higher is better.
Figure 6
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
ALFWorld
WebShop
ScienceWorld
Environment turn limit
30
15
30
Maximum prompt tokens
10,240
10,240
10,240
Maximum response tokens
512
512
512
Training temperature
1.0
1.0
1.0
Evaluation temperature
0.4
0.4
0.4
Evaluation task-set budget
134 steps
100 steps
1,308 steps
Appendix
Table 4: Benchmark-specific evaluation and training settings used by the Frontier-OPD workflows.
1
for collection iteration k=0,1,… do
2
Initialize Dkstu←∅ and Dkbr←∅ .
3
for task x in the current batch do
4
Sample M student trajectories from the prescribed starting histories.
5
Store ordinary student responses in Dkstu ; retain contexts, actions, and both models’ token log-probabilities.
6
Construct Gk(x) from shared state keys and observed transitions.
7
Group eligible visits by node and trajectory; compute sk(v) using trajectory-balanced weights in equation 5 .
On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.
Chishui Chen, Yaoyou Fan, Te Sun +11
Meituan LongCat Interaction · Peking University · Shanghai Jiao Tong University +6
On-policy distillation (OPD) trains a student on its own trajectories with dense teacher supervision. Recent work on OPD for multi-turn autonomous agents often treats large teacher-student token-level distributional gaps as promising intervention points, linking larger gaps to a greater need for correction. Yet, our empirical analysis reveals a supervision-benefit mismatch: large gaps can be benign, while small gaps can be outcome-critical. Teacher-student gaps capture differences at the current turn, whereas the benefit of teacher guidance depends on how the current student interacts with the environment afterward. The student may still succeed despite choosing an action that differs from the teacher's, while a teacher-preferred action may lead to a state from which the student cannot complete the task. Local gaps alone are therefore not enough to determine whether teacher guidance benefits the current student. Effective supervision should instead emphasize guidance that the current student can translate into better final task outcomes. Accordingly, we propose Outcome-Guided On-Policy Distillation (OG-OPD), which applies trajectory-relative weighting to teacher supervision and calibrates these weights using final task outcomes from paired student continuations. This calibration selectively strengthens supervision on the student's original trajectories at turns where teacher guidance benefits the current student. Across ALFWorld, ScienceWorld, and WebShop, OG-OPD consistently outperforms baselines under diverse settings. It improves task success rates by 3.6-17.7 percentage points over vanilla OPD and by up to 7.0 percentage points over the strongest baseline.
Tong Zhang, Zhou Liu, Yihao Liu +8
Qwen Large Model Application Team, Alibaba · Peking University · University of Chinese Academy of Sciences +1
On-policy distillation (OPD) trains a student on its own rollouts using dense supervision from a teacher. In multi-turn environments, a mistake at a critical decision step can redirect the subsequent rollout toward poor outcomes. We use low teacher confidence on student actions to select high-uncertainty steps for correction. In a controlled ALFWorld study, a single teacher correction at a low-confidence step improves subsequent student behavior and task success, motivating selective intervention during distillation. We propose UOPD, an uncertainty-aware intervention method for on-policy distillation. At low-uncertainty turns, UOPD executes student actions and applies the standard OPD loss. At high-uncertainty turns, it samples and executes teacher actions and trains the student to imitate them through supervised fine-tuning, which minimizes forward Kullback-Leibler divergence in expectation. UOPD utilizes adaptive uncertainty thresholds to target a scheduled intervention rate. Empirically, we evaluate UOPD across a broad range of agentic tasks, including ALFWorld, WebShop, and Search, demonstrating its superior performance over OPD methods and their variants. UOPD improves WebShop score by up to 15.8% relative to standard OPD.
Wenbo Zhang, Pengcheng Xu, Weizhi Du +2
University of California, Irvine · University of Michigan, Ann Arbor