Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond

Zheng Jiang; Yiming Chen; Houde Qian; Ling Li; Chaoyang Li; Lifeng Sun

cs.CLOct 23, 2026

Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond

Authors: Zheng Jiang, Yiming Chen, Houde Qian, Ling Li, Chaoyang Li, Lifeng Sun

Organizations: Tsinghua University

Abstract

On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires identifying both where guidance is needed and whether the student can sustain the interaction afterward. We introduce Frontier-OPD, which addresses this problem through the student's learning frontier: decision situations that remain challenging but admit useful autonomous continuation after brief teacher guidance. To operationalize this frontier, Frontier-OPD organizes on-policy experience into a shared-state trajectory graph, where trajectory-balanced aggregation of teacher-student disagreement identifies distinct intervention candidates while preserving the histories needed for replay. Short teacher bridges are then assessed by comparing agreement over the bridge and the ensuing student continuation with an equally long segment of the original trajectory, providing a behavioral proxy for guidance that the current student can build on. Accepted teacher responses augment ordinary distillation data, and fresh rollouts renew the frontier estimate as the policy changes. Experiments on ALFWorld, WebShop, and ScienceWorld report higher mean autonomous performance than the compared distillation baselines across the evaluated student scales and teacher configurations, with ablations indicating complementary effects of candidate selection and guidance filtering. In one ALFWorld timing setting, the additional collection cost is offset by a shorter time to a target success rate.

Figures & tables

Figure 1: Motivation for Frontier-OPD. Repeated student–teacher disagreement is treated as evidence of a potential learning need; a teacher bridge may be useful when it enables the student to continue; their overlap is operationalized as the learning frontier used for targeted distillation. The shared results strip reports success rate (highlighting the student, teacher, and Frontier-OPD outcomes.

Figure 2: Overview of Frontier-OPD. Trajectory-balanced disagreement aggregation ranks shared-state frontier candidates, each of which supplies a concrete history for teacher intervention. Matched-window validation assesses the bridge and subsequent student behavior before accepted teacher responses enter distillation, after which fresh rollouts renew the frontier estimate.

	ALFWorld	WebShop		ScienceWorld
Method	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$
Qwen3-32B teacher $\bm{\rightarrow}$ Qwen3-1.7B student
Student (zero-shot)	8.2	40.5	31.3	8.3	1.8
Teacher (zero-shot)	47.8	54.0	52.0	45.1	17.6
OPD	$22.1\pm 2.8$	$35.2\pm 2.0$	$33.7\pm 2.9$	$27.5\pm 1.5$	$6.2\pm 1.0$
TCOD-F2B	$31.1\pm 3.0$	$41.3\pm 1.9$	$39.3\pm 1.5$	$29.8\pm 0.8$	$8.5\pm 0.9$

Table 1: Performance comparison with Qwen3-32B as teacher and Qwen3-1.7B or Qwen3-4B as student. Trained-policy results are mean

\pm

sample standard deviation over three seeds. Boldface marks the highest mean among trained students within each configuration; blue shading highlights Frontier-OPD.

	Qwen3-1.7B student			Qwen3-4B student
	ALFWorld	WebShop		ALFWorld	WebShop
Method	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$
Student (zero-shot)	8.2	40.5	31.3	21.1	33.0	29.3
Teacher (zero-shot)	82.1	64.0	62.3	82.1	64.0	62.3
OPD	$55.2\pm 4.2$	$60.5\pm 2.7$	$58.3\pm 2.1$	$67.9\pm 3.9$	$64.1\pm 2.8$	$64.3\pm 2.1$
TCOD-F2B	$57.7\pm 2.2$	$60.1\pm 2.4$	$59.3\pm 2.1$	$66.9\pm 0.9$	$63.8\pm 2.8$	$64.3\pm 2.1$

Table 2: Teacher-configuration comparison with Qwen3-8B-RL teacher and Qwen3-1.7B or Qwen3-4B students. Trained-policy results are mean

\pm

sample standard deviation over three seeds. Boldface marks the highest mean among trained students within each configuration; blue shading highlights Frontier-OPD.

	Qwen3-1.7B student			Qwen3-4B student
Variant	ALFWorld	WebShop		ALFWorld	WebShop
	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$	SR $\uparrow$	Score $\uparrow$	SR $\uparrow$
Student (zero-shot)	8.2	40.5	31.3	21.1	33.0	29.3
Teacher (zero-shot)	82.1	64.0	62.3	82.1	64.0	62.3
w/o bridge data	$62.2\pm 3.0$	$58.4\pm 1.5$	$58.4\pm 1.6$	$70.9\pm 1.3$	$61.7\pm 1.2$	$62.0\pm 1.6$
Random selection	$58.7\pm 3.5$	$56.0\pm 1.8$	$56.7\pm 1.5$	$67.4\pm 2.3$	$59.3\pm 1.7$	$60.3\pm 1.4$

Table 3: Core component ablations with a Qwen3-8B-RL teacher and Qwen3-1.7B or Qwen3-4B students on ALFWorld and WebShop. The full configuration uses

M=4

B=4

K=1

H=3

, and

\eta=0

. Trained-policy results are mean

\pm

sample standard deviation over three seeds; higher is better.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	ALFWorld	WebShop	ScienceWorld
Environment turn limit	30	15	30
Maximum prompt tokens	10,240	10,240	10,240
Maximum response tokens	512	512	512
Training temperature	1.0	1.0	1.0
Evaluation temperature	0.4	0.4	0.4
Evaluation task-set budget	134 steps	100 steps	1,308 steps

Appendix

Table 4: Benchmark-specific evaluation and training settings used by the Frontier-OPD workflows.

1	for collection iteration $k=0,1,\ldots$ do
2	Initialize $\mathcal{D}_{k}^{\mathrm{stu}}\leftarrow\emptyset$ and $\mathcal{D}_{k}^{\mathrm{br}}\leftarrow\emptyset$ .
3	for task $x$ in the current batch do
4	Sample $M$ student trajectories from the prescribed starting histories.
5	Store ordinary student responses in $\mathcal{D}_{k}^{\mathrm{stu}}$ ; retain contexts, actions, and both models’ token log-probabilities.
6	Construct $\mathcal{G}_{k}(x)$ from shared state keys and observed transitions.
7	Group eligible visits by node and trajectory; compute $s_{k}(v)$ using trajectory-balanced weights in equation 5 .

Appendix

Table 8

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 702 papers

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
Aug 3, 2026Chishui Chen, Yaoyou Fan, Te Sun +11LLM Agent Training On-Policy Distillation
Meituan LongCat Interaction · Peking University · Shanghai Jiao Tong University +6
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Sep 28, 2026Tong Zhang, Zhou Liu, Yihao Liu +8Language Model Distillation On-Policy Distillation
Qwen Large Model Application Team, Alibaba · Peking University · University of Chinese Academy of Sciences +1
UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents
Sep 27, 2026Wenbo Zhang, Pengcheng Xu, Weizhi Du +2LLM Agents Uncertainty-Aware Knowledge Distillation
University of California, Irvine · University of Michigan, Ann Arbor

Aug 3, 2026J/K move · Enter open · S save

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation

Chishui Chen, Yaoyou Fan, Te Sun, Yi Yang+10

Meituan LongCat Interaction · Peking University · Shanghai Jiao Tong University+6

LLM Agent Training On-Policy Distillation Knowledge Distillation RL for LLM Agents

Open paper AskPDF arXiv

On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.

Abstract

LLM Agents

Explore similar work

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation

Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents

UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents