Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

Shu Zhao; Tan Yu

cs.CLOct 20, 2026

Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

Authors: Shu Zhao, Tan Yu

Organizations: NVIDIA

Abstract

Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout-scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell. Project page: https://tree-shu-zhao.github.io/r2s/.

Figures & tables

Figure 1: OPD can reinforce the delays its teacher lets pass. (a) The teacher’s scores let delaying thoughts pass and the episode runs out of turns; its own takeover from that state finishes. (b) With the takeover in context, the teacher scores the delay lower; R2S trains a copy to do so without it.

	Success (%)			Drop (%)
Environment	Student	Takeover	Rescued	Hedging	Searching	Other
ALFWorld	$39.9$	$44.2$	$45.2$	$8.2$	$6.2$	$3.5$
WebShop	$6.1$	$10.2$	$14.4$	$8.3$	$7.7$	$4.2$
ScienceWorld	$18.6$	$27.5$	$51.7$	$13.0$	$16.6$	$5.0$

Table 1: The teacher finishes from the student’s states, but its plain scores let the student delay. States: highest-disagreement turns from turn five on; Rescued : lost states a takeover finishes. Drop : in the teacher’s probability of the student’s words with a finishing takeover in context.

Figure 2: R2S repairs the teacher’s scores on the full assistant suffix before ordinary OPD. With a selected takeover from the untrained student’s state in context, the frozen teacher

T_{0}

scores the student’s thinking and actions from the divergence turn onward (

q^{+}

). A copy learns these targets without the takeover and becomes the sole OPD scorer.

Method	Qwen3.5-9B $\to$ Qwen3.5-2B / 4B
	ALFWorld		WebShop		ScienceWorld
	2B	4B	2B	4B	2B	4B
Teacher alone	$63.1{\pm}1.7$		$33.5{\pm}0.3$		$32.0{\pm}1.8$
Student alone	$15.1{\pm}1.8$	$43.4{\pm}1.0$	$17.6{\pm}1.1$	$33.7{\pm}0.9$	$3.1{\pm}0.4$	$28.0{\pm}1.5$
Vanilla OPD	$58.0{\pm}2.4$	$65.8{\pm}2.6$	$35.7{\pm}1.6$	$34.6{\pm}2.9$	$23.7{\pm}2.0$	$34.6{\pm}3.5$
TCOD-F2B	$62.7{\pm}2.0$	$59.6{\pm}1.5$	$33.1{\pm}0.5$	$35.1{\pm}0.9$	$28.0{\pm}2.6$	$30.6{\pm}0.7$

Table 2: Main results: R2S versus vanilla OPD, TCOD-F2B, Guided-OPD and FutureBridge. Success (%) of one training run per cell, mean

\pm

std over three decodes of its final checkpoint; every split in Table 12 . The best and second best of the five trained students in each column are highlighted.

Panel of Figure 3 — Figure 3: R2S students separate from every baseline during training. Qwen3.5-9B teaches Qwen3.5-4B; mean $\pm$ one standard deviation over three training runs; dashed: the teacher acting alone. Other teacher–student pairs are in Appendix F .

Environment	Student alone	Vanilla OPD	TCOD-F2B	Guided-OPD	FutureBridge	R2S (ours)
ALFWorld	$43.4{\pm}1.0$	$49.9{\pm}2.6$	$49.7{\pm}2.6$	$48.5{\pm}3.8$	$50.6{\pm}1.9$	$72.7{\pm}1.3$
WebShop	$33.7{\pm}0.9$	$33.3{\pm}2.2$	$31.3{\pm}0.9$	$34.0{\pm}1.8$	$31.1{\pm}1.5$	$58.1{\pm}2.5$
ScienceWorld	$28.0{\pm}1.5$	$28.2{\pm}2.3$	$27.2{\pm}0.8$	$29.3{\pm}1.8$	$29.2{\pm}2.9$	$42.4{\pm}3.4$

Table 3: Self-distillation: Qwen3.5-4B teaches Qwen3.5-4B. R2S first repairs the 4B teacher on its own takeovers; the other arms learn from the unrepaired 4B.

	ALFWorld	WebShop	ScienceWorld
Vanilla OPD	$65.8{\pm}2.6$	$34.6{\pm}2.9$	$34.6{\pm}3.5$
Consensus vote	$81.9{\pm}2.4$	$63.1{\pm}1.6$	$39.5{\pm}2.9$
LLM judge	$82.6{\pm}0.9$	$55.9{\pm}1.4$	$50.3{\pm}1.5$
Ground truth	$85.9{\pm}1.2$	$65.0{\pm}1.4$	$49.0{\pm}1.1$

Table 4: Ablation on takeover selection . Success (%). Ground truth reads the task reward and is not ranked.

Takeovers	Informed targets	ALFWorld	WebShop	ScienceWorld
✗	✗	$65.8{\pm}2.6$	$34.6{\pm}2.9$	$34.6{\pm}3.5$
✓	✗	$76.8{\pm}3.5$	$46.7{\pm}1.3$	$41.0{\pm}3.0$
✓	✓	$81.9{\pm}2.4$	$55.9{\pm}1.4$	$47.8{\pm}3.0$

Table 5: Ablation on components . Success (%); the rows are vanilla OPD, label-SFT and R2S.

	ALFWorld	WebShop	ScienceWorld	Avg.
Divergence turn	$82.5{\pm}2.3$	$48.5{\pm}1.4$	$48.7{\pm}0.4$	59.9
Prefix	$81.0{\pm}1.8$	$52.7{\pm}0.8$	$47.6{\pm}2.1$	60.5
Suffix	$81.9{\pm}2.4$	$55.9{\pm}1.4$	$47.8{\pm}3.0$	61.9

Table 6: Ablation on supervised turns . Success (%); the suffix, from the divergence turn to the end, is R2S.

Panel of Figure 4 — Figure 4: Repair moves the teacher toward its informed targets, and its student runs out of turns about half as often. Left: each teacher’s top-16 KL to $q^{+}$ on held-out states, with 95% goal-bootstrap intervals. Right: held-out episodes that end unfinished at the turn budget, mean $\pm$ std over three decodes.

Method	ALFWorld		WebShop		ScienceWorld
Method	Out of turns	Turns	Out of turns	Turns	Out of turns	Turns
Teacher alone	$39.8{\pm}1.5$	$20.0$	$56.7{\pm}0.9$	$11.9$	$35.7{\pm}1.9$	$19.7$
Student alone	$55.0{\pm}3.6$	$22.3$	$56.7{\pm}1.2$	$11.9$	$38.6{\pm}2.1$	$17.7$
Vanilla OPD	$34.3{\pm}4.7$	$19.0$	$55.9{\pm}4.3$	$11.8$	$31.9{\pm}1.5$	$18.9$
R2S (ours)	$19.5{\pm}2.2$	$14.7$	$27.5{\pm}2.7$	$9.0$	$17.1{\pm}0.7$	$14.5$

Table 7: The vanilla student inherits the teacher’s endings; the repaired student does not. Held-out episodes that run out of turns (%) and turns per episode, seen split, over three decodes.

Method	ALFWorld			WebShop			ScienceWorld
Method	Turns	Searching	Task	Turns	Searching	Task	Turns	Searching	Task
Vanilla OPD	$20.7{\pm}0.9$	$15.9{\pm}1.2$	$4.0{\pm}0.3$	$11.5{\pm}0.4$	$7.8{\pm}0.5$	$3.6{\pm}0.1$	$18.2{\pm}0.5$	$10.9{\pm}0.6$	$6.9{\pm}0.7$
R2S	$15.6{\pm}0.5$	$11.4{\pm}0.6$	$3.7{\pm}0.3$	$8.8{\pm}0.2$	$4.1{\pm}0.1$	$4.7{\pm}0.1$	$14.6{\pm}0.3$	$8.2{\pm}0.6$	$6.1{\pm}0.3$

Table 8: The repaired student spends fewer turns searching. Turns per episode in the last ten training updates, mean

\pm

std over three training runs. Task turns act on the goal, such as taking or buying.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	RQ-free	Repairs external teacher	Freeze, then distill
TCOD-F2B ( Wang et al., 2026 )	✓	✗	✗
Guided-OPD ( Li et al., 2026a )	✓	✗	✗
FutureBridge † ( Chen et al., 2026 )	✓	✗	✗
ReOPD ( Liao et al., 2026 )	✓	✗	✗
SDFT / OPCD ( Shenfeld et al., 2026 ; Ye et al., 2026 )	✓	✗	✗
CANON ( Gkountouras et al., 2026 )	✓	✗	✗

Appendix

Table 9: Among these methods, only R2S repairs an external teacher without task reward and refreezes it for distillation. RQ-free: no task-reward query for distillation supervision. The two right columns concern an external scorer, not an evolving self-teacher. † FutureBridge uses a success-selected reference pool.

Environment	All	Turn $\geq 5$	Lost	States	Turns	Hedging	Searching	Committing	Action
	States			Scored		Against other		Drop (%)
ALFWorld	2,399	814	489	458	4,957	$2.4$ $[2.1,2.8]$	$1.8$ $[1.4,2.2]$	$6.1$	$0.1$
WebShop	2,398	689	647	107	906	$2.0$ $[1.3,2.9]$	$1.9$ $[0.9,3.0]$	$6.8$	$0.5$
ScienceWorld	2,205	682	555	403	4,085	$2.7$ $[2.4,3.1]$	$3.5$ $[3.0,4.1]$	$11.3$	$0.3$

Appendix

Table 10: The states and scores behind Table 1 . All : the untrained student’s highest-disagreement states with a replayable takeover; Turn

\geq 5

: those from turn five on, which Table 1 uses; Lost : of these, the states the student went on to lose. Scored : states where at least one takeover finishes, and the student’s turns through the mined one, which the drop columns read. Against other : a class’s mean change in log-probability over that of the rest of the thinking, with a 95% interval over states; Drop : as in Table 1 , for the student’s committing words and its action tokens.

Teacher $\to$ student	ALFWorld		WebShop		ScienceWorld
Teacher $\to$ student	Student	Takeover	Student	Takeover	Student	Takeover
Qwen3.5-9B $\to$ 2B	$11.9$	$54.6$	$5.0$	$21.8$	$1.2$	$31.2$
Qwen3.5-9B $\to$ 4B	$51.8$	$56.7$	$19.6$	$22.0$	$22.4$	$35.6$
Qwen3-30B-A3B $\to$ 4B	$33.3$	$41.8$	$10.4$	$14.0$	$10.2$	$36.6$
Qwen3-30B-A3B $\to$ 8B	$45.5$	$42.2$	$20.1$	$31.7$	$21.6$	$38.8$

Appendix

Table 11: The teacher’s takeovers finish from the student’s states in every cell. Success (%) of the untrained student and of the teacher’s

K{=}8

takeovers over all mined states, for every pair of Table 2 .

Panel A: Qwen3.5-9B $\to$ Qwen3.5-2B / 4B, three environments
Student	Split	Student alone	Vanilla OPD	TCOD-F2B	Guided-OPD	FutureBridge	R2S (ours)
ALFWorld (success, %)				teacher alone: seen $60.2{\pm}1.5$ , unseen $65.9{\pm}3.0$
Qwen3.5-2B	seen	$19.8{\pm}1.5$	$55.0{\pm}2.6$	$59.0{\pm}2.2$	$60.2{\pm}4.8$	$54.0{\pm}2.9$	$\mathbf{76.9{\pm}3.2}$
Qwen3.5-2B	unseen	$10.4{\pm}3.7$	$60.9{\pm}2.3$	$66.4{\pm}2.0$	$66.9{\pm}1.1$	$59.5{\pm}4.3$	$\mathbf{78.9{\pm}3.0}$
Qwen3.5-4B	seen	$45.0{\pm}3.6$	$65.7{\pm}4.7$	$59.3{\pm}2.6$	$60.2{\pm}4.4$	$58.1{\pm}2.9$	$\mathbf{80.5{\pm}2.2}$
Qwen3.5-4B	unseen	$41.8{\pm}3.9$	$65.9{\pm}1.1$	$60.0{\pm}1.6$	$62.9{\pm}1.1$	$58.7{\pm}4.5$	$\mathbf{83.3{\pm}3.1}$

Appendix

Table 12: Main results by split. The cells of Table 2 before averaging: ALFWorld and ScienceWorld seen / unseen, WebShop the 500 canonical goals / the disjoint 256-goal pool; protocol as in Table 2 . Bold: the best of the five trained arms where its gap to the runner-up exceeds the larger decode std; ties are unmarked.

Environment	Split	Student alone	Vanilla OPD	TCOD-F2B	Guided-OPD	FutureBridge	R2S (ours)
ALFWorld	seen	$45.0{\pm}3.6$	$53.1{\pm}2.3$	$50.5{\pm}3.2$	$50.0{\pm}4.5$	$56.4{\pm}3.8$	$\mathbf{69.5{\pm}2.3}$
ALFWorld	unseen	$41.8{\pm}3.9$	$46.8{\pm}4.1$	$49.0{\pm}4.1$	$47.0{\pm}3.4$	$44.8{\pm}0.0$	$\mathbf{75.9{\pm}1.6}$
WebShop	canonical	$33.7{\pm}0.9$	$33.3{\pm}2.2$	$31.3{\pm}0.9$	$34.0{\pm}1.8$	$31.1{\pm}1.5$	$\mathbf{58.1{\pm}2.5}$
ScienceWorld	seen	$26.7{\pm}1.8$	$25.5{\pm}1.8$	$25.7{\pm}2.1$	$30.0{\pm}0.0$	$28.1{\pm}3.9$	$\mathbf{45.2{\pm}5.7}$
ScienceWorld	unseen	$29.4{\pm}1.1$	$30.8{\pm}4.6$	$28.6{\pm}1.1$	$28.6{\pm}3.5$	$30.3{\pm}2.8$	$\mathbf{39.6{\pm}2.7}$

Appendix

Table 13: Self-distillation, every split. The cells of Table 3 by split (ALFWorld and ScienceWorld seen / unseen; WebShop canonical goals); protocol as in Table 3 . Bold: the best of the five trained arms where its gap to the runner-up exceeds the larger decode std.

Panel of Figure 5 — Figure 5: Success over training in every cell of Table 2 . Vanilla and R2S students, one decode every 25 updates (update 0: untrained). Solid: seen split or WebShop’s canonical goals; dotted: unseen split or its 256-goal pool; dashed: the teacher acting alone on the seen split.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Oct 18, 2026

Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD

Jike Zhong, Ming Li, Yuxiang Lai +9

University of Southern California · University of Florida · Emory University +6

On-Policy Distillation Agentic RL Selective Knowledge Distillation

Ask PDF

Oct 7, 2026

Stochastic Teacher Intervention for Agentic On-Policy Distillation

On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning. However, in multi-turn agentic tasks, student decisions shape subsequent observations, causing early errors to accumulate across turns. The resulting trajectories can drift away from the teacher's rollout distribution, making the teacher's token-level supervision less reliable or even counterproductive for OPD training. To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD. During multi-turn interaction, STI-OPD uses teacher intervention guided by teacher-student policy discrepancy to replace the student's proposed action with a teacher-generated one to maximize the acquisition of reliable supervision. We further develop a stochastic intervention strategy, addressing the limitations of previous threshold-based or fixed-schedule approaches, that estimates policy discrepancy using KL divergence and maps it to an intervention probability. By sampling whether to intervene from this probability, STI-OPD adaptively balances teacher control with student exploration. To learn from the resulting mixed-policy trajectories, we introduce an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective. Across tool-integrated reasoning and long-horizon interaction, STI-OPD outperforms the strongest prior OPD baseline on every evaluated benchmark and student size. Ablations further show that both discrepancy-guided intervention and importance weighting contribute to these gains.

Junnan Liu, Linhao Luo, Zhijun Chen +3

Department of Data Science and AI, Faculty of Information Technology, Monash University, Australia · The Hong Kong Polytechnic University, Hong Kong SAR, China · Zhongguancun Laboratory, Beijing, P.R. China

On-Policy Distillation Teacher-Student Learning RL for LLM Agents

Ask PDF

Jul 6, 2026

Multi-Turn On-Policy Distillation with Prefix Replay

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4

\times

faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.

Baohao Liao, Hanze Dong, Christof Monz +3

1Microsoft Research · University of Amsterdam

LLM Agent Training On-Policy Distillation Policy Learning RL for LLM Agents

Ask PDF

Abstract

On-Policy Distillation

Explore similar work

Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

Stochastic Teacher Intervention for Agentic On-Policy Distillation

Multi-Turn On-Policy Distillation with Prefix Replay