Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

Jike Zhong; Ming Li; Yuxiang Lai; Xi Xiao; Sheng Zhang; Ritwick Chaudhry; Tianle Chen; Jingyu Xie

cs.LGOct 18, 2026

Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

Authors: Jike Zhong, Ming Li, Yuxiang Lai, Xi Xiao, Sheng Zhang, Ritwick Chaudhry, Tianle Chen, Jingyu Xie,

Organizations: University of Southern California · University of Florida · Emory University · University of Alabama ·

Abstract

On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD

Figures & tables

Figure 1: Understanding and preserving distillation overshoot. (a) Ordinary OPD students can substantially surpass their teachers mid-training, but then regress as training continues. (b) OPD affects decisions unevenly: repair of the student’s weak decisions initially outpaces erosion of its own successful choices, producing overshoot; continued distillation then erodes those choices, driving regression. (c) SPOD withholds conflicting supervision on successful choices, preserving and extending gains in all settings, effectively curbing regression.

Figure 2: Schematic illustration of the temporary overshoot “window effect” in agentic OPD.

Figure 3: Overshoot-then-regression appears across multiple OPD methods (9 out of 11 runs).

Figure 4: Where the overshoot comes from. At its peak, ordinary OPD substantially outperforms the teacher on tasks the teacher usually fails while retaining strong performance on tasks it usually solves. Subsequent regression is concentrated in the former group. Tasks are grouped by majority teacher outcome on ALFWorld, and

n

denotes cohort size.

Figure 5: Action erosion accompanies post-peak regression. Top: ordinary OPD’s success on Qwen/Gemma ALFWorld. Bottom: As task success declines, the majority of tracked actions lose over half their probability within five updates of the peak. Action halving occurs at or before the corresponding task’s first failure in 96%/92% of lost tasks, linking decision-level erosion to task-level regression.

Qwen3.5-2B $\leftarrow$ Qwen3.5-27B
	ALFWorld							WebShop	ScienceWorld	Overall
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg	Succ.	Succ.	Avg
Teacher 0-Shot	98.3	80.6	74.7	53.0	72.5	95.1	80.0	34.1	42.4	52.2
Student 0-Shot	18.6	55.9	0.0	0.0	0.0	11.4	11.6	7.9	6.1	8.5
Ordinary OPD	99.4	77.4	71.3	67.5	69.6	94.3	80.7	38.0	41.3	53.3
best ckpt. (peak) ∗	97.2	90.3	86.8	73.5	79.0	86.2	86.1	44.0	44.4	58.2

Table 1: Final success rates (%). ALFWorld is reported per task family; WebShop and ScienceWorld overall; Overall averages the three benchmarks. Subscripts indicate SPOD’s gain over ordinary OPD. ∗ Uses evaluation results to choose the checkpoint (oracle). SPOD outperforms all compared methods on every benchmark.

Benchmark	Domain	Task families	Train	Test
ALFWorld	Household tasks	6	3,553	274
WebShop	Online shopping	–	6,410	500
ScienceWorld	Science experiments	6/30	2,880	180

Table 2: Three benchmarks we adopt and evaluate.

Method	Final (%)
Ordinary OPD	80.7
best ckpt. ∗	86.1
Lower learning rate ( $\times 0.3$ )	82.8
Weight averaging (EMA)	80.9
Reverse KL	82.5
Reverse KL, lower learning rate	77.0

Table 3: Alternatives to SPOD. Final success on Qwen ALFWorld. ∗ Selected on evaluation results (oracle).

Method	Final (%)
Guided-OPD	54.8
SPOD	61.7
Guided-OPD + SPOD	66.5

Table 4: Complementary effect of SPOD.

Figure 6: SPOD keeps successful choices the teacher disfavors. Each dot is one such action (Gemma, ALFWorld); bars mark medians. Left: the student learns these actions during training. Right: at the final checkpoints, ordinary OPD suppresses them, while SPOD keeps most above

\tau_{0}

(dotted line).

Figure 7: SPOD preserves and extends the overshoot. On tasks the teacher usually fails (left bars), ordinary OPD drops from its peak to its final checkpoint, while SPOD ends above that peak.

Variant	Mask	Floor	Qwen	Gemma
Ordinary OPD	–	–	80.7	73.7
Floor only	–	$\checkmark$	84.5	72.6
Episode-level mask	episode	–	87.5	86.3
Step-level mask	step	–	87.8	87.0
SPOD (ours)	step	$\checkmark$	91.0	88.9

Table 5: Masking and the floor work best together. Final ALFWorld success (%). Episode-level masking removes teacher supervision from every action in successful episodes; step-level masking removes it only where the teacher assigns the action lower probability than the student. The floor applies to these same teacher-disfavored successful actions.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Constructive overshoot in the two-decision model (Proposition 3.1). With the parameters of Section 3.4, success rises from 9.9% to a peak of 82.0%, above the teacher’s 68.9%, then converges to the teacher’s level while forward-KL loss decreases monotonically. The triangle marks the analytic lower bound on success at

t_{\epsilon}

(74.9%). Time denotes gradient-flow time, not training updates.

	Final success (%)
$\tau_{0}$ ( $\lambda=1$ )	0.3	0.5	0.7
	89.4	91.0	93.8
$\lambda$ ( $\tau_{0}=0.5$ )	0.5	1	2
	84.4	91.0	89.4

Appendix

Table 6: Sensitivity around fixed defaults. Final success on Qwen ALFWorld when varying one hyperparameter at a time. Bold marks the defaults used for all main results. Ordinary OPD reaches 80.7% at its final checkpoint and 86.1% at its best checkpoint.

Objective	Peak (%)	Final (%)
Forward KL (ordinary OPD)	86.1	80.7
Reverse KL	84.7	82.5
Reverse KL, lower learning rate	83.5	77.0
SPOD (forward KL)	92.0	91.0

Appendix

Table 7: Overshoot and regression also occur under reverse KL. Peak and final success on Qwen ALFWorld (teacher: 80.0%).

Benchmark	GRPO	SPOD (ours)
ALFWorld	85.2	91.0 (+5.8)
WebShop	42.8	50.3 (+7.5)
ScienceWorld	52.8	55.9 (+3.1)
Average	60.3	65.7 (+5.5)

Appendix

Table 8: SPOD outperforms tuned GRPO. Final success (%) on Qwen3.5-2B, averaged over three seeds. Parentheses show percentage-point gains over GRPO; the average gain is computed before rounding.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 488 papers

Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Sep 13, 2026Zhiyu Gui, Kexin Huang, Jia Guo +6On-Policy Distillation
University of Science and Technology of China · Ant Group
Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
Aug 3, 2026Chishui Chen, Yaoyou Fan, Te Sun +11LLM Agent Training On-Policy Distillation
Meituan LongCat Interaction · Peking University · Shanghai Jiao Tong University +6
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
May 2, 2026Jianze Wang, Ying Liu, Jinlong Chen +7Multi-Agent Debate Multi-Teacher Knowledge Distillation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Alibaba Group

Sep 13, 2026J/K move · Enter open · S save

Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation

Zhiyu Gui, Kexin Huang, Jia Guo, Junkang Wu+5

University of Science and Technology of China · Ant Group

On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn agentic settings. Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets, despite substantial variation in teacher-signal reliability both within and across trajectories. Our empirical analysis on

τ^2

-bench reveals a clear structure in this variation: informative supervision is concentrated in the prefix of each turn, and, most importantly for multi-turn agentic training, the cross-turn loss of teacher endorsement is temporally locked to the student's first erroneous action rather than accumulating gradually over turns. Building on these findings, we propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques: adaptive early stopping, which terminates a rollout once the cumulative teacher log-probability falls below an out-of-distribution threshold, and a prefix buffer, which caches high-quality prefixes and restarts generation at the weakest correct turn. Together, these mechanisms induce a data-driven curriculum that progressively extends coverage to later turns. On

τ^2

-bench retail, our method matches full-trajectory OPD and exceeds the 30B teacher at a

3.73\times

speedup, surpasses the baseline itself at

2.34\times

, and retains a

4.51\times

speedup under cross-domain multi-teacher training. As a supplementary generalization test beyond the agentic setting, STRIDE outperforms full OPD on AIME 2025 at a

5.10\times

speedup and on AIME 2024 at a

3.08\times

speedup; averaged across the two evaluations, both fixed-budget truncation baselines remain below full OPD.

Abstract

On-Policy Distillation

Explore similar work

Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation

MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate