On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn agentic settings. Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets, despite substantial variation in teacher-signal reliability both within and across trajectories. Our empirical analysis on τ2-bench reveals a clear structure in this variation: informative supervision is concentrated in the prefix of each turn, and, most importantly for multi-turn agentic training, the cross-turn loss of teacher endorsement is temporally locked to the student's first erroneous action rather than accumulating gradually over turns. Building on these findings, we propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques: adaptive early stopping, which terminates a rollout once the cumulative teacher log-probability falls below an out-of-distribution threshold, and a prefix buffer, which caches high-quality prefixes and restarts generation at the weakest correct turn. Together, these mechanisms induce a data-driven curriculum that progressively extends coverage to later turns. On τ2-bench retail, our method matches full-trajectory OPD and exceeds the 30B teacher at a 3.73× speedup, surpasses the baseline itself at 2.34×, and retains a 4.51× speedup under cross-domain multi-teacher training. As a supplementary generalization test beyond the agentic setting, STRIDE outperforms full OPD on AIME 2025 at a 5.10× speedup and on AIME 2024 at a 3.08× speedup; averaged across the two evaluations, both fixed-budget truncation baselines remain below full OPD.
Figures & tables
Figure 1: Overview of STRIDE. (a) Where the teacher signal remains reliable in agentic OPD (Observations 1–3, § 3.2 ). (b) STRIDE on one task across training epochs: adaptive early stopping terminates the rollout once the cumulative teacher log-probability crosses λ , and the prefix buffer caches the correct prefix and restarts the next rollout from the weakest correct turn k∗ , which shifts rightward as the student improves (per-turn values are illustrative). (c) On τ2 -bench retail, STRIDE matches full-trajectory OPD and exceeds the 30B teacher at a 3.73× speedup.
Figure 2: (a) Per-token student–teacher KL versus the relative position within a turn (blue) and versus the turn index (red) on τ2 -bench retail (student Qwen3-4B, teacher Qwen3-30B-A3B-Thinking-2507). (b) Residualized teacher log-probability (solid) and student–teacher KL (dashed) versus the offset from the first-error turn t∗ ; shaded bands indicate ±1 standard error.
Figure 3: Joint training framework of early stopping and prefix buffer.
Method
s/step
Speedup
mean@16
pass@16
Reference
Teacher (30B)
—
—
0.461
0.847
Baselines
Full OPD
273.8
1.00 ×
0.477
0.850
Fast OPD Zhang et al. (2026)
101.4
2.70 ×
0.434
0.800
TCOD-F2B Wang et al. (2026)
135.9
2.01 ×
0.437
0.822
Table 1: Single-teacher agentic OPD results ( τ2 -bench retail). Bold marks the best result in each column.
Figure 4: Wall-clock efficiency: mean@16 vs. cumulative training hours.
Method
s/step
Speedup
Retail
Telecom
Reference: expert teachers
Retail
—
—
0.649
0.384
Telecom
—
—
0.622
0.960
Reference: single-domain full OPD
Retail
157.6
—
0.586
—
Telecom
423.1
—
—
0.816
Table 2: Multi-teacher agentic OPD results ( τ2 -bench retail + telecom, mean@4). Bold marks the best non-reference result in each domain column. Italic rows are references.
Figure 5: Multi-teacher agentic OPD: retail ( a ) and telecom ( b ) mean@4 vs. training step. Solid lines use the prefix buffer ( α=−0.8 ); dashed lines are the early-stopping-only variants. The horizontal lines mark the single-domain baselines (gray dashed) and the domain expert teachers (purple dotted) of the respective domain.
Method
s/step
Speedup
Retail
Telecom
Reference: expert teachers
Retail
—
—
0.649
0.384
Telecom
—
—
0.622
0.960
Baselines
Full OPD
221.5
1.00 ×
0.625
0.882
Fast OPD Zhang et al. (2026)
60.5
3.66 ×
0.504
0.857
Table 3: Multi-teacher agentic OPD results with a Qwen3-8B student ( τ2 -bench retail + telecom, mean@4). Bold marks the best non-reference result in each domain column. Italic rows are references.
Figure 6: Single-teacher λ ablation: mean@16 vs. training step without ( a ) and with ( b ) the prefix buffer.
Figure 7: α ablation at λ=−2 on retail ( τ2 -bench), prefix reuse enabled, first 100 training steps.
Method
s/step
Speedup
AIME25
AIME24
Baselines
Full OPD
220.5
1.00 ×
0.458
0.546
Fast OPD 2048 Zhang et al. (2026)
22.4
9.83 ×
0.408
0.554
Fast OPD 4096 Zhang et al. (2026)
36.4
6.05 ×
0.429
0.542
STRIDE w/o prefix buffer (ours)
λ=−1000
41.6
5.30 ×
0.429
0.542
Table 4: Math OPD results (AIME 2024/2025, mean@8). Bold marks the best result in each column.
Figure 8: Single-teacher agentic OPD: mean@16 vs. training step for full OPD, STRIDE ( λ=−1 with the prefix buffer), and the curriculum baselines.
On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD
Jike Zhong, Ming Li, Yuxiang Lai +9
University of Southern California · University of Florida · Emory University +6
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
On-policy distillation (OPD) has shown strong potential for transferring reasoning ability from frontier or domain-specific models to smaller students. While effective on static single-turn tasks, its behavior in multi-turn agent settings remains underexplored. In this work, we identify a key limitation of vanilla OPD in such settings, which we term Trajectory-Level KL Instability. Specifically, we observe that KL divergence increases together with a drop in success rate, and even after convergence, the KL remains high, leading to unstable training. This instability arises from inter-turn error compounding: as errors accumulate, the student is driven beyond the teacher's effective support, rendering the supervision signal unreliable. To address this, we propose TCOD (Temporal Curriculum On-Policy Distillation), a simple yet effective framework that controls the trajectory depth exposed to the student and progressively expands it from short to long with a curriculum schedule. Experimental results across four student-teacher pairs on three multi-turn agent benchmarks (ALFWorld, WebShop, ScienceWorld) show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD. Further evaluations show that TCOD can even surpass the teacher's performance and generalize to tasks on which the teacher fails. Our code is available at https://github.com/kokolerk/TCOD.
Jiaqi Wang, Wenhao Zhang, Weijie Shi +2
Tongyi Lab , Alibaba Group · † The Chinese University of Hong Kong.