On-policy distillation (OPD) enhances the capabilities of small language agents by providing dense, token-level supervision from a teacher model on student-generated trajectories. However, standard OPD implicitly assumes that credit is uniformly distributed across long-horizon trajectories, assigning the same strength of teacher supervision to different actions. Through credit-distribution analysis and outcome-reversal counterfactual experiments, we find that trajectory credit is concentrated on a small number of critical actions. Moreover, even when the trajectory is held fixed, reversing only the final outcome substantially changes the credit assigned to multiple interaction turns. These findings show that the uniform-credit assumption fails to identify critical decisions and ultimately degrades the model's reasoning accuracy. To address these challenges, we propose ToolOPD, an outcome-aware on-policy distillation framework designed for long-horizon tool use. ToolOPD first summarizes each completed trajectory and its outcome into a reusable hindsight skill. It then measures how this skill changes the teacher's probability of each student action token and converts the change into outcome-aware credit. The resulting credit preserves supervision that supports successful decisions or discourages failed ones, while suppressing irrelevant and conflicting signals. ToolOPD uses these credits to reweight the action-level distillation loss and jointly optimizes it with reinforcement learning. Extensive experiments across multiple long-horizon tool interaction benchmarks demonstrate that ToolOPD significantly outperforms existing reinforcement learning and self-distillation baselines.
Figures & tables
Figure 1: Empirical Study Results.
Figure 2: ToolOPD framework overview.
Method
Pick
Look
Clean
Heat
Cool
Pick2
Scc.
GRPO
17.1
15.4
29.6
18.8
16.0
8.3
14.8
GRPO + Vanilla OPD
55.6
66.7
62.5
30.7
92.9
25.0
53.9
GRPO + Skill-OPD
82.4
50.0
46.2
19.1
60.0
52.6
54.7
ToolOPD (Ours)
65.1
75.0
66.7
33.3
79.0
22.2
59.4
Table 3: Ablation study on the ALFWorld benchmark. We progressively add components to GRPO: standard action-level distillation (Vanilla OPD), hindsight skill conditioning (Skill-OPD), and the full ToolOPD. The best results in each column are highlighted in bold.
Figure 3: Training dynamics across different tasks.
Figure 4: Training dynamics of the model across ALFWorld (c) and FTRL (a, b, and d).
Figure 5: Hyperparameter sensitivity analysis.
Figure 6: Further analysis of Tool-OPD. (Left) Performance under different credit-assignment ablation settings. (Right) Quality of training signals with different top- k selections.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
ALFWorld
WebShop
SearchQA
Training steps
150
150
150
Training batch size
16
16
128
Rollout group size G
8
8
8
Maximum prompt length
2048
4096
4096
Maximum response length
512
512
512
Maximum interaction turns
50
15
4
Appendix
Table 4: Core ToolOPD training configurations across the three tool-use benchmarks.
Hyperparameter
Value
Student model
Qwen3-0.6B / Qwen3-1.7B
Teacher model
GRPO-trained Qwen3-4B
Policy optimization algorithm
Vanilla GRPO
Group size G (rollouts per task)
8
Training batch size
32
PPO mini-batch size
32
Appendix
Table 5: Core training hyperparameters for ToolOPD on the tool-use benchmarks.
Figure 7: Computational overhead analysis on FTRL.
ALFWorld
SearchQA
WebShop
Abbr.
Full name
Abbr.
Full name
Abbr.
Full name
Pick
Pick and Place
NQ
Natural Questions
Score
Average Score
Look
Look at Object in Light
Triv
TriviaQA
Succ.
Success Rate
Clean
Pick, Clean, and Place
Pop
PopQA
Heat
Pick, Heat, and Place
Hotp
HotpotQA
Cool
Pick, Cool, and Place
2Wk
2WikiMultiHopQA
Appendix
Table 6: Task and metric abbreviations used in Table 1 .
FTRL
BFCL
ToolHop
Abbr.
Full name
Abbr.
Full name
Abbr.
Full name
Solve-F1
Solve F1 Score
Base
Multi-Turn Base
Acc.
Accuracy
Solve-R
Solve Recall
MF
Multi-Turn Missing Function
Solve-P
Solve Precision
MP
Multi-Turn Missing Parameter
LC
Multi-Turn Long Context
MTOA
Multi-Turn Overall Accuracy
Appendix
Table 7: Metric abbreviations used in the FTRL, BFCL, and ToolHop evaluation.
Long-horizon tool-use reinforcement learning learns from outcome verification, but trajectory-level advantages are broadcast over reasoning, API, and answer tokens. Direct self-distillation can supply a denser signal, but in our experiments it can also destroy tool use by rehearsing teacher behavior without identifying which actions the verifier rewards. We introduce Sibling-Guided Credit Distillation (SGCD), which uses distillation for bounded credit weighting rather than as a competing actor loss. Dynamic sampling produces mixed successful and failed sibling rollouts; an external LLM summarizes their contrast into a training-only credit reference; and detached teacher/student divergence reshapes GRPO token advantages. The deployed student receives only the clean task prompt. Across AppWorld and tau^3-airline, SGCD reports higher held-out point estimates than GRPO-family comparators: AppWorld TGC improves from 42.9 to 45.6 on test_normal and from 24.7 to 27.0 on test_challenge, and tau^3-airline held-out evaluator score improves from 0.583 to 0.602. These results support a narrow design rule for long-horizon tool-use agents: use distillation to guide credit assignment while keeping policy gradient in charge of the actor update.
Tianyu Ding, Jianhong Xin, Juan Pablo De la Cruz Weinstein
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.