Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.
Figures & tables
Figure 1: Illustration of the four source configurations. The first letter denotes the interaction-history source and the second the current-action-prefix source. In all configurations, the PI-conditioned teacher provides distillation supervision, while gradients update only the student.
Model
ToolQA
OfficeBench
ALFWorld
WebShop
SR
pass@3
Action Count
SR
pass@3
Action Count
SR
pass@3
Action Count
SR
pass@3
Task Score
Action Count
Qwen3-8B
17.2 2.7
20.1
11.3
7.0 0.9
8.1
14.2
37.2 1.7
48.2
30.1
6.6 1.0
9.4
38.1 1.5
11.3
+ TT
81.7 0.9
89.2
6.3
70.9 3.3
76.7
9.1
69.7 2.3
77.4
16.7
70.2 0.7
78.0
78.5 0.8
7.9
+ ST
81.1 1.5
91.3
6.8
74.8 2.1
84.9
8.2
71.7 2.0
79.9
15.1
73.3 0.2
83.2
82.2 1.0
7.1
+ TS
79.8 1.1
87.9
6.4
71.7 2.5
80.2
8.7
69.3 1.5
75.5
15.4
69.1 0.9
79.2
75.0 0.3
7.5
+ SS
74.5 1.7
82.4
7.5
65.9 1.8
73.3
9.6
60.8 1.8
67.9
20.0
58.9 1.1
66.8
73.1 0.9
9.9
Table 1: Stage 1 results with forward KL. Higher SR, pass@3, and Task Score is better. Bold and underlined values mark the best and second-best performance among the four distilled variants of each backbone. Subscripts denote standard error over three trials.
Figure 2: Pairing at fixed marginal on-policiness with Qwen3-14B. Dashed lines indicate standalone ST performance from Stage 1.
Figure 3: Retaining both diagonal configurations improves success rate with Qwen3-14B. Holding ST and TS at 0.25 each, the remaining allocation is assigned to TT only (TT-more), SS only (SS-more), or equally to both (Independent)
Figure 4: Validation success rate across eight training rounds under static and scheduled allocations of the independent mixture. initial denotes the initial student.
Table 3: Training hyperparameters for the controlled study. All settings are shared across source configurations.
Model
Temperature (teacher)
Top- p (teacher)
Temperature (student)
Top- p (student)
Qwen3-8B
0.7
0.95
0.7
0.8
Qwen3-14B
0.7
0.95
0.7
0.8
Ministral-3
0.09
1.0
0.08
1.0
Appendix
Table 4: Decoding parameters for the teacher (data collection) and the student (data collection+evaluation).
Model
ToolQA
OfficeBench
ALFWorld
WebShop
SR
pass@3
Action Count
SR
pass@3
Action Count
SR
pass@3
Action Count
SR
pass@3
Task Score
Action Count
Qwen3-8B
+ TT
↓0.1
↓0.4
↓0.2
↓0.5
↓0.6
↑0.3
↓0.4
↓0.4
↓0.5
↓0.6
↓0.7
↓1.1
↓0.6
+ ST
↓0.1
↓0.2
↑0.4
↓0.2
↓0.4
↑0.2
↑0.2
↓0.2
↓0.1
↑0.5
↓0.3
↓0.3
↑0.4
+ TS
↓0.3
↓0.1
↓0.4
↓0.2
↓0.5
↓0.1
↓0.5
↓0.3
↓0.1
↓0.6
↓0.7
↓0.3
↑1.0
+ SS
↑0.2
↑0.4
↓0.3
↓0.1
↓0.6
↑0.2
↑0.2
↓0.3
↓0.1
↑0.3
↓0.4
↑0.2
↓0.2
Appendix
Table 5: Comparison with results in Table 1 . Arrows denote the value change.
Model
Configuration
Unresolved(%)
Identified(%)
Addressed(%)
Qwen3-8B
ST
85.0
77.1
83.0
TT
19.1
90.2
81.2
Qwen3-14B
ST
82.7
80.7
84.5
TT
14.0
89.3
86.0
Ministral-3
ST
83.3
80.6
82.9
TT
15.7
88.1
82.9
Appendix
Table 6: Corrective supervision in TT and ST training exercises. Unresolved measures the percentage of histories containing an uncorrected earlier error. Identified is conditional on an unresolved error; Addressed is conditional on identification and includes either resolution or an appropriate corrective step.
Model
Configuration
Repeated failure
Premature completion
Rule inconsistency
Operational mistake
Planning mistake
Qwen3-8B
TT
20%
7%
2 %
2 %
11%
ST
9 %
5 %
7%
2 %
7 %
TS
17%
7%
7%
5%
7 %
SS
22%
24%
7%
12%
20%
Qwen3-14B
TT
14%
7%
1 %
2%
9%
ST
7 %
2 %
6%
2%
5 %
Appendix
Table 7: Error profiles of trained students. Error categories are nonexclusive, so a trajectory may receive multiple labels.
Model
Retained student prefix
Executability (%)
Appropriateness (%)
Qwen3-8B
None ( ST )
92.0
80.2
Reasoning
81.0
70.7
Reasoning + half operation
73.2
56.2
Qwen3-14B
None ( ST )
94.7
83.8
Reasoning
85.2
76.0
Reasoning + half operation
76.0
60.2
Appendix
Table 8: Teacher-continuation quality at matched student histories with unresolved errors. The teacher generates either the entire current action or a continuation after the retained student reasoning, optionally followed by the first half of the student’s operation.
Figure 5: Evolution of corrective supervision across training rounds for Qwen3-14B on ALFWorld.
Figure 6: Validation success rate for Qwen3-14B on ToolQA, ALFWorld, and WebShop before training and after rounds 1, 3, 5, and 8.
Figure 7: Recovery evaluation with Qwen3-14B on shared erroneous histories from initial-model failures. We evaluate 100 histories per benchmark. “corrected” measures correction of the existing issue, and “succ.” measures eventual task success.
Figure 8: Regional teacher–student disagreement around the first annotated error in SS and TS training actions. At each selected round, we compute mean per-token KL before, within, and after the error span and normalize the three regional means by their sum.
Figure 9: Pairing at fixed marginal on-policiness with Qwen3-8B. Dashed lines indicate standalone ST performance from Stage 1.
Figure 10: Pairing at fixed marginal on-policiness with Ministra-3. Dashed lines indicate standalone ST performance from Stage 1.
Figure 11: Retaining both diagonal configurations improves success rate with Qwen3-8B. Holding ST and TS at 0.25 each, the remaining allocation is assigned to TT only (TT-more), SS only (SS-more), or equally to both (Independent)
Figure 12: Retaining both diagonal configurations improves success rate with Ministral-3. Holding ST and TS at 0.25 each, the remaining allocation is assigned to TT only (TT-more), SS only (SS-more), or equally to both (Independent)
Figure 13: Comparing the two hybrid directions. ST-more uses weights (0.125,0.5,0.25,0.125) , while TS-more uses (0.125,0.25,0.5,0.125) , ordered as (TT,ST,TS,SS) .
Model
ToolQA
OfficeBench
ALFWorld
WebShop
Qwen3-8B
0.1
0.7
1.9
0.7
Qwen3-14B
0.3
1.4
1.6
0.5
Ministral-3
0.3
0.6
0.7
0.9
Appendix
Table 9: Success-rate gains, in percentage points, from replacing the Independent mixture (0.25,0.25,0.25,0.25) with the hybrid-heavy mixture (0.125,0.375,0.375,0.125) .
Department of Computer Sciences, University of Wisconsin–Madison · Machine Learning Research, Morgan Stanley · Department of Electrical and Computer Engineering, Rutgers University–New Brunswick