LLM agent distillation transfers agentic capabilities to student models by training on complete teacher trajectories that typically interleave thoughts and actions. We question how critical thought supervision is for agents' ultimate action performance. To answer this, we disentangle their impacts by independently perturbing teacher thoughts and actions during distillation. On widely adopted agent benchmarks across model scales, altering or discarding thoughts has little effect on student performance. We find, however, that thought supervision becomes important when information about the thought is not disclosed in the corresponding action during training. By identifying such discrepant cases, our findings challenge the common practice of distilling entire teacher trajectories for LLM agents, and offer a new insight that thought supervision can be a discernible choice for efficiency depending on whether the information of thought is disclosed in the action.
Figures & tables
Figure 1: Two students distilled from the same teacher trajectories on HotpotQA ( Yang et al., 2018 ) , differing only in whether teacher thoughts are included in the supervision during training.
Figure 2: Retention under thought and action corruption at p=1.0 (Qwen2.5-3B-Instruct student). Error bars denote standard deviation across three runs. Action drop and action replace yield 0% retention on WebShop and ALFWorld.
Figure 3: Retention under thought corruption across four benchmarks at p=1.0 . (a) Results across student model sizes. (b) Results across teacher models. (c) Results across student model families.
Figure 4: Retention under thought drop across corruption ratios p , comparing span-level and token-level corruption under SFT and span-level corruption under the KL objective (Qwen2.5-3B-Instruct student).
Figure 5: Out-of-distribution robustness under thought interventions ( p=1.0 ), with HotpotQA as in-distribution.
Figure 6: Examples illustrating whether task-relevant information of the thought is disclosed in the action during training. (a) Parts of thought information are disclosed in the action; (b) Thought is not disclosed in the action.
Figure 7: Retention under thought drop and replace across student sizes for high- and low-disclosure settings ( p=1.0) .
Figure 8: Example of the disclosure intervention applied to the MathQA instance in Figure 6 (b). Anonymous variables in the action are renamed with semantic names, exposing information that was previously available only in the thought.
Figure 9Figure 10
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Teacher Trajectories
Training Data
Eval Set Size
HotpotQA
1,000
597
500
FEVER
1,000
650
500
ALFWorld
3,553
2,872
274
WebShop
10,587
3,555
500
FinQA
5,695
1,500
300
MultiHiertt
6,217
1,500
832
Appendix
Table 2: Dataset statistics for agent distillation experiments.
Thought
Action
Benchmark
Size
Clean
Drop
Replace
Paraphrase
Drop
Replace
HotpotQA
0.5B
33.4 (0.9)
34.2 (0.6)
31.0 (1.5)
31.0 (1.2)
0 7.2 (1.6)
10.0 (0.5)
3B
48.0 (0.7)
46.8 (0.3)
45.6 (1.5)
46.8 (1.3)
25.5 (1.4)
0 5.0 (1.0)
7B
50.8 (1.1)
54.0 (0.6)
48.0 (1.7)
52.2 (1.6)
29.0 (1.8)
0 4.2 (2.1)
14B
56.4 (1.2)
58.6 (0.9)
52.4 (1.8)
55.0 (1.2)
0 2.2 (1.3)
0 5.2 (1.7)
FEVER
0.5B
51.2 (0.8)
48.8 (0.8)
52.4 (1.2)
50.6 (0.5)
40.0 (1.2)
44.6 (1.2)
Appendix
Table 3: Task accuracy under thought and action corruption across Qwen2.5-{0.5B, 3B, 7B, 14B}-Instruct students distilled from Qwen2.5-32B-Instruct ( p=1.0 ). Results are averaged over three independent training and evaluation runs, with standard deviations reported in parentheses.
Figure 17: Retention under thought corruption (top) and action corruption (bottom) at varying corruption ratios p (Qwen2.5-3B-Instruct student). Corruption is applied at the span level: each thought ti or action ai is dropped, replaced, or paraphrased as a whole with probability p . Paraphrase is defined for thoughts only.
Benchmark
Intervention
Qwen2.5-32B-Instruct
GPT-5.4
Claude-Opus-5
Gemini-3.1-Pro
HotpotQA
Clean
48.0 (0.7)
52.6 (0.9)
41.0 (0.6)
42.8 (0.9)
Drop
46.8 (0.3)
50.0 (0.6)
39.2 (0.4)
41.6 (0.6)
Replace
45.6 (1.5)
48.8 (1.5)
38.4 (1.4)
41.6 (1.1)
Paraphrase
46.8 (1.3)
52.4 (1.4)
41.2 (0.7)
46.8 (0.9)
FEVER
Clean
63.4 (1.2)
61.0 (1.0)
59.2 (0.8)
59.0 (1.2)
Drop
64.9 (0.8)
57.4 (0.9)
57.2 (0.7)
57.4 (0.9)
Appendix
Table 4: Task accuracy under thought interventions across teacher models, using a Qwen2.5-3B-Instruct student ( p=1.0 ).
Benchmark
Intervention
Llama-3.2-3B-Instruct
Qwen2.5-3B-Instruct
Phi-4-mini-instruct
HotpotQA
Clean
51.2 (0.7)
48.0 (0.7)
45.4 (1.3)
Drop
49.4 (0.7)
46.8 (0.3)
42.4 (1.0)
Replace
47.6 (1.6)
45.6 (1.5)
42.4 (2.0)
Paraphrase
51.6 (1.0)
46.8 (1.3)
47.2 (1.3)
FEVER
Clean
67.6 (0.8)
63.4 (1.2)
64.8 (1.0)
Drop
66.0 (1.1)
64.9 (0.8)
65.4 (1.5)
Appendix
Table 5: Task accuracy under thought interventions across student model families, using a fixed Qwen2.5-32B-Instruct teacher ( p=1.0 ).
Drop
Replace
Paraphrase
Benchmark
p
Span
Token
Span
Token
Span
Token
HotpotQA
0 (clean)
48.0
0.25
48.2
47.8
47.6
48.4
48.2
47.0
0.50
47.6
47.6
47.2
46.0
49.0
47.4
0.75
45.6
44.7
45.4
45.0
49.8
50.6
1.00
46.8
46.4
45.6
44.8
46.8
48.0
Appendix
Table 6: Task accuracy under span-level and token-level thought corruption at varying corruption ratios p (Qwen2.5-3B-Instruct student, Qwen2.5-32B-Instruct teacher). p=0 corresponds to the unperturbed trajectories.
Objective
p
HotpotQA
FEVER
WebShop
ALFWorld
SFT
0 (clean)
48.0
63.4
40.2
88.3
0.25
48.2
65.8
41.6
93.3
0.50
47.6
63.6
44.2
95.7
0.75
45.6
63.6
40.0
96.0
1.00
46.8
64.9
43.1
95.1
KL
0 (clean)
43.8
61.6
39.0
89.1
Appendix
Table 7: Accuracy under SFT and KL distillation objectives as a function of the span-level thought-drop ratio p (Qwen2.5-3B-Instruct student, Qwen2.5-32B-Instruct teacher). p=0 corresponds to the unperturbed trajectories.
Evaluation Dataset
Clean
Thought Drop
Thought Replace
Thought Paraphrase
HotpotQA (In-distribution)
56.4
58.6
52.4
55.0
FEVER (Out-of-distribution)
63.2
65.2
65.8
62.8
2WikiMultiHopQA (Out-of-distribution)
49.4
50.0
47.4
46.8
Bamboogle (Out-of-distribution)
50.4
52.0
47.2
50.4
MuSiQue (Out-of-distribution)
21.2
21.6
19.8
23.8
Appendix
Table 8: In-distribution and out-of-distribution performance of a Qwen2.5-14B-Instruct student distilled on HotpotQA under thought drop, replacement, and paraphrase ( p=1.0 ).
Thought
Benchmark
Size
Clean
Drop
Replace
FinQA
0.5B
52.7 (0.6)
0 3.0 (0.3)
0 3.0 (0.3)
3B
63.3 (0.7)
47.7 (1.5)
49.0 (1.9)
7B
68.0 (0.6)
54.4 (1.2)
52.9 (1.7)
14B
72.7 (0.7)
56.2 (1.6)
56.6 (1.3)
MultiHiertt
0.5B
32.3 (1.2)
23.8 (1.3)
21.6 (1.8)
Appendix
Table 9: Accuracy under clean training, thought drop, and thought replacement across Qwen2.5-{0.5B, 3B, 7B, 14B}-Instruct students on low-disclosure settings, using a Qwen2.5-32B-Instruct teacher ( p=1.0 ).
Training
Inference
Benchmark
Full (s)
Action-only (s)
Reduction (%)
Full (s)
Action-only (s)
Reduction (%)
HotpotQA
441.3 (11.8)
410.0 (1.0)
7.1
687.8 (5.2)
492.6 (1.2)
28.4
FEVER
377.3 (4.2)
353.7 (2.3)
6.3
361.7 (1.5)
321.7 (2.1)
11.1
ALFWorld
2744.0 (21.3)
2463.7 (21.4)
10.2
891.0 (34.7)
519.3 (5.7)
41.7
WebShop
1890.3 (15.9)
1791.0 (8.5)
5.3
608.3 (13.0)
512.0 (19.0)
15.8
Appendix
Table 10: Wall-clock training and inference times under full-trajectory and action-only distillation. Absolute times are reported in seconds, and reductions are relative to full-trajectory distillation.
Benchmark
Action representation
Clean
Drop
Δ
Retention (%)
FinQA
Anonymous (= before disclosing)
63.3
47.7
− 15.6
75.4
Semantic (= after disclosing)
65.3
61.8
− 3.5
94.6
Mismatched
45.3
36.5
− 8.8
80.6
MultiHiertt
Anonymous (= before disclosing)
49.2
39.5
− 9.7
80.3
Semantic (= after disclosing)
50.6
44.6
− 6.0
88.1
Mismatched
33.8
26.3
− 7.5
77.8
Appendix
Table 11: Thought-drop retention under the anonymous, semantic, and mismatched-name action representations in the low-disclosure settings, using a Qwen2.5-3B-Instruct student. Δ denotes the absolute accuracy change from Clean to Drop (percentage points).
Benchmark
Full + Thought
Full + Direct
Action-only
FinQA
63.3 (0.7)
57.1 (1.0)
47.7 (1.5)
MultiHiertt
49.2 (0.6)
43.5 (0.9)
39.5 (1.1)
MathQA
17.1 (0.9)
15.0 (1.2)
10.7 (0.8)
GeoQA
38.8 (0.6)
32.0 (0.7)
20.1 (0.5)
Appendix
Table 12: Accuracy of full-trajectory-trained students, with and without inference-time thought generation, and action-only students in the low-disclosure settings. Full + Thought uses the standard thought–action inference format, whereas Full + Direct suppresses thought generation through the prompt and directly predicts the action. Action-only denotes students trained without thoughts. All results use Qwen2.5-3B-Instruct students with the anonymous action representation.
Student
Clean
Thought Drop
Retention (%)
Qwen2.5-0.5B-Instruct
0.525
0.550
104.8
Qwen2.5-3B-Instruct
0.588
0.595
101.2
Qwen2.5-7B-Instruct
0.563
0.550
97.7
Qwen2.5-14B-Instruct
0.485
0.462
95.3
Appendix
Table 13: Robustness to thought removal on InterCode, a low-level action environment.
Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage via on-policy distillation (OPD). OPD preserves student-generated trajectories and provides dense token-level teacher targets at student-visited prefixes. Privileged reasoning is used during distillation rather than student inference. Uniform ThinkOPD, a natural think-enabled OPD baseline, conditions a fixed teacher on one shared think trace and uniformly distills every sibling student response. Although its prefixes are on-policy, the trace need not follow a route compatible with every complete response: the same privileged trace can induce different teacher-student discrepancies even when responses reach the same outcome. We summarize this interaction with trace-response divergence (TRD) and introduce ThinkOPD, which routes supervision at the response level by combining group-relative reward gain with a TRD-based compatibility proxy. Final response weights are normalized within each rollout group. Across mathematical reasoning and code generation, ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison. Controlled interventions show that outcome benefit and the TRD-based proxy provide complementary routing signals in this setting. Think-enabled OPD provides a controlled setting for studying how teacher advantage becomes transferable along student responses.
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On τ2-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.
Deploying LLM agents typically requires a compact test-time student, even if a stronger teacher is available during training. We study knowledge brewing: distilling a teacher's interactive experience into a persistent external memory for the student. Crucially, this requires no weight updates, expert demonstrations, ground-truth labels, or test-time teacher access. This setting poses two challenges: environments provide only sparse, binary feedback, and teacher-authored notes must be inherently tailored to be concretely executable by a substantially weaker student. To address these hurdles, we propose AgentBrew, comprising two coupled components. First, a failure-triggered teacher--Ralph Loop mitigates sparse feedback by transforming student failures into environment-validated notes. Second, student-aware synthesis calibrates teacher knowledge to the weak executor's operational granularity, yielding model-specific, actionable guidance. Extensive evaluations and comprehensive ablations across coding, math, and tool-use tasks demonstrate that this asymmetric, training-free brewing paradigm produces highly capable yet deployable LLM agents.