Agents, Think Only When it Helps

Min-Seon Kim; Seungdong Yoa; Sanghyu Yoon; Ye Seul Sim; Suhee Yoon; Dongmin Kim; Jung-Eun Kim

cs.AINov 3, 2026

Agents, Think Only When it Helps

Authors: Min-Seon Kim, Seungdong Yoa, Sanghyu Yoon, Ye Seul Sim, Suhee Yoon, Dongmin Kim, Jung-Eun Kim

Organizations: Computer Science, North Carolina State University · LG AI Research

Abstract

LLM agent distillation transfers agentic capabilities to student models by training on complete teacher trajectories that typically interleave thoughts and actions. We question how critical thought supervision is for agents' ultimate action performance. To answer this, we disentangle their impacts by independently perturbing teacher thoughts and actions during distillation. On widely adopted agent benchmarks across model scales, altering or discarding thoughts has little effect on student performance. We find, however, that thought supervision becomes important when information about the thought is not disclosed in the corresponding action during training. By identifying such discrepant cases, our findings challenge the common practice of distilling entire teacher trajectories for LLM agents, and offer a new insight that thought supervision can be a discernible choice for efficiency depending on whether the information of thought is disclosed in the action.

Figures & tables

Panel of Figure 1 — Figure 1: Two students distilled from the same teacher trajectories on HotpotQA ( Yang et al., 2018 ) , differing only in whether teacher thoughts are included in the supervision during training.

Panel of Figure 2 — Figure 2: Retention under thought and action corruption at $p{=}1.0$ (Qwen2.5-3B-Instruct student). Error bars denote standard deviation across three runs. Action drop and action replace yield 0% retention on WebShop and ALFWorld.

Panel of Figure 3 — Figure 3: Retention under thought corruption across four benchmarks at $p=1.0$ . (a) Results across student model sizes. (b) Results across teacher models. (c) Results across student model families.

Panel of Figure 4 — Figure 4: Retention under thought drop across corruption ratios $p$ , comparing span-level and token-level corruption under SFT and span-level corruption under the KL objective (Qwen2.5-3B-Instruct student).

Panel of Figure 5 — Figure 5: Out-of-distribution robustness under thought interventions ( $p{=}1.0$ ), with HotpotQA as in-distribution.

Panel of Figure 6 — Figure 6: Examples illustrating whether task-relevant information of the thought is disclosed in the action during training. (a) Parts of thought information are disclosed in the action; (b) Thought is not disclosed in the action.

Panel of Figure 7 — Figure 7: Retention under thought drop and replace across student sizes for high- and low-disclosure settings ( $p{=}1.0)$ .

Panel of Figure 8 — Figure 8: Example of the disclosure intervention applied to the MathQA instance in Figure 6 (b). Anonymous variables in the action are renamed with semantic names, exposing information that was previously available only in the thought.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Teacher Trajectories	Training Data	Eval Set Size
HotpotQA	1,000	597	500
FEVER	1,000	650	500
ALFWorld	3,553	2,872	274
WebShop	10,587	3,555	500
FinQA	5,695	1,500	300
MultiHiertt	6,217	1,500	832

Appendix

Table 2: Dataset statistics for agent distillation experiments.

			Thought			Action
Benchmark	Size	Clean	Drop	Replace	Paraphrase	Drop	Replace
HotpotQA	0.5B	33.4 (0.9)	34.2 (0.6)	31.0 (1.5)	31.0 (1.2)	0 7.2 (1.6)	10.0 (0.5)
	3B	48.0 (0.7)	46.8 (0.3)	45.6 (1.5)	46.8 (1.3)	25.5 (1.4)	0 5.0 (1.0)
	7B	50.8 (1.1)	54.0 (0.6)	48.0 (1.7)	52.2 (1.6)	29.0 (1.8)	0 4.2 (2.1)
	14B	56.4 (1.2)	58.6 (0.9)	52.4 (1.8)	55.0 (1.2)	0 2.2 (1.3)	0 5.2 (1.7)
FEVER	0.5B	51.2 (0.8)	48.8 (0.8)	52.4 (1.2)	50.6 (0.5)	40.0 (1.2)	44.6 (1.2)

Appendix

Table 3: Task accuracy under thought and action corruption across Qwen2.5-{0.5B, 3B, 7B, 14B}-Instruct students distilled from Qwen2.5-32B-Instruct (

p{=}1.0

). Results are averaged over three independent training and evaluation runs, with standard deviations reported in parentheses.

Panel of Figure 17 — Figure 17: Retention under thought corruption (top) and action corruption (bottom) at varying corruption ratios $p$ (Qwen2.5-3B-Instruct student). Corruption is applied at the span level: each thought $t_{i}$ or action $a_{i}$ is dropped, replaced, or paraphrased as a whole with probability $p$ . Paraphrase is defined for thoughts only.

Benchmark	Intervention	Qwen2.5-32B-Instruct	GPT-5.4	Claude-Opus-5	Gemini-3.1-Pro
HotpotQA	Clean	48.0 (0.7)	52.6 (0.9)	41.0 (0.6)	42.8 (0.9)
	Drop	46.8 (0.3)	50.0 (0.6)	39.2 (0.4)	41.6 (0.6)
	Replace	45.6 (1.5)	48.8 (1.5)	38.4 (1.4)	41.6 (1.1)
	Paraphrase	46.8 (1.3)	52.4 (1.4)	41.2 (0.7)	46.8 (0.9)
FEVER	Clean	63.4 (1.2)	61.0 (1.0)	59.2 (0.8)	59.0 (1.2)
	Drop	64.9 (0.8)	57.4 (0.9)	57.2 (0.7)	57.4 (0.9)

Appendix

Table 4: Task accuracy under thought interventions across teacher models, using a Qwen2.5-3B-Instruct student (

p{=}1.0

Benchmark	Intervention	Llama-3.2-3B-Instruct	Qwen2.5-3B-Instruct	Phi-4-mini-instruct
HotpotQA	Clean	51.2 (0.7)	48.0 (0.7)	45.4 (1.3)
	Drop	49.4 (0.7)	46.8 (0.3)	42.4 (1.0)
	Replace	47.6 (1.6)	45.6 (1.5)	42.4 (2.0)
	Paraphrase	51.6 (1.0)	46.8 (1.3)	47.2 (1.3)
FEVER	Clean	67.6 (0.8)	63.4 (1.2)	64.8 (1.0)
	Drop	66.0 (1.1)	64.9 (0.8)	65.4 (1.5)

Appendix

Table 5: Task accuracy under thought interventions across student model families, using a fixed Qwen2.5-32B-Instruct teacher (

p{=}1.0

		Drop		Replace		Paraphrase
Benchmark	$p$	Span	Token	Span	Token	Span	Token
HotpotQA	0 (clean)	48.0
	0.25	48.2	47.8	47.6	48.4	48.2	47.0
	0.50	47.6	47.6	47.2	46.0	49.0	47.4
	0.75	45.6	44.7	45.4	45.0	49.8	50.6
	1.00	46.8	46.4	45.6	44.8	46.8	48.0

Appendix

Table 6: Task accuracy under span-level and token-level thought corruption at varying corruption ratios

p

(Qwen2.5-3B-Instruct student, Qwen2.5-32B-Instruct teacher).

p=0

corresponds to the unperturbed trajectories.

Objective	$p$	HotpotQA	FEVER	WebShop	ALFWorld
SFT	0 (clean)	48.0	63.4	40.2	88.3
	0.25	48.2	65.8	41.6	93.3
	0.50	47.6	63.6	44.2	95.7
	0.75	45.6	63.6	40.0	96.0
	1.00	46.8	64.9	43.1	95.1
KL	0 (clean)	43.8	61.6	39.0	89.1

Appendix

Table 7: Accuracy under SFT and KL distillation objectives as a function of the span-level thought-drop ratio

p

(Qwen2.5-3B-Instruct student, Qwen2.5-32B-Instruct teacher).

p{=}0

corresponds to the unperturbed trajectories.

Evaluation Dataset	Clean	Thought Drop	Thought Replace	Thought Paraphrase
HotpotQA (In-distribution)	56.4	58.6	52.4	55.0
FEVER (Out-of-distribution)	63.2	65.2	65.8	62.8
2WikiMultiHopQA (Out-of-distribution)	49.4	50.0	47.4	46.8
Bamboogle (Out-of-distribution)	50.4	52.0	47.2	50.4
MuSiQue (Out-of-distribution)	21.2	21.6	19.8	23.8

Appendix

Table 8: In-distribution and out-of-distribution performance of a Qwen2.5-14B-Instruct student distilled on HotpotQA under thought drop, replacement, and paraphrase (

p{=}1.0

			Thought
Benchmark	Size	Clean	Drop	Replace
FinQA	0.5B	52.7 (0.6)	0 3.0 (0.3)	0 3.0 (0.3)
	3B	63.3 (0.7)	47.7 (1.5)	49.0 (1.9)
	7B	68.0 (0.6)	54.4 (1.2)	52.9 (1.7)
	14B	72.7 (0.7)	56.2 (1.6)	56.6 (1.3)
MultiHiertt	0.5B	32.3 (1.2)	23.8 (1.3)	21.6 (1.8)

Appendix

Table 9: Accuracy under clean training, thought drop, and thought replacement across Qwen2.5-{0.5B, 3B, 7B, 14B}-Instruct students on low-disclosure settings, using a Qwen2.5-32B-Instruct teacher (

p{=}1.0

Benchmark	Full (s)	Action-only (s)	Reduction (%)	Full (s)	Action-only (s)	Reduction (%)
	Training			Inference
HotpotQA	$441.3$ (11.8)	$410.0$ (1.0)	7.1	$687.8$ (5.2)	$492.6$ (1.2)	28.4
FEVER	$377.3$ (4.2)	$353.7$ (2.3)	6.3	$361.7$ (1.5)	$321.7$ (2.1)	11.1
ALFWorld	$2744.0$ (21.3)	$2463.7$ (21.4)	10.2	$891.0$ (34.7)	$519.3$ (5.7)	41.7
WebShop	$1890.3$ (15.9)	$1791.0$ (8.5)	5.3	$608.3$ (13.0)	$512.0$ (19.0)	15.8

Appendix

Table 10: Wall-clock training and inference times under full-trajectory and action-only distillation. Absolute times are reported in seconds, and reductions are relative to full-trajectory distillation.

Benchmark	Action representation	Clean	Drop	$\Delta$	Retention (%)
FinQA	Anonymous (= before disclosing)	63.3	47.7	$-$ 15.6	75.4
	Semantic (= after disclosing)	65.3	61.8	$-$ 3.5	94.6
	Mismatched	45.3	36.5	$-$ 8.8	80.6
MultiHiertt	Anonymous (= before disclosing)	49.2	39.5	$-$ 9.7	80.3
	Semantic (= after disclosing)	50.6	44.6	$-$ 6.0	88.1
	Mismatched	33.8	26.3	$-$ 7.5	77.8

Appendix

Table 11: Thought-drop retention under the anonymous, semantic, and mismatched-name action representations in the low-disclosure settings, using a Qwen2.5-3B-Instruct student.

\Delta

denotes the absolute accuracy change from Clean to Drop (percentage points).

Benchmark	Full + Thought	Full + Direct	Action-only
FinQA	63.3 (0.7)	57.1 (1.0)	47.7 (1.5)
MultiHiertt	49.2 (0.6)	43.5 (0.9)	39.5 (1.1)
MathQA	17.1 (0.9)	15.0 (1.2)	10.7 (0.8)
GeoQA	38.8 (0.6)	32.0 (0.7)	20.1 (0.5)

Appendix

Table 12: Accuracy of full-trajectory-trained students, with and without inference-time thought generation, and action-only students in the low-disclosure settings. Full + Thought uses the standard thought–action inference format, whereas Full + Direct suppresses thought generation through the prompt and directly predicts the action. Action-only denotes students trained without thoughts. All results use Qwen2.5-3B-Instruct students with the anonymous action representation.

Student	Clean	Thought Drop	Retention (%)
Qwen2.5-0.5B-Instruct	0.525	0.550	104.8
Qwen2.5-3B-Instruct	0.588	0.595	101.2
Qwen2.5-7B-Instruct	0.563	0.550	97.7
Qwen2.5-14B-Instruct	0.485	0.462	95.3

Appendix

Table 13: Robustness to thought removal on InterCode, a low-level action environment.

CoT Distillation

Chain-of-thought distillation transfers a model’s intermediate reasoning to smaller models so they can solve tasks with less compute.

See all 83 papers

Learning from Think-Mode Advantage via On-Policy Distillation
Sep 29, 2026Wanqi Ren, Jianxiang Wang, Danxuan Liu +2CoT Distillation Language Model Distillation
ByteDance, China
ActionOPD: Distilling Decisions, Not Tokens
Oct 14, 2026Huining Yuan, Xiangmin Yi, Zelai Xu +5LLM Agent Training Knowledge Distillation
Tsinghua University
AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents
Jul 18, 2026Yangqin Jiang, Chao HuangLLM Agent Self-Improvement LLM Agent Skill Learning
The University of Hong Kong

Sep 29, 2026J/K move · Enter open · S save

Learning from Think-Mode Advantage via On-Policy Distillation

Wanqi Ren, Jianxiang Wang, Danxuan Liu, Linyi Ding+1

ByteDance, China

CoT Distillation Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage via on-policy distillation (OPD). OPD preserves student-generated trajectories and provides dense token-level teacher targets at student-visited prefixes. Privileged reasoning is used during distillation rather than student inference. Uniform ThinkOPD, a natural think-enabled OPD baseline, conditions a fixed teacher on one shared think trace and uniformly distills every sibling student response. Although its prefixes are on-policy, the trace need not follow a route compatible with every complete response: the same privileged trace can induce different teacher-student discrepancies even when responses reach the same outcome. We summarize this interaction with trace-response divergence (TRD) and introduce ThinkOPD, which routes supervision at the response level by combining group-relative reward gain with a TRD-based compatibility proxy. Final response weights are normalized within each rollout group. Across mathematical reasoning and code generation, ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison. Controlled interventions show that outcome benefit and the TRD-based proxy provide complementary routing signals in this setting. Think-enabled OPD provides a controlled setting for studying how teacher advantage becomes transferable along student responses.

Abstract

CoT Distillation

Explore similar work

Learning from Think-Mode Advantage via On-Policy Distillation

ActionOPD: Distilling Decisions, Not Tokens

AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents