TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

Juseon Do; Sangmin Bae; Jeesu Jung; Hwanjun Song

cs.CLNov 1, 2026

TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

Authors: Juseon Do, Sangmin Bae, Jeesu Jung, Hwanjun Song

Organizations: Data Intelligence System Lab, KAIST · Optimization and Statistical Inference Lab, KAIST AI · Cluvion

Abstract

LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.

Figures & tables

Figure 1: TAC-CoT improves over the Explicit-CoT baseline in both agentic math and question-answering (QA) settings. The two methods use the same tool-using agents and differ only in their reasoning backbone: TAC-CoT uses implicit CoT, whereas the baseline uses explicit CoT. # Fwd denotes the number of model forward passes Kuzina et al. (2026) ; Wei et al. (2026) .

Figure 2: Overview of TAC-CoT compared with the Explicit-CoT agent and implicit-CoT methods. TAC-CoT uses implicit CoT as its reasoning backbone while preserving interaction segments in explicit natural language. This design enables implicit CoT to use tools and allocate query-dependent implicit reasoning budgets, improving efficiency, performance, and robustness over the Explicit-CoT agent. Here,

n

denotes the number of discrete tokens generated by the Explicit-CoT agent at each step,

b

denotes the number of continuous tokens in each implicit reasoning segment of TAC-CoT , and

m

denotes the fixed continuous-token budget used by prior implicit-CoT methods.

Figure 3: Training objective of TAC-CoT . The student generates continuous thought tokens, from which explicit reasoning chunks are reconstructed for supervision. Explicit outputs are trained with cross-entropy loss, while answer-boundary hidden states are distilled from an explicit-CoT teacher. The final objective combines student, teacher, and distillation losses.

Task	Benchmark	Setting
Math	GSM8K Cobbe et al. (2021)	In-domain
	GSM-Hard Gao et al. (2023)	Out-of-domain
	SVAMP Patel et al. (2021)	Out-of-domain
QA	HotpotQA Yang et al. (2018)	In-domain
	2WikiMultihopQA Ho et al. (2020)	Out-of-domain
	KILT-TriviaQA Petroni et al. (2021)	Out-of-domain

Table 1: Evaluation benchmarks for agentic mathematical reasoning, agentic QA and tool-use capability.

		Math								Question-Answering
Method	TU	In-domain		Out-of-domain							In-domain		Out-of-domain
Method	TU	GSM8K		GSM-Hard		SVAMP		Average			HotpotQA		2WikiMulti		TriviaQA		Average
		Acc↑	# Fwd↓	Acc↑	# Fwd↓	Acc↑	# Fwd↓	Acc↑	# Fwd↓		EM↑	# Fwd↓	EM↑	# Fwd↓	EM↑	# Fwd↓	EM↑	# Fwd↓
Qwen2.5-0.5B-Instruct
Explicit-CoT	✗	50.9	117.3	13.4	160.0	59.0	61.9	41.1	113.1		13.2	951.2	21.0	739.1	11.3	891.6	15.2	860.6
TokenSkip	✗	46.2	73.6	10.4	104.7	49.8	38.5	35.5	72.3		14.2	502.8	21.7	452.3	9.9	474.1	15.3	476.4

Table 2: Main results on the math and QA benchmarks (Qwen2.5-0.5B, Llama-3.2-1B, and Qwen3-4B). # Fwd is the average number of forward passes per query. TU indicates whether tools are used. Best scores are highlighted in bold ; * and

\dagger

mark

p<0.01

and

p<0.05

improvements over the second-best via paired bootstrap with

10^{5}

samples ( Koehn, 2004 ) .

Method	HotpotQA		2WikiMulti		TriviaQA
Method	@5	@10	@5	@10	@5	@10
Explicit-CoT	48.8	51.7	70.9	71.6	72.6	75.5
TAC-CoT	51.8 †	55.4 *	73.7 *	74.4 †	74.5	77.4

Table 3: Answer Coverage @

k

(%) over retrieval steps on the QA benchmarks, Qwen2.5-0.5B-Instruct. All AC values are reported as percentages; Best scores are bold ; * and

\dagger

mark

p<0.01

and

p<0.05

improvements over the underlined Explicit-CoT.

Figure 4: Solvable win rate (SoWR) of TAC-CoT against Explicit-CoT on ToolBench, judged by GPT-5.1. ‘Single’ denotes the single-tool scenario, and ‘Multi’ denotes the multi-tool scenario. * marks

p<0.01

Metric	E-CoT	# Replaced				Ours
Metric	E-CoT	1	2	3	4	Ours
Mean Acc $\uparrow$	41.1	41.7	42.4	43.4	44.9	45.4
# Fwd $\downarrow$	113.1	96.2	79.2	63.4	58.9	53.7

Table 4: Effect of replacement depth on Qwen2.5-0.5B-Instruct average accuracy over GSM8K, GSM-Hard, and SVAMP. All accuracy values are reported as percentages. E-CoT denotes the Explicit-CoT agent, and # Replaced denotes the number of explicit reasoning segments swapped for continuous thought tokens.

Figure 5: GSM8K accuracy stratified by the offset between the number of generated reasoning steps and the gold step count in the gold solution (negative

=

under-thinking,

0

=

matched depth, positive

=

over-thinking).

Figure 6: Accuracy comparison across GSM8K subsets grouped by the number of required reasoning steps. Each

n

denotes the number of samples in the corresponding reasoning-step group.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Qwen2.5 0.5B-Instruct	Llama-3.2 1B-Instruct	Qwen3 4B
Fine-tuning strategy	Full fine-tuning	LoRA	LoRA
LoRA rank	–	128	128
LoRA scaling factor	–	32	32
LoRA dropout	–	0.1	0.1
Learning rate	$1\times 10^{-4}$	$8\times 10^{-4}$	$1\times 10^{-4}$
LR scheduler	Cosine	Cosine	Cosine

Appendix

Table 5: Training hyperparameters used for each backbone model. Values are shared across all compared methods unless otherwise specified.

# CT	Math
# CT	GSM8K	GSM-Hard	SVAMP	Average
0	55.7	13.9	56.3	42.0
1	58.5	14.7	61.5	44.9
3	58.8	14.2	63.3	45.4

Appendix

Table 6: Effect of the number of continuous thought tokens (# CT) per continuous segment on math datasets, using Qwen2.5-0.5B-Instruct. All scores are reported as percentages.

Method	GSM8K	GSM-Hard	SVAMP	Average
No-tool setting
No Reasoning	55.7	13.9	56.3	42.0
TAC-CoT	58.8	14.2	63.3	45.4
Tool-enabled setting
TokenSkip	52.3	27.0	57.0	45.4
TAC-CoT	61.2	32.8	67.7	53.9

Appendix

Table 7: Controlled comparisons on the mathematical reasoning benchmarks using Qwen2.5-0.5B-Instruct. The no-natural-language reasoning control removes natural-language rationales and continuous thought tokens while retaining arithmetic equations. TokenSkip + Tool provides a concise explicit-CoT control under the tool-enabled setting. All scores are accuracy (%).

	Math
Method	GSM8K	GSM-Hard	SVAMP	Average
Explicit-CoT	52.2	13.9	68.5	44.9
TAC-CoT	62.0	14.9	68.7	48.5

Appendix

Table 8: Math accuracy of Explicit-CoT agent and TAC-CoT on GSM8K, GSM-Hard, and SVAMP when Llama-3.2-1B-Instruct is fully fine-tuned (no LoRA), reported for reference against the LoRA-adapted Llama numbers in Table 2 .

	Math
Method	GSM8K	GSM-Hard	SVAMP	Average
Explicit-CoT	32.8	6.6	38.2	25.9
TAC-CoT	47.7	10.8	45.0	34.5

Appendix

Table 9: Math accuracy of Explicit-CoT agent and TAC-CoT on GSM8K, GSM-Hard, and SVAMP under a GPT2 backbone with full fine-tuning, complementing the Qwen and Llama backbones reported in Table 2 .

Figure 7: Effect of reasoning depth on error accumulation.

Figure 8: Impact of the number of tool calls on GSM8K accuracy. TC=0 denotes no tool calling, and TC=all denotes unlimited tool calls.

Method	Dataset	# Fwd	ms / sample
Agentic Math
TAC-CoT	GSM8K	43.9	545.0
	GSM-Hard	57.0	712.2
	SVAMP	26.2	335.9
	Average	42.4	531.0
Explicit-CoT	GSM8K	109.2	1232.2

Appendix

Table 10: Inference-time comparison between TAC-CoT and Explicit-CoT on the Qwen2.5-0.5B-Instruct backbone. ms/sample denotes the average wall-clock inference latency in milliseconds per sample, measured on a single NVIDIA L40S GPU.

Panel of Figure 9 — Figure 9: Case studies on Qwen2.5-0.5B-Instruct. Gold Explicit-CoT (SFT) SIM-CoT TAC-CoT (ours) . Red : error step. (a) Explicit-CoT applies the overtime rate to all 45 h instead of 5 h; SIM-CoT’s fixed latent budget yields a wrong answer with no intermediate steps; TAC-CoT grounds each arithmetic step via calculator calls. (b) SIM-CoT hallucinates “Wake Island” without retrieval; Explicit-CoT searches but rationalizes a wrong answer with a false claim (Thought 9); TAC-CoT spots “Laysan” in search suggestions and retrieves the evidence.

Panel of Figure 11 — Figure 11: Prompt used to construct QA tool-use trajectories. We prompt DeepSeek-V3 as a ReAct-style Wikipedia agent with access to Search , Lookup , and Finish .

Panel of Figure 12 — Figure 12: Prompt used to construct QA tool-use trajectories, continued from Figure 11 . The few-shot demonstrations follow the ReAct prompt format.

Implicit Reasoning in Language Models

Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.

See all 102 papers

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
May 19, 2026Dachuan Shi, Hanlin Zhu, Xiangchi Yuan +4Agentic Reasoning Efficient Language Model Inference
Georgia Tech · UC Berkeley · Stanford University +1
Adaptive Latent Agentic Reasoning
Jun 1, 2026Dongwon Jung, Peng Shi, Yi Zhang +2Agentic Reasoning LLM Agents
University of California, Davis · University of Waterloo · Greenshoe, Inc.
Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents
Apr 30, 2026Kaituo Zhang, Zhen Xiong, Mingyu Zhong +4Tool-Augmented Language Model Agents Agentic Reasoning
University of Houston · University of Southern California · 3New York University +2

May 19, 2026J/K move · Enter open · S save

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

Dachuan Shi, Hanlin Zhu, Xiangchi Yuan, Wanjia Zhao+3

Georgia Tech · UC Berkeley · Stanford University+1

Agentic Reasoning Efficient Language Model Inference LLM Reasoning

Open paper AskPDF arXiv

Chain-of-thought (CoT) is a standard approach for eliciting reasoning capabilities from large language models (LLMs). However, the common CoT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative reasoning. In this paper, we introduce CopT, a reformulated reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous embeddings as inference-time contrastive verifiers. Specifically, it contrasts the model's support for the same generated tokens under discrete-token inputs and continuous-embedding inputs, yielding a sequence-level reverse KL estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, CopT performs further on-policy thinking, where a second KL estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic reasoning tasks, CopT improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.

Abstract

Implicit Reasoning in Language Models

Explore similar work

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

Adaptive Latent Agentic Reasoning

Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents