LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.
Figures & tables
Figure 1: TAC-CoT improves over the Explicit-CoT baseline in both agentic math and question-answering (QA) settings. The two methods use the same tool-using agents and differ only in their reasoning backbone: TAC-CoT uses implicit CoT, whereas the baseline uses explicit CoT. # Fwd denotes the number of model forward passes Kuzina et al. (2026) ; Wei et al. (2026) .
Figure 2: Overview of TAC-CoT compared with the Explicit-CoT agent and implicit-CoT methods. TAC-CoT uses implicit CoT as its reasoning backbone while preserving interaction segments in explicit natural language. This design enables implicit CoT to use tools and allocate query-dependent implicit reasoning budgets, improving efficiency, performance, and robustness over the Explicit-CoT agent. Here, n denotes the number of discrete tokens generated by the Explicit-CoT agent at each step, b denotes the number of continuous tokens in each implicit reasoning segment of TAC-CoT , and m denotes the fixed continuous-token budget used by prior implicit-CoT methods.
Figure 3: Training objective of TAC-CoT . The student generates continuous thought tokens, from which explicit reasoning chunks are reconstructed for supervision. Explicit outputs are trained with cross-entropy loss, while answer-boundary hidden states are distilled from an explicit-CoT teacher. The final objective combines student, teacher, and distillation losses.
Task
Benchmark
Setting
Math
GSM8K Cobbe et al. (2021)
In-domain
GSM-Hard Gao et al. (2023)
Out-of-domain
SVAMP Patel et al. (2021)
Out-of-domain
QA
HotpotQA Yang et al. (2018)
In-domain
2WikiMultihopQA Ho et al. (2020)
Out-of-domain
KILT-TriviaQA Petroni et al. (2021)
Out-of-domain
Table 1: Evaluation benchmarks for agentic mathematical reasoning, agentic QA and tool-use capability.
Math
Question-Answering
Method
TU
In-domain
Out-of-domain
In-domain
Out-of-domain
GSM8K
GSM-Hard
SVAMP
Average
HotpotQA
2WikiMulti
TriviaQA
Average
Acc↑
# Fwd↓
Acc↑
# Fwd↓
Acc↑
# Fwd↓
Acc↑
# Fwd↓
EM↑
# Fwd↓
EM↑
# Fwd↓
EM↑
# Fwd↓
EM↑
# Fwd↓
Qwen2.5-0.5B-Instruct
Explicit-CoT
✗
50.9
117.3
13.4
160.0
59.0
61.9
41.1
113.1
13.2
951.2
21.0
739.1
11.3
891.6
15.2
860.6
TokenSkip
✗
46.2
73.6
10.4
104.7
49.8
38.5
35.5
72.3
14.2
502.8
21.7
452.3
9.9
474.1
15.3
476.4
Table 2: Main results on the math and QA benchmarks (Qwen2.5-0.5B, Llama-3.2-1B, and Qwen3-4B). # Fwd is the average number of forward passes per query. TU indicates whether tools are used. Best scores are highlighted in bold ; * and † mark p<0.01 and p<0.05 improvements over the second-best via paired bootstrap with 105 samples ( Koehn, 2004 ) .
Method
HotpotQA
2WikiMulti
TriviaQA
@5
@10
@5
@10
@5
@10
Explicit-CoT
48.8
51.7
70.9
71.6
72.6
75.5
TAC-CoT
51.8 †
55.4 *
73.7 *
74.4 †
74.5
77.4
Table 3: Answer Coverage @ k (%) over retrieval steps on the QA benchmarks, Qwen2.5-0.5B-Instruct. All AC values are reported as percentages; Best scores are bold ; * and † mark p<0.01 and p<0.05 improvements over the underlined Explicit-CoT.
Figure 4: Solvable win rate (SoWR) of TAC-CoT against Explicit-CoT on ToolBench, judged by GPT-5.1. ‘Single’ denotes the single-tool scenario, and ‘Multi’ denotes the multi-tool scenario. * marks p<0.01 .
Metric
E-CoT
# Replaced
Ours
1
2
3
4
Mean Acc ↑
41.1
41.7
42.4
43.4
44.9
45.4
# Fwd ↓
113.1
96.2
79.2
63.4
58.9
53.7
Table 4: Effect of replacement depth on Qwen2.5-0.5B-Instruct average accuracy over GSM8K, GSM-Hard, and SVAMP. All accuracy values are reported as percentages. E-CoT denotes the Explicit-CoT agent, and # Replaced denotes the number of explicit reasoning segments swapped for continuous thought tokens.
Figure 5: GSM8K accuracy stratified by the offset between the number of generated reasoning steps and the gold step count in the gold solution (negative = under-thinking, 0= matched depth, positive = over-thinking).
Figure 6: Accuracy comparison across GSM8K subsets grouped by the number of required reasoning steps. Each n denotes the number of samples in the corresponding reasoning-step group.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Qwen2.5 0.5B-Instruct
Llama-3.2 1B-Instruct
Qwen3 4B
Fine-tuning strategy
Full fine-tuning
LoRA
LoRA
LoRA rank
–
128
128
LoRA scaling factor
–
32
32
LoRA dropout
–
0.1
0.1
Learning rate
1×10−4
8×10−4
1×10−4
LR scheduler
Cosine
Cosine
Cosine
Appendix
Table 5: Training hyperparameters used for each backbone model. Values are shared across all compared methods unless otherwise specified.
# CT
Math
GSM8K
GSM-Hard
SVAMP
Average
0
55.7
13.9
56.3
42.0
1
58.5
14.7
61.5
44.9
3
58.8
14.2
63.3
45.4
Appendix
Table 6: Effect of the number of continuous thought tokens (# CT) per continuous segment on math datasets, using Qwen2.5-0.5B-Instruct. All scores are reported as percentages.
Method
GSM8K
GSM-Hard
SVAMP
Average
No-tool setting
No Reasoning
55.7
13.9
56.3
42.0
TAC-CoT
58.8
14.2
63.3
45.4
Tool-enabled setting
TokenSkip
52.3
27.0
57.0
45.4
TAC-CoT
61.2
32.8
67.7
53.9
Appendix
Table 7: Controlled comparisons on the mathematical reasoning benchmarks using Qwen2.5-0.5B-Instruct. The no-natural-language reasoning control removes natural-language rationales and continuous thought tokens while retaining arithmetic equations. TokenSkip + Tool provides a concise explicit-CoT control under the tool-enabled setting. All scores are accuracy (%).
Math
Method
GSM8K
GSM-Hard
SVAMP
Average
Explicit-CoT
52.2
13.9
68.5
44.9
TAC-CoT
62.0
14.9
68.7
48.5
Appendix
Table 8: Math accuracy of Explicit-CoT agent and TAC-CoT on GSM8K, GSM-Hard, and SVAMP when Llama-3.2-1B-Instruct is fully fine-tuned (no LoRA), reported for reference against the LoRA-adapted Llama numbers in Table 2 .
Math
Method
GSM8K
GSM-Hard
SVAMP
Average
Explicit-CoT
32.8
6.6
38.2
25.9
TAC-CoT
47.7
10.8
45.0
34.5
Appendix
Table 9: Math accuracy of Explicit-CoT agent and TAC-CoT on GSM8K, GSM-Hard, and SVAMP under a GPT2 backbone with full fine-tuning, complementing the Qwen and Llama backbones reported in Table 2 .
Figure 7: Effect of reasoning depth on error accumulation.
Figure 8: Impact of the number of tool calls on GSM8K accuracy. TC=0 denotes no tool calling, and TC=all denotes unlimited tool calls.
Method
Dataset
# Fwd
ms / sample
Agentic Math
TAC-CoT
GSM8K
43.9
545.0
GSM-Hard
57.0
712.2
SVAMP
26.2
335.9
Average
42.4
531.0
Explicit-CoT
GSM8K
109.2
1232.2
Appendix
Table 10: Inference-time comparison between TAC-CoT and Explicit-CoT on the Qwen2.5-0.5B-Instruct backbone. ms/sample denotes the average wall-clock inference latency in milliseconds per sample, measured on a single NVIDIA L40S GPU.
Figure 9: Case studies on Qwen2.5-0.5B-Instruct. Gold Explicit-CoT (SFT) SIM-CoT TAC-CoT (ours) . Red : error step. (a) Explicit-CoT applies the overtime rate to all 45 h instead of 5 h; SIM-CoT’s fixed latent budget yields a wrong answer with no intermediate steps; TAC-CoT grounds each arithmetic step via calculator calls. (b) SIM-CoT hallucinates “Wake Island” without retrieval; Explicit-CoT searches but rationalizes a wrong answer with a false claim (Thought 9); TAC-CoT spots “Laysan” in search suggestions and retrieves the evidence.
Figure 11: Prompt used to construct QA tool-use trajectories. We prompt DeepSeek-V3 as a ReAct-style Wikipedia agent with access to Search , Lookup , and Finish .
Figure 12: Prompt used to construct QA tool-use trajectories, continued from Figure 11 . The few-shot demonstrations follow the ReAct prompt format.
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.
Chain-of-thought (CoT) is a standard approach for eliciting reasoning capabilities from large language models (LLMs). However, the common CoT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative reasoning. In this paper, we introduce CopT, a reformulated reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous embeddings as inference-time contrastive verifiers. Specifically, it contrasts the model's support for the same generated tokens under discrete-token inputs and continuous-embedding inputs, yielding a sequence-level reverse KL estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, CopT performs further on-policy thinking, where a second KL estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic reasoning tasks, CopT improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.
Dachuan Shi, Hanlin Zhu, Xiangchi Yuan +4
Georgia Tech · UC Berkeley · Stanford University +1
Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents. Current LLM agents often generate verbose textual reasoning at every decision step and allocate reasoning effort nearly uniformly across turns, leading to substantial inefficiency in multi-turn agentic trajectories. We propose Adaptive Latent Agentic Reasoning (ALAR), a dual-mode framework that uses compact latent reasoning for routine turns and selectively escalates to explicit chain-of-thought when deeper deliberation is needed. ALAR learns latent reasoning by using the agent's actions as supervision anchors and is further optimized to use latent reasoning when it is sufficient for task success and reserve explicit CoT for harder decisions. Experiments on agentic search and tool-use benchmarks show that ALAR maintains comparable or better task accuracy while substantially reducing generated tokens by up to 43.6% in search and 84.6% in tool use. These results demonstrate that ALAR improves the accuracy-efficiency trade-off of LLM agents by reducing unnecessary textual reasoning while preserving explicit deliberation for harder decision steps.
Dongwon Jung, Peng Shi, Yi Zhang +2
University of California, Davis · University of Waterloo · Greenshoe, Inc.
Tool-augmented reasoning has become a popular direction for LLM-based agents, and it is widely assumed to improve reasoning and reliability. However, we demonstrate that this consensus does not always hold: in the presence of semantic distractors, tool-augmented reasoning does not necessarily outperform native CoT. To explain this performance gap, we propose a Factorized Intervention Framework that isolates the cost of prompt formatting, the overhead of the tool-calling protocol, and the actual gain from executing tools. Our analysis reveals a critical tradeoff: under semantic noise, the gains from tools often fail to offset the "tool-use tax", which is the performance degradation introduced by the tool-calling protocol itself. To address this, we introduce G-STEP, a lightweight inference-time gate to mitigate protocol-induced errors. While this yields partial recovery, our findings suggest that more substantial improvements still require strengthening the model's intrinsic reasoning and tool-interaction capabilities.
Kaituo Zhang, Zhen Xiong, Mingyu Zhong +4
University of Houston · University of Southern California · 3New York University +2