Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI

Md Jueal Mia; M. Hadi Amini

cs.AIAug 26, 2026

Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI

Authors: Md Jueal Mia, M. Hadi Amini

Organizations: Knight Foundation School of Computing and Information Sciences, Security, Optimization, and Learning for InterDependent Networks Laboratory (solid lab) Florida International University Miami, Florida, USA

Abstract

Recent advances in Large Language Models (LLMs) have shown that increased inference-time reasoning can improve performance on complex tasks. However, many existing approaches rely on fixed or preallocated reasoning controls, such as fixed token budgets, pre-execution difficulty estimates, or activation-space interventions, and are often evaluated on standalone reasoning benchmarks rather than full agentic workflows. These assumptions may not hold in agentic AI systems, where reasoning requirements evolve dynamically through planning, tool use, memory retrieval, and agent-to-agent interactions. Consequently, reasoning can become either excessive or insufficient, resulting in unnecessary computation, increased latency, planning drift, excessive tool use, or incomplete solutions. We argue that a major challenge for next-generation agentic AI is not merely how much reasoning a language model should perform, but how it should allocate reasoning according to evolving task demands. We characterize over-reasoning and under-reasoning as recurring failure modes of misallocated reasoning and evaluate them on MATH-500 and the GAIA public validation benchmark. Using tool-decision latency, token consumption, token-limit exhaustion, and answer correctness, our results suggest that cases classified as over-reasoning are associated with higher computational cost without proportional accuracy gains, whereas cases classified as under-reasoning are consistently associated with incorrect or incomplete solutions. These findings motivate future research on adaptive reasoning mechanisms for agentic AI.

Figures & tables

Figure 1. Overview of over-reasoning and under-reasoning in agentic AI. The agent loop is annotated with four failure modes: redundant reasoning, weak reasoning, repeated tool usage, and no tool usage (skipped tool calls).

Dataset	Final Model	Avg. Total Time (s)	Avg. Tool Decision Time (s)	Avg. Final Answer Time (s)	Avg. Tool Calls/Sample	Avg. Tool Input Tokens	Avg. Tool Output Tokens	Avg. Final Output Tokens	Avg. Final Reasoning Tokens	Token Limit Hits
MATH-500	Qwen3.5-4B	105.76	34.95	70.21	00.39	583.30	1224.89	2461.67	1732.39	190
	Llama-3.1-8B-Instruct	57.76	35.75	21.42	00.39	583.39	1225.09	904.48	N/A	71
	Phi-4-reasoning	216.46	35.08	180.80	00.39	583.30	1224.89	4046.59	2220.12	482
GAIA	Qwen3.5-4B	79.64	10.92	67.49	00.93	582.24	379.53	2353.90	1219.76	75
	Llama-3.1-8B-Instruct	22.77	10.94	10.71	00.93	582.24	379.53	452.23	N/A	10
	Phi-4-reasoning	195.00	10.74	182.94	00.93	582.24	379.53	4091.12	2897.95	164

Table 1. Average runtime and reasoning statistics in agentic AI across MATH-500 and GAIA. Tool agent model: Qwen3.5-4B.

Dataset	Final Model	Accuracy(%)	Answer Correct	Answer Incorrect	Incomplete	Token Limit Hits	Hit Accuracy(%)	No-Hit Accuracy(%)	Over-Reasoning	Under-Reasoning	Adequate Reasoning	Over-Reasoning Rate(%)	Under-Reasoning Rate(%)
MATH-500	Qwen3.5-4B	87.60	438	14	48	190	73.68	96.13	340	40	120	68.00	8.00
	Llama-3.1-8B-Instruct	49.80	249	187	64	71	9.86	56.41	99	110	291	19.80	22.00
	Phi-4-reasoning	92.40	462	5	33	482	92.12	100.00	447	24	29	89.40	4.80
GAIA	Qwen3.5-4B	12.12	20	79	66	75	13.33	11.11	63	61	41	38.18	36.97
	Llama-3.1-8B-Instruct	5.45	9	100	56	10	0.00	5.81	17	118	30	10.30	71.52
	Phi-4-reasoning	11.52	19	16	130	164	11.59	0.00	58	103	4	35.15	62.42

Table 2. Reasoning efficiency and answer quality in agentic AI across MATH-500 and GAIA. Tool agent model: Qwen3.5-4B. Judge model: GPT-4.1. Hit Accuracy denotes accuracy on samples where the final model reached the maximum output token limit, while No-Hit Accuracy denotes accuracy on samples that did not reach the maximum output token limit.

Agentic Reasoning

Agentic reasoning studies how AI systems plan, act, use tools, and revise decisions across multi-step tasks and changing environments.

See all 102 papers

Jun 1, 2026

Adaptive Latent Agentic Reasoning

Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents. Current LLM agents often generate verbose textual reasoning at every decision step and allocate reasoning effort nearly uniformly across turns, leading to substantial inefficiency in multi-turn agentic trajectories. We propose Adaptive Latent Agentic Reasoning (ALAR), a dual-mode framework that uses compact latent reasoning for routine turns and selectively escalates to explicit chain-of-thought when deeper deliberation is needed. ALAR learns latent reasoning by using the agent's actions as supervision anchors and is further optimized to use latent reasoning when it is sufficient for task success and reserve explicit CoT for harder decisions. Experiments on agentic search and tool-use benchmarks show that ALAR maintains comparable or better task accuracy while substantially reducing generated tokens by up to 43.6% in search and 84.6% in tool use. These results demonstrate that ALAR improves the accuracy-efficiency trade-off of LLM agents by reducing unnecessary textual reasoning while preserving explicit deliberation for harder decision steps.

Dongwon Jung, Peng Shi, Yi Zhang +2

University of California, Davis · University of Waterloo · Greenshoe, Inc.

Agentic Reasoning LLM Agents LLM Agent Training Efficient Language Model Inference Latent CoT Reasoning

Ask PDF

Oct 8, 2026

When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

Large language model (LLM)-based agents have demonstrated strong capabilities on complex tasks. They typically perform reasoning before each action throughout an interaction trajectory. However, reasoning may not be necessary at every turn, as reasoning produced earlier can continue to support subsequent actions. A key challenge is therefore to determine when existing reasoning remains sufficient and when a new reasoning step is needed, without relying on costly generation-based verification. We find that decreases in the likelihood of subsequent reference actions after removing additional reasoning closely track whether those actions remain recoverable given earlier reasoning, providing an effective and lightweight signal for estimating cross-turn action support. Based on this observation, we propose Reasoning Adaptation through Cross-Turn Estimation (RACE), a training approach for adaptive agent reasoning. RACE introduces a Likelihood-Guided Progressive Reasoning Cover Detection (LoGiC) procedure that progressively identifies reasoning turns whose removal has limited impact on the current and subsequent reference actions. The resulting removal signals are incorporated into both supervised fine-tuning and agentic reinforcement learning, enabling the policy to learn when to reason and when to act directly. Extensive experiments on four representative agent benchmarks show that RACE substantially reduces reasoning cost while maintaining or improving task performance.

Yiruo Cheng, Shen Huang, Xiaoshuai Song +5

Gaoling School of Artificial Intelligence, Renmin University of China · Alibaba Token Hub, Alibaba Group

Adaptive Reasoning in Language Models Agentic Reasoning Efficient Language Model Reasoning

Ask PDF

Aug 17, 2025

Fast, Slow, and Tool-augmented Thinking for LLMs: A Review

Large Language Models (LLMs) have demonstrated remarkable progress in reasoning across diverse domains. However, effective reasoning in real-world tasks requires adapting the reasoning strategy to the demands of the problem, ranging from fast, intuitive responses to deliberate, step-by-step reasoning and tool-augmented thinking. Drawing inspiration from cognitive psychology, we propose a novel taxonomy of LLM reasoning strategies along two knowledge boundaries: a fast/slow boundary separating intuitive from deliberative processes, and an internal/external boundary distinguishing reasoning grounded in the model's parameters from reasoning augmented by external tools. We systematically survey recent work on adaptive reasoning in LLMs and categorize methods based on key decision factors. We conclude by highlighting open challenges and future directions toward more adaptive, efficient, and reliable LLMs.

Xinda Jia, Jinpeng Li, Zezhong Wang +6

School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Technologies Co., Ltd., Beijing 100084, China · Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong 999077, China +3

Adaptive Reasoning in Language Models LLM Tool Use LLM Reasoning Large Reasoning Models

Ask PDF

Abstract

Agentic Reasoning

Explore similar work

Adaptive Latent Agentic Reasoning

When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

Fast, Slow, and Tool-augmented Thinking for LLMs: A Review