Large language model agents that use external tools are often implemented through reactive execution, in which reasoning is repeatedly recomputed after each observation, increasing latency and sensitivity to error propagation. This work introduces Profile--Then--Reason (PTR), a bounded execution framework for structured tool-augmented reasoning, in which a language model first synthesizes an explicit workflow, deterministic or guarded operators execute that workflow, a verifier evaluates the resulting trace, and repair is invoked only when the original workflow is no longer reliable. A mathematical formulation is developed in which the full pipeline is expressed as a composition of profile, routing, execution, verification, repair, and reasoning operators; under bounded repair, the number of language-model calls is restricted to two in the nominal case and three in the worst case. Experiments against a ReAct baseline on six benchmarks and four language models show that PTR achieves the pairwise exact-match advantage in 16 of 24 configurations. The results indicate that PTR is particularly effective on retrieval-centered and decomposition-heavy tasks, whereas reactive execution remains preferable when success depends on substantial online adaptation.
Figures & tables
Dataset
Model
PTR
ReAct
Adv.
TriviaQA
GPT-4o-Mini
0.660
0.160
PTR
TriviaQA
Claude Haiku 4.5
0.760
0.580
PTR
TriviaQA
GPT-5.4
0.780
0.700
PTR
TriviaQA
Claude Sonnet 4.6
0.820
0.520
PTR
NQ-Open
GPT-4o-Mini
0.260
0.060
PTR
NQ-Open
Claude Haiku 4.5
0.300
0.020
PTR
Table 1: Exact-match results for PTR and ReAct across all datasets and models. The column Adv. indicates which framework holds the pairwise advantage for each configuration.
Dataset
PTR adv.
ReAct adv.
Avg. Δ EM
TriviaQA
4/4
0/4
+0.265
NQ-Open
4/4
0/4
+0.292
StrategyQA
3/4
1/4
+0.070
GSM8K
3/4
1/4
+0.129
AQuA-RAT
1/4
3/4
-0.120
HotPotQA
1/4
3/4
-0.120
Table 2: Dataset-level comparison between PTR and ReAct. Pairwise advantage counts indicate the number of model configurations in which each framework achieves the strictly higher exact-match score. The average advantage is computed as PTR EM minus ReAct EM, averaged over the four models.
Figure 2: Dataset-level average exact-match difference ΔEM=EMPTR−EMReAct , averaged over the four evaluated language models. Positive values indicate a PTR advantage, whereas negative values indicate a ReAct advantage. PTR is favored on retrieval-centered and decomposition-heavy tasks (NQ-Open, TriviaQA, GSM8K, StrategyQA), while ReAct retains an advantage on AQuA-RAT and HotPotQA, where success depends more strongly on symbolic flexibility or online search refinement.
Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a 2×2 taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale 3.66× slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.
Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5
Shanghai AI Laboratory · East China Normal University · Soochow University +2
Large language model agents must discover relevant tools from large libraries, e.g., thousands of tools, as new needs arise during task execution. When several candidates could satisfy the same need, execution may be required to identify the relevant one. Investigating candidates in a shared trajectory can leave alternatives unexplored or allow failed attempts to influence subsequent reasoning. To address this, we present BRIA (BRanch-Inspect-Adopt), a simple yet effective agent that temporarily isolates candidate investigations in separate branches. Each branch retrieves and executes tools to investigate its candidate; a task agent inspects the resulting execution evidence and adopts only useful results into its continuing trajectory. BRIA retrieves additional tools as new capability needs arise during task execution. We evaluate BRIA on PlanBench-XL and MCP-Atlas using a shared pool of 1897 tools, three language models, and static and adaptive retrieval baselines. Across all settings, BRIA matches or exceeds the strongest retrieval baseline. On MCP-Atlas, it achieves an average claim score of 89.77, compared with 75.40 for MCP-Zero, a gain of 14.37 percentage points. BRIA also lies on the Pareto frontier for task performance and input-token cost.
Tool-augmented reasoning has become a popular direction for LLM-based agents, and it is widely assumed to improve reasoning and reliability. However, we demonstrate that this consensus does not always hold: in the presence of semantic distractors, tool-augmented reasoning does not necessarily outperform native CoT. To explain this performance gap, we propose a Factorized Intervention Framework that isolates the cost of prompt formatting, the overhead of the tool-calling protocol, and the actual gain from executing tools. Our analysis reveals a critical tradeoff: under semantic noise, the gains from tools often fail to offset the "tool-use tax", which is the performance degradation introduced by the tool-calling protocol itself. To address this, we introduce G-STEP, a lightweight inference-time gate to mitigate protocol-induced errors. While this yields partial recovery, our findings suggest that more substantial improvements still require strengthening the model's intrinsic reasoning and tool-interaction capabilities.
Kaituo Zhang, Zhen Xiong, Mingyu Zhong +4
University of Houston · University of Southern California · 3New York University +2