ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

Hyeonje Choi; Jeongsoo Lee; Hyojun Lee; Jay-Yoon Lee

cs.CLFeb 24, 2026

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

Authors: Hyeonje Choi, Jeongsoo Lee, Hyojun Lee, Jay-Yoon Lee

Organizations: Seoul National University

Abstract

We introduce \ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. \ToolMATH converts stepwise MATH solutions into reusable Python tools with natural-language descriptions and typed schemas, and pairs each problem with a tool environment requiring sequential tool use, intermediate-output reuse, and logically connected tool-call chains. \ToolMATH controls tool availability and catalog difficulty by constructing gold tools and graded distractors with varying similarity to gold tools. \ToolMATH also incorporates behavior-conditioned metrics, enabling diagnostic evaluation beyond final accuracy. Building on these measurements, \ToolMATH emphasizes three evaluation axes: (1) Adaptability measures how much Gold-only success is retained when gold tools are replaced entirely by distractors; (2) Robustness measures stability under adding distractors as a noise; and (3) Tool Connectivity measures whether models preserve accuracy over long executed tool-call chains. Furthermore, trace-level failure analyses characterize how models fail under each tool-catalog condition. Together, these diagnostics reveal distinct model profiles: reliable tool use, tool avoidance, adaptive substitution, and impacts of unreliable tool catalogs. Overall, \ToolMATH provides a controlled testbed for evaluating how language models adapt to changing tool availability, remain robust to distractors, and maintain correctness across long-horizon tool-use trajectories.

Figures & tables

Figure 1: ToolMATH construction and evaluation pipeline. Step 1: Tool extraction and validation. We convert MATH solution steps into schema-specified Python tools and retain only tools whose described semantics are consistent with their executed behavior via validation. Step 2: Tool-grounded evaluation with controlled redundancy. For each problem, we form a tool environment by combining its gold tools with distractor tools sampled from other problems.

Model	Condition	Accuracy	Tool-Acc	NoTool-Acc	Tool-call rate
Claude-4.6	Gold-only	98.4	97.9	99.5	65.6
Claude-4.6	Gold-present	97.8	96.5	99.4	67.8
Claude-4.6	Distractors-only	97.8	98.1	97.8	15.2
Gemini-3.1-Pro	Gold-only	99.2	90.4	99.7	5.4
Gemini-3.1-Pro	Gold-present	99.0	97.7	99.3	18.9
Gemini-3.1-Pro	Distractors-only	98.9	99.4	98.8	17.6

Table 1: Gold-only condition provides only the gold tools. Gold-present condition provides gold tools together with the same distractor budget of k = 5 with Level 3 sampling. Distractors-only condition removes the gold tools and provides only the same distractor budget of k = 5 with Level 3 sampling. All values are percentages.

Model	Adaptability	Rob.(L1)	Rob.(L2)	Rob.(L3)	Rob.(L4)	Rob.(L5)	Robustness Avg.
GPT-4o-mini	59.68	89.95	89.64	89.44	90.84	89.61	$89.90\pm 0.56$
Llama3-8B	8.63	66.07	67.65	66.82	77.20	72.95	$70.14\pm 4.78$
Qwen2.5-7B	65.12	94.29	94.81	90.32	89.55	86.91	$91.18\pm 3.33$

Table 2: Level-wise Adaptability and Robustness metrics in percentages. Robustness is reported level-wise across Levels 1–5, with the final column summarizing the mean

\pm

standard deviation.

Figure 2: Accuracy by the number of executed tool calls across models. Accuracy is plotted as a function of tool-call count under the main Plan+ReAct protocol and the same distractor budget of $k=5$ with Level 3 sampling. For shorter trajectories, vertical dashed lines mark the last observed tool-call count rather than implying failure beyond that point.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: GPT-4o-mini accuracy by logical-hop group, averaged over distractor budgets $k$ . Curves show the No tools baseline, the Gold-only condition, and the Gold-present condition with distractor similarity Levels 1–5. Accuracy decreases with hop count; higher-similarity distractors separate most clearly in higher-hop regimes.

Figure 4: Gold-present condition results for GPT-4o-mini (full grid). The grid reports accuracy over logical hops (columns) and distractor set size $k$ (rows), for distractor sampling levels 1–5. Baselines (No tools and Gold-only) are shown at the top.

Figure 5: Gold-present condition results for Qwen 2.5-7B (full grid).

Figure 6: Gold-present condition results for Llama 3-8B (full grid).

Figure 7: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for GPT-4o-mini (full grid).

Figure 8: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for Llama 3-8B (full grid).

Figure 9: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for Qwen 2.5-7B (full grid).

Figure 10: Hop-wise framework comparison under Gold-only tool lists. Framework differences are small at low hops but separate at higher hops, where Plan+ReAct most consistently maintains accuracy.

Figure 11: Hard-set frameworks for GPT-4o-mini (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 12: Hard-set frameworks for Llama 3-8B (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 13: Hard-set frameworks for Qwen 2.5-7B (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 14: ToolMATH vs. ToolMATH-Hard : hop-wise accuracy under tool availability and insufficiency (all models). Left: ToolMATH . Right: ToolMATH-Hard . We report accuracy by logical-hop group under the No tools baseline, the Gold-only condition, and the Distractors-only condition. For all settings involving distractors, we fix the distractor list to pure random sampling (Level 2) with $k=10$ tools.

Label	Definition
Plan Error ( $e_{1}$ )	The initial plan is ineffective (logically incorrect, or missing essential steps/subgoals).
Tool Selection Error ( $e_{2}$ )	At least one call selects an irrelevant tool or fails to use a suitable tool for a subproblem.
Tool Hallucination ( $e_{3}$ )	At least one call references a tool name that does not exist in the provided tool list.
Wrong Parameter Value ( $e_{4}$ )	At least one call uses schema-valid argument types but incorrect/irrelevant values (e.g., wrong constants/inputs, mismatched intermediate values).
Formatting Error ( $e_{5}$ )	At least one call violates the input schema or fails JSON conversion (e.g., wrong types/missing fields, malformed JSON, tool calls in plain text).
Thought Error ( $e_{6}$ )	Reasoning is incorrect given the final answer/tool logic, contradicts observations, or hallucinates Action / Observation to bypass interaction.

Appendix

Table 3: Failure-type taxonomy used in human trace analysis.

Figure 15: Failure cases under Gold-present (Level 3, $k=5$ ). Counts are computed from 100 failed instances per model; multiple labels per instance are allowed.

Figure 16: Failure co-occurrence heatmaps across models and tool-catalog conditions. Columns correspond to the Gold-only, Gold-present, and Distractors-only conditions. For Gold-present and Distractors-only, we use a distractor budget of $k=5$ with Level 3 sampling. Rows correspond to GPT-4o-mini, Llama 3-8B, and Qwen 2.5-7B. Each cell reports $P(e_{a}\mid e_{b})$ over failed traces. The labels $e_{1}$ – $e_{9}$ correspond to the failure taxonomy listed in Table 3 .

Tool Selection for Language Model Agents

See all 27 papers

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
Aug 23, 2026YiShan Zheng, Yuan Wu, Yi ChangLLM Agent Evaluation LLM Tool Use
Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics
May 9, 2026Zicheng Lyu, Wenjie Yang, Shengzhong Zhang +1Mathematical Reasoning Benchmarks LLM Evaluation
Fudan University Shanghai, China · Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Fudan University Shanghai Innovation Institute Shanghai, China

Jul 6, 2026J/K move · Enter open · S save

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Open paper AskPDF arXiv

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Abstract

Tool Selection for Language Model Agents

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

Re2^22Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics