ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

Hyeonje Choi; Jeongsoo Lee; Hyojun Lee; Jay-Yoon Lee

cs.CLFeb 24, 2026

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

Authors: Hyeonje Choi, Jeongsoo Lee, Hyojun Lee, Jay-Yoon Lee

Organizations: Seoul National University

Abstract

We introduce \ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. \ToolMATH converts stepwise MATH solutions into reusable Python tools with natural-language descriptions and typed schemas, and pairs each problem with a tool environment requiring sequential tool use, intermediate-output reuse, and logically connected tool-call chains. \ToolMATH controls tool availability and catalog difficulty by constructing gold tools and graded distractors with varying similarity to gold tools. \ToolMATH also incorporates behavior-conditioned metrics, enabling diagnostic evaluation beyond final accuracy. Building on these measurements, \ToolMATH emphasizes three evaluation axes: (1) Adaptability measures how much Gold-only success is retained when gold tools are replaced entirely by distractors; (2) Robustness measures stability under adding distractors as a noise; and (3) Tool Connectivity measures whether models preserve accuracy over long executed tool-call chains. Furthermore, trace-level failure analyses characterize how models fail under each tool-catalog condition. Together, these diagnostics reveal distinct model profiles: reliable tool use, tool avoidance, adaptive substitution, and impacts of unreliable tool catalogs. Overall, \ToolMATH provides a controlled testbed for evaluating how language models adapt to changing tool availability, remain robust to distractors, and maintain correctness across long-horizon tool-use trajectories.

Figures & tables

Figure 1: ToolMATH construction and evaluation pipeline. Step 1: Tool extraction and validation. We convert MATH solution steps into schema-specified Python tools and retain only tools whose described semantics are consistent with their executed behavior via validation. Step 2: Tool-grounded evaluation with controlled redundancy. For each problem, we form a tool environment by combining its gold tools with distractor tools sampled from other problems.

Model	Condition	Accuracy	Tool-Acc	NoTool-Acc	Tool-call rate
Claude-4.6	Gold-only	98.4	97.9	99.5	65.6
Claude-4.6	Gold-present	97.8	96.5	99.4	67.8
Claude-4.6	Distractors-only	97.8	98.1	97.8	15.2
Gemini-3.1-Pro	Gold-only	99.2	90.4	99.7	5.4
Gemini-3.1-Pro	Gold-present	99.0	97.7	99.3	18.9
Gemini-3.1-Pro	Distractors-only	98.9	99.4	98.8	17.6

Table 1: Gold-only condition provides only the gold tools. Gold-present condition provides gold tools together with the same distractor budget of k = 5 with Level 3 sampling. Distractors-only condition removes the gold tools and provides only the same distractor budget of k = 5 with Level 3 sampling. All values are percentages.

Model	Adaptability	Rob.(L1)	Rob.(L2)	Rob.(L3)	Rob.(L4)	Rob.(L5)	Robustness Avg.
GPT-4o-mini	59.68	89.95	89.64	89.44	90.84	89.61	$89.90\pm 0.56$
Llama3-8B	8.63	66.07	67.65	66.82	77.20	72.95	$70.14\pm 4.78$
Qwen2.5-7B	65.12	94.29	94.81	90.32	89.55	86.91	$91.18\pm 3.33$

Table 2: Level-wise Adaptability and Robustness metrics in percentages. Robustness is reported level-wise across Levels 1–5, with the final column summarizing the mean

\pm

standard deviation.

Figure 2: Accuracy by the number of executed tool calls across models. Accuracy is plotted as a function of tool-call count under the main Plan+ReAct protocol and the same distractor budget of $k=5$ with Level 3 sampling. For shorter trajectories, vertical dashed lines mark the last observed tool-call count rather than implying failure beyond that point.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: GPT-4o-mini accuracy by logical-hop group, averaged over distractor budgets $k$ . Curves show the No tools baseline, the Gold-only condition, and the Gold-present condition with distractor similarity Levels 1–5. Accuracy decreases with hop count; higher-similarity distractors separate most clearly in higher-hop regimes.

Figure 4: Gold-present condition results for GPT-4o-mini (full grid). The grid reports accuracy over logical hops (columns) and distractor set size $k$ (rows), for distractor sampling levels 1–5. Baselines (No tools and Gold-only) are shown at the top.

Figure 5: Gold-present condition results for Qwen 2.5-7B (full grid).

Figure 6: Gold-present condition results for Llama 3-8B (full grid).

Figure 7: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for GPT-4o-mini (full grid).

Figure 8: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for Llama 3-8B (full grid).

Figure 9: Gold-present condition (Level 1) vs. Distractors-only condition (Level 1) for Qwen 2.5-7B (full grid).

Figure 10: Hop-wise framework comparison under Gold-only tool lists. Framework differences are small at low hops but separate at higher hops, where Plan+ReAct most consistently maintains accuracy.

Figure 11: Hard-set frameworks for GPT-4o-mini (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 12: Hard-set frameworks for Llama 3-8B (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 13: Hard-set frameworks for Qwen 2.5-7B (heatmap): no tools vs. ReAct vs. DFSDT vs. Plan+ReAct.

Figure 14: ToolMATH vs. ToolMATH-Hard : hop-wise accuracy under tool availability and insufficiency (all models). Left: ToolMATH . Right: ToolMATH-Hard . We report accuracy by logical-hop group under the No tools baseline, the Gold-only condition, and the Distractors-only condition. For all settings involving distractors, we fix the distractor list to pure random sampling (Level 2) with $k=10$ tools.

Label	Definition
Plan Error ( $e_{1}$ )	The initial plan is ineffective (logically incorrect, or missing essential steps/subgoals).
Tool Selection Error ( $e_{2}$ )	At least one call selects an irrelevant tool or fails to use a suitable tool for a subproblem.
Tool Hallucination ( $e_{3}$ )	At least one call references a tool name that does not exist in the provided tool list.
Wrong Parameter Value ( $e_{4}$ )	At least one call uses schema-valid argument types but incorrect/irrelevant values (e.g., wrong constants/inputs, mismatched intermediate values).
Formatting Error ( $e_{5}$ )	At least one call violates the input schema or fails JSON conversion (e.g., wrong types/missing fields, malformed JSON, tool calls in plain text).
Thought Error ( $e_{6}$ )	Reasoning is incorrect given the final answer/tool logic, contradicts observations, or hallucinates Action / Observation to bypass interaction.

Appendix

Table 3: Failure-type taxonomy used in human trace analysis.

Figure 15: Failure cases under Gold-present (Level 3, $k=5$ ). Counts are computed from 100 failed instances per model; multiple labels per instance are allowed.

Figure 16: Failure co-occurrence heatmaps across models and tool-catalog conditions. Columns correspond to the Gold-only, Gold-present, and Distractors-only conditions. For Gold-present and Distractors-only, we use a distractor budget of $k=5$ with Level 3 sampling. Rows correspond to GPT-4o-mini, Llama 3-8B, and Qwen 2.5-7B. Each cell reports $P(e_{a}\mid e_{b})$ over failed traces. The labels $e_{1}$ – $e_{9}$ correspond to the failure taxonomy listed in Table 3 .

Tool Selection for Language Model Agents

See all 27 papers

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Aug 23, 2026

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyond text generation. However, clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call. We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents, where a tool-calling agent is an LLM system that selects a tool, supplies structured arguments, and interprets its returned feedback. ToolRobustBench aligns four perturbation families with the tool-use pipeline: tool-interface, user-intent, tool-output/observation, and runtime-environment perturbations. It attributes failures to tool selection, schema grounding, argument binding, tool-output/runtime-feedback handling, and E2E task success. Experiments on 15,456 single-family instances across 7 models, 16 sampled local tools, 4 perturbation families, and 14 subtypes show high but non-uniform clean performance and substantial robustness degradation, with tool-output/observation perturbation the dominant bottleneck. Mixed-family experiments reveal non-additive failure patterns that are not explained by isolated single-family results. Thus, ToolRobustBench provides a deterministic and cascade-aware benchmark for diagnosing robustness beyond clean tool-calling accuracy;

YiShan Zheng, Yuan Wu, Yi Chang

LLM Agent Evaluation LLM Tool Use AI Agent Benchmarks

Ask PDF

May 9, 2026

Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

Large language models are increasingly capable at closed-world mathematical reasoning, but research assistance also requires source-grounded use of the literature. When a proof reaches a non-trivial step, a useful assistant should determine whether the needed tool (e.g., a lemma) already exists, identify a suitable scholarly source, and verify that its assumptions align with the current proof context. To rigorously evaluate such capabilities, we introduce Re

^2

Math, a benchmark for tool-grounded retrieval from partial mathematical proofs. Each instance is built from a candidate instrumental citation in the proof of a main theorem, with hierarchical context and an optional leakage-controlled anchor hint. We also make the task source-grounded yet citation-agnostic in that any admissible theorem sufficient for the proof transition is accepted. Evaluation uses a release-frozen retrieval artifact, ensuring reproducibility, while the benchmark itself supports automatic, continual expansion with newly constructed instances. On the current benchmark test set, the best fixed-judge ToolAcc reaches 7.0%, despite substantially higher rates of source grounding, indicating that current systems often retrieve valid statements but fail to establish their applicability to the local proof step. By decoupling citation recall, grounding, and proof-gap sufficiency, Re

^2

Math transforms literature-grounded mathematical tool use into a controlled diagnostic task.

Zicheng Lyu, Wenjie Yang, Shengzhong Zhang +1

Fudan University Shanghai, China · Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Fudan University Shanghai Innovation Institute Shanghai, China

Mathematical Reasoning Benchmarks LLM Evaluation Mathematical Reasoning Information Retrieval Evidence Retrieval

Ask PDF

Abstract

Tool Selection for Language Model Agents

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

Re2^22Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics