ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems

Akshey Sigdel; Rista Baral

cs.SEApr 2, 2026

ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems

Authors: Akshey Sigdel, Rista Baral

Organizations: Independent Researcher

Abstract

Tool using agents often fail for operational reasons even when language understanding is strong. Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior. We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets. The benchmark covers CRUD, retrieval, file, and scheduling environments with replayable fault injection. It reports success, invalid call behavior, policy violations, recovery quality, and budgeted efficiency. We release a public dataset with 6800 tasks and a reproducible evaluation pipeline. Baseline results show fault specific recovery gains for schema aware methods, while overall success remains limited under the released authorization and hard failure settings.

Figures & tables

Agent	Success	Violations	Recovery	Calls
Heuristic	0.250	0.168	0.000	2.95
Schema repair	0.250	0.166	0.250	3.25
Policy aware	0.250	0.166	0.250	3.25

TABLE I: Overall metrics on the public test split

Setting	Heuristic	Schema repair	Policy aware
Timeout success	0.499	0.502	0.502
Timeout recovery	0.000	0.502	0.502
Schema drift success	0.503	0.497	0.497
Schema drift recovery	0.000	0.497	0.497
Authz success	0.000	0.000	0.000
Rate limit success	0.000	0.000	0.000

TABLE II: Fault specific success and recovery rates

Tool-Use Evaluation

Tool-use evaluation measures whether AI agents select, call, and rely on tools correctly, efficiently, and safely across realistic tasks.

See all 96 papers

Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Jun 24, 2026Yang Tian, Zhengpeng Shi, Yu Zhou +1AI Agent Reliability AI Agent Benchmarks
School of AI, Shanghai Jiao Tong University1 · School of Cyber Science and Engineering, Southeast University2
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents
Oct 4, 2026Dolly Sah, Tanmay Sah, Harshul Jain +1Agent Failure Analysis AI Agent Reliability
Independent Researcher

Jun 24, 2026J/K move · Enter open · S save

Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability

Yang Tian, Zhengpeng Shi, Yu Zhou, Bo Zhao

School of AI, Shanghai Jiao Tong University1 · School of Cyber Science and Engineering, Southeast University2

AI Agent Reliability AI Agent Benchmarks Tool-Using Agents Tool Use LLM Agent Reliability

Open paper AskPDF arXiv

Large language models are increasingly deployed as agents that solve tasks by interacting with external tool environments. Although recent tool-use benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.

Abstract

Tool-Use Evaluation

Explore similar work

Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents