cs.AINov 16, 2026

AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery

Authors: Sachin Gupta

Abstract

Does recovery advice help an agent finish a task after a tool fails? AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response. The prompt supplies the expected issue and report text, so the task tests execution rather than content generation. In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive. Later experiments test a revised recovery policy. GPT-4o mini improves, while Sol and Claude already complete the recoverable cases. In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included. Removing issue IDs gives 47/48 successes versus 46/48 with IDs; other execution information remains. A second checker agrees on 1,879 observed primary outcomes and 241 constructed test cases. No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.

Figures & tables

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents

    Oct 7, 2026Obada KraishanTool-Using AgentsLLM Agent Reliability

  2. AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

    Jul 13, 2026Aritra Mazumder, Nusrat jahan LiaAI Agent ReliabilityLLM Agent Evaluation