Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents

Obada Kraishan

cs.AIOct 7, 2026

Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents

Authors: Obada Kraishan

Organizations: College of Media and Communication Texas Tech University Lubbock, TX, USA

Abstract

Tool-using agents are usually scored on whether they finish a task while the tools work. Deployments are less forgiving: services time out, endpoints disappear, parameter names change, and results come back well formed but wrong. Prior work has shown that language models over-trust tool outputs that fail silently; we ask how that over-trust plays out across the stages of failure handling in multi-turn agents. Wrapping the executable environments of an established function-calling benchmark in a fault-injection layer, we inject one of four typed faults at a controlled point in the trajectory and record whether the agent notices, changes plan, recovers the task, or repeats itself. Six models from three families, half of them reasoning variants, ran 1,920 trials over 24 multi-step tasks. Agents treat a failure as a problem in 91.3% of trials when the tool returns an explicit error, but in 58.8% of trials when it returns a plausible wrong value, against a 26.8% rate of reporting problems when nothing was wrong. Reasoning models are not better placed: paired against instruct siblings, they notice less (-9.3 points, p < .001) and change plan more (+10.4 points, p < .001), and recovery is unchanged (p = .512). Because agents are stochastic, two fault-free runs of the same task end in the same state only 63.3% of the time; against that baseline, only a missing tool clearly lowers recovery (39.9%), while timeouts, schema drift, and corruption stay within run-to-run variation. After a fault, agents return to the same tool three or more times in a row in up to 22.2% of trials, though strictly identical repeats are rare. A prompt line asking the agent to check each result did not move detection. Agents respond to the error channel rather than to the content of what a tool returns, so failures that stay inside the expected format pass through.

Figures & tables

Study	Setting	Faults	Primary outcome
Sun et al. [ 26 ]	calculator; embodied planning	silent tool errors	detection when asked to accept or reject
PALADIN [ 27 ]	ToolBench	execution errors	recovery rate after training
ReliabilityBench [ 28 ]	four simulated domains	timeouts, rate limits, partial responses, schema drift	success under fault, by end state
ToolMaze [ 29 ]	multi-path tool tasks	explicit/implicit, transient/permanent	recovery rate under perturbation
This work	BFCL multi-turn	timeout, missing tool, schema drift, silent corruption	detection, replanning, recovery, repetition, separately

TABLE I: Studies that inject faults into the tool layer, by what they measure.

Condition	Visibility	Trials	Fired	Analysed
clean	baseline	384	–	384
timeout	loud	384	99%	379
missing tool	loud	384	71%	273
schema drift	loud	384	74%	284
silent corruption	quiet	384	97%	374

TABLE II: Conditions, visibility, and how often each fault fired.

Model	Family	Type	Trials	Prompt arm
claude-haiku-4.5-thinking	claude	reasoning	480	yes
claude-haiku-4.5	claude	instruct	480	yes
deepseek-r1	deepseek	reasoning	240	–
deepseek-v3	deepseek	instruct	240	–
qwen3-thinking	qwen	reasoning	240	–
qwen3-instruct	qwen	instruct	240	–

TABLE III: Model panel. Each family contributes a matched reasoning and instruct pair.

Fig. 1: Detection by failure visibility. Left: share of trials in which the agent treated the tool result as a problem, with task-clustered bootstrap intervals. Right: how those responses divide, where “none” means the result was used or reported as if it were fine.

Condition	$n$	Detected	95% CI
no fault (baseline)	306	26.8%	[17.8, 36.5]
quiet failure	374	58.8%	[45.7, 71.2]
loud failure	936	91.3%	[88.3, 94.1]
timeout	379	91.6%	[88.0, 94.5]
missing tool	273	91.2%	[86.5, 95.2]
schema drift	284	91.2%	[86.3, 95.4]

TABLE IV: Detection rates with task-clustered bootstrap intervals.

Fig. 2: Reasoning variants minus matched instruct siblings on fault trials, paired within family by task, fault, and seed. Points are mean differences in percentage points with task-clustered 95% bootstrap intervals; asterisks mark contrasts significant in task-clustered models after Holm correction. Noticing sits left of zero in every family while acting sits right of it, and recovery straddles the line.

Outcome	Family	Reasoning	Instruct	$\Delta$ (points)	Clustered 95% CI	OR	$p_{\text{Holm}}$
detection	claude	82.0%	88.2%	$-6.2$	$[-11.4,-1.2]$	0.61	.042
detection	deepseek	78.6%	82.5%	$-4.0$	$[-14.8,+5.0]$	0.78	.434
detection	qwen	67.5%	84.0%	$-16.6$	$[-27.2,-6.2]$	0.40	.003
replanning	claude	75.2%	70.8%	$+4.3$	$[-2.3,+11.8]$	1.25	.417
replanning	deepseek	74.6%	65.9%	$+8.7$	$[-5.5,+23.8]$	1.52	.417
replanning	qwen	92.6%	74.8%	$+17.8$	$[+11.3,+24.2]$	4.23	$<$ .001

TABLE V: Reasoning versus instruct siblings on fault trials, paired within family by task, fault, and seed.

Fig. 3: Consequences of a single fault. Left: share of trials reaching the state of an independent fault-free run of the same model and task, against the agreement of two fault-free runs (63.3%). Right: how often the agent called the same tool three or more times in a row after the fault, with the extra steps a fault costs relative to a clean run listed alongside.

Condition	Recovered	Same tool	Identical	Cap hit	Extra steps
clean	63.3%	–	–	23.2%	$+0.01$
timeout	59.6%	21.4%	1.1%	24.8%	$+0.53$
missing tool	39.9%	12.5%	1.1%	29.3%	$+1.47$
schema drift	59.2%	17.6%	1.8%	31.7%	$+1.85$
silent corruption	60.4%	22.2%	0.8%	25.9%	$+0.56$

TABLE VI: Recovery and effort by condition.

Explore similar work

Sep 28, 2026cs.AI

Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models

Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before generation, making post-failure claims directly auditable. FTA contains 100 tasks with deterministic failure traces spanning five failure families, a neutral control, and four user-pressure conditions, and evaluates unsupported claims alongside useful recovery. Across six models, three response policies, and 3,600 human-annotated responses, false-success rates are 22.8% under the baseline policy, 9.3% with a transparency instruction, and 0.8% with a structured evidence contract. Fabricated-detail rates decrease from 28.3% to 14.3% and 0.8%, while useful responses increase from 74.9% to 89.2% and 98.8%, respectively. The tested evidence-contract policy is associated with substantially lower post-failure reporting errors while useful-response rates remain high within this blocked-task benchmark.

Junru Zhu, Shiming Xie, Aime Lu Fan Chen +4

Independent Researcher · Ant Group · Tsinghua University +2

Latent Failure Patterns

Ask PDF

Sep 13, 2026cs.SE

Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return

Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one. We isolate this post-failure decision with a benchmark of 1,024 items spanning 16 internal-system domains and eight tool-failure types, in which a tool call is enforced and the returned payload is guaranteed to be unusable. Under a deployment-style system prompt, 14.10% of responses are dishonest: the model either asserts a value the payload cannot support or declines while citing a fabricated policy or capability limit. The rate is governed almost entirely by whether the failure is signalled. When the tool returns status:error, dishonesty is absent (0.0%); when it returns status:ok with a redacted, corrupted, stale, malformed, empty or truncated value, dishonesty reaches 45.3%. The behaviour is not an artefact of our prompts: it appears under a neutral prompt (10.17%) and under the shipped prompt of every production agent framework we evaluate, reaching 24.67% under CrewAI's, and none of the nine frameworks we audit specifies what the model should do when a tool fails. Comparing prompt-level defences, we find that the operative variable is not deference to tool output but the absence of a named failure state. Appending a single sentence that requires the model to emit retrieval_status: OK or FAILED before answering reduces dishonesty from 14.10% to 0.87%, with one item of 688 worsening against 92 improving, and transfers unchanged into three foreign agent scaffolds. The emitted flag is faithful in 99.7-99.9% of declarations, giving a runtime detector that needs only a regular expression.

Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi +3

The Shishukunj International School · Haileybury Astana · UWCSEA East Campus +2

Honesty Systemverilog Assertions

Ask PDF

Jun 4, 2026cs.AI

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a

2 \times 2

taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale

3.66\times

slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.

Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5

Shanghai AI Laboratory · East China Normal University · Soochow University +2

Large Language Model Agents Fault Tolerance Iterative Re-Planning

Ask PDF