Tool-using LLM agents increasingly answer e-commerce queries through multi-step tool interactions. The expected output of a tool call can shift between agent development and execution, allowing a plausible but incorrect value to be returned without an explicit failure signal. These silent errors may propagate through subsequent decisions into final answers. To study agents' resilience across this end-to-end process, we formulate Silent-error Resilience Task, an interactive task that evaluates whether they complete the original query after a silent error corrupts an intermediate state. We introduce RITE, a benchmark built from human-authored industrial workflows and production-log-derived inputs. To enable controlled and reproducible evaluation, we use a semantic error taxonomy to inject errors into fixed workflow prefixes, freeze recorded tool behavior, and let agents continue freely. It contains 167 matched units across nine e-commerce workflows, yielding 501 silent-error, correct-value, and explicit-failure instances. Across 10 recent LLMs, silent errors roughly halve task success and are consistently harder than explicit failures. Recognition, use of an available verification path, and integration of the resulting evidence form successive bottlenecks. We also propose a light State Checker as an intervention to test whether resilience can be improved.
Figures & tables
Figure 1: Silent errors from build-time–runtime mismatch. Tool-side data or implementation changes can make a runtime result diverge from what the agent was built to expect while signaling no failure.
Figure 2: RITE benchmark construction and rollout pipeline. A workflow Wi and paired log-derived input xi are used to activate a side-effect-free workflow replica and reconstruct a clean trace τi . A task-relevant slot then yields a shared query and result oracle together with three matched prefixes (correct, explicit, and implicit). The workflow and clean trace define the fixed evaluation environment Ei , in which the evaluated LLM continues from each prefix to produce a complete rollout.
Noticed
RSR
Resilience
Category
Model
Implicit ↑
Explicit ↑
Gap
Correct ↑
Implicit ↑
Explicit ↑
Retention ↑
Paired gap
Closed frontier
Claude-Opus-4.8
0.336
0.579
0.243
0.765
0.403
0.507
0.526
+0.089
GPT-5.5
0.280
0.594
0.314
0.716
0.363
0.526
0.507
+0.185
Open-weight frontier
Kimi-K3
0.263
0.590
0.327
0.784
0.302
0.555
0.385
+0.228
GLM-5.2
0.217
0.487
0.270
0.805
0.277
0.507
0.345
+0.243
Hy3
0.215
0.494
0.278
0.806
0.310
0.552
0.385
+0.236
Table 1: Results across the three matched conditions, grouped by model class and ordered within each group by implicit-condition Noticed. The Noticed gap is explicit minus implicit. Bold marks the best value in each column.
Noticed
n
Gold
Polluted
Other
No
1198
0.3%
99.6%
0.1%
Yes
377
23.3%
57.8%
18.8%
Table 2: Latest valid producer observation by Noticed in the implicit condition.
Table 3: E-commerce task-output impact and result robustness.
Table 4: Damage localization. Slots are paired units; Not. and RSR are measured in the implicit condition.
Figure 3: (a) State Checker adds a reliability check before a tool result is used. (b) Noticed, producer-refetch, and implicit-condition RSR with/without State Checker on paired implicit slots.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Task abstraction
Log-derived execution context
Graph size
Final units
wf01
Product recommendation from user needs, creator profiles, and selection history
Anonymized viewer, creator, and request context. A task query is constructed from the available profile context
55 / 14
11
wf02
Live-assistant scheduling and intervention selection
Anonymized session, creator, task, and configuration context
58 / 11
22
wf04
Pre-session script generation from creator and audience context
Anonymized creator identity. Profiles and historical content are retrieved by read-only nodes
22 / 11
15
wf05
Real-time product-presentation coaching
Anonymized session, creator, task, and available configuration context
47 / 12
17
wf06
Cross-domain intent and interest modeling
An anonymized user identity passing data-availability screening. Behavioral and profile records are retrieved downstream
52 / 14
52
wf07
Post-session summary over content, interaction, and commerce signals
Anonymized session, creator, request, and a session-derived time window
56 / 10
15
Appendix
Table 5: The nine golden workflows used by RITE. Log-derived inputs are described by semantic role rather than internal field names. Graph size is reported as total nodes / static tool-model interfaces.
ID
Leaf
Operational definition
Primary boundary
L1.1
category_sibling_drift
Replace a category with a nearby category at the same hierarchy level.
Parent–child substitutions belong to L1.2.
L1.2
category_granularity_mismatch
Replace a category with an ancestor or descendant, making its scope too broad or too narrow.
Same-level substitutions belong to L1.1.
L1.3
entity_near_duplicate
Replace the intended entity with another real, similar entity from the candidate domain.
If entity identity is preserved and only an attribute changes, use L2.1.
L2.1
profile_attribute_drift
Preserve entity identity but replace a profile, preference, label, or other attribute with another valid value.
Whole-entity replacement belongs to L1.3.
L2.2
numeric_bucket_shift
Move a numeric value or interval to a neighboring valid range.
Malformed, impossible, or plainly out-of-range values are explicit failures.
L2.3
semantic_synonym_drift
Substitute text that appears semantically close but changes the business interpretation used downstream.
Meaning-preserving paraphrases are not errors.
Appendix
Table 6: Operational definitions and boundaries of the seven silent-error leaves.
Measure
Evaluated object
Operational interpretation
Producer refetch
Post-prefix tool actions
Whether the agent re-invokes the producer of the affected value, regardless of what the new call returns.
Noticed
Reasoning plus remediation
Whether the agent explicitly identifies a related anomaly and follows it with a remedial action.
Observation state
Latest valid producer observation
Recovered matches gold, Residual matches the polluted value, and Neither matches neither reference value.
RSR
Complete final answer
Whether the answer resolves the original query using the available evidence; it is independent of observation-state attribution.
Appendix
Table 7: Objects evaluated by the process measures and judge outputs. These quantities are complementary rather than successive scoring rules.
Label
Majority agreement
Cohen’s κ
Human Fleiss’ κ
Precision
Recall
F1
Review agreement
Result
76/100 (76%)
0.318
0.477
97.1%
75.3%
84.8%
252/300 (84%)
Noticed
87/100 (87%)
0.739
0.569
87.0%
85.1%
86.0%
267/300 (89%)
Appendix
Table 8: Human validation of the automatic Result and Noticed labels over 100 rollouts. Each rollout received three independent annotations. Blind-stage statistics provide the primary validation; post-reveal agreement is complementary.
Figure 4: English translation of the original Chinese two-stage annotation interface used in the human-validation study. Blind labels were locked before the automatic evaluation labels and rationales were revealed.
Figure 5: Distribution of the 167 paired units. Left : units contributed by each workflow. Right : units per silent-error type; blue bars denote object or category errors and orange bars denote attribute or value errors. Numeric labels give exact counts, and temporal staleness has no retained instance.
Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a 2×2 taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale 3.66× slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.
Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5
Shanghai AI Laboratory · East China Normal University · Soochow University +2
Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyond text generation. However, clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call. We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents, where a tool-calling agent is an LLM system that selects a tool, supplies structured arguments, and interprets its returned feedback. ToolRobustBench aligns four perturbation families with the tool-use pipeline: tool-interface, user-intent, tool-output/observation, and runtime-environment perturbations. It attributes failures to tool selection, schema grounding, argument binding, tool-output/runtime-feedback handling, and E2E task success. Experiments on 15,456 single-family instances across 7 models, 16 sampled local tools, 4 perturbation families, and 14 subtypes show high but non-uniform clean performance and substantial robustness degradation, with tool-output/observation perturbation the dominant bottleneck. Mixed-family experiments reveal non-additive failure patterns that are not explained by isolated single-family results. Thus, ToolRobustBench provides a deterministic and cascade-aware benchmark for diagnosing robustness beyond clean tool-calling accuracy;
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.