Tool-using LLM agents increasingly answer e-commerce queries through multi-step tool interactions. The expected output of a tool call can shift between agent development and execution, allowing a plausible but incorrect value to be returned without an explicit failure signal. These silent errors may propagate through subsequent decisions into final answers. To study agents' resilience across this end-to-end process, we formulate Silent-error Resilience Task, an interactive task that evaluates whether they complete the original query after a silent error corrupts an intermediate state. We introduce RITE, a benchmark built from human-authored industrial workflows and production-log-derived inputs. To enable controlled and reproducible evaluation, we use a semantic error taxonomy to inject errors into fixed workflow prefixes, freeze recorded tool behavior, and let agents continue freely. It contains 167 matched units across nine e-commerce workflows, yielding 501 silent-error, correct-value, and explicit-failure instances. Across 10 recent LLMs, silent errors roughly halve task success and are consistently harder than explicit failures. Recognition, use of an available verification path, and integration of the resulting evidence form successive bottlenecks. We also propose a light State Checker as an intervention to test whether resilience can be improved.
Figures & tables
Figure 1: Silent errors from build-time–runtime mismatch. Tool-side data or implementation changes can make a runtime result diverge from what the agent was built to expect while signaling no failure.
Figure 2: RITE benchmark construction and rollout pipeline. A workflow Wi and paired log-derived input xi are used to activate a side-effect-free workflow replica and reconstruct a clean trace τi . A task-relevant slot then yields a shared query and result oracle together with three matched prefixes (correct, explicit, and implicit). The workflow and clean trace define the fixed evaluation environment Ei , in which the evaluated LLM continues from each prefix to produce a complete rollout.
Noticed
RSR
Resilience
Category
Model
Implicit ↑
Explicit ↑
Gap
Correct ↑
Implicit ↑
Explicit ↑
Retention ↑
Paired gap
Closed frontier
Claude-Opus-4.8
0.336
0.579
0.243
0.765
0.403
0.507
0.526
+0.089
GPT-5.5
0.280
0.594
0.314
0.716
0.363
0.526
0.507
+0.185
Open-weight frontier
Kimi-K3
0.263
0.590
0.327
0.784
0.302
0.555
0.385
+0.228
GLM-5.2
0.217
0.487
0.270
0.805
0.277
0.507
0.345
+0.243
Hy3
0.215
0.494
0.278
0.806
0.310
0.552
0.385
+0.236
Table 1: Results across the three matched conditions, grouped by model class and ordered within each group by implicit-condition Noticed. The Noticed gap is explicit minus implicit. Bold marks the best value in each column.
Noticed
n
Gold
Polluted
Other
No
1198
0.3%
99.6%
0.1%
Yes
377
23.3%
57.8%
18.8%
Table 2: Latest valid producer observation by Noticed in the implicit condition.
Table 3: E-commerce task-output impact and result robustness.
Table 4: Damage localization. Slots are paired units; Not. and RSR are measured in the implicit condition.
Figure 3: (a) State Checker adds a reliability check before a tool result is used. (b) Noticed, producer-refetch, and implicit-condition RSR with/without State Checker on paired implicit slots.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Task abstraction
Log-derived execution context
Graph size
Final units
wf01
Product recommendation from user needs, creator profiles, and selection history
Anonymized viewer, creator, and request context. A task query is constructed from the available profile context
55 / 14
11
wf02
Live-assistant scheduling and intervention selection
Anonymized session, creator, task, and configuration context
58 / 11
22
wf04
Pre-session script generation from creator and audience context
Anonymized creator identity. Profiles and historical content are retrieved by read-only nodes
22 / 11
15
wf05
Real-time product-presentation coaching
Anonymized session, creator, task, and available configuration context
47 / 12
17
wf06
Cross-domain intent and interest modeling
An anonymized user identity passing data-availability screening. Behavioral and profile records are retrieved downstream
52 / 14
52
wf07
Post-session summary over content, interaction, and commerce signals
Anonymized session, creator, request, and a session-derived time window
56 / 10
15
Appendix
Table 5: The nine golden workflows used by RITE. Log-derived inputs are described by semantic role rather than internal field names. Graph size is reported as total nodes / static tool-model interfaces.
ID
Leaf
Operational definition
Primary boundary
L1.1
category_sibling_drift
Replace a category with a nearby category at the same hierarchy level.
Parent–child substitutions belong to L1.2.
L1.2
category_granularity_mismatch
Replace a category with an ancestor or descendant, making its scope too broad or too narrow.
Same-level substitutions belong to L1.1.
L1.3
entity_near_duplicate
Replace the intended entity with another real, similar entity from the candidate domain.
If entity identity is preserved and only an attribute changes, use L2.1.
L2.1
profile_attribute_drift
Preserve entity identity but replace a profile, preference, label, or other attribute with another valid value.
Whole-entity replacement belongs to L1.3.
L2.2
numeric_bucket_shift
Move a numeric value or interval to a neighboring valid range.
Malformed, impossible, or plainly out-of-range values are explicit failures.
L2.3
semantic_synonym_drift
Substitute text that appears semantically close but changes the business interpretation used downstream.
Meaning-preserving paraphrases are not errors.
Appendix
Table 6: Operational definitions and boundaries of the seven silent-error leaves.
Measure
Evaluated object
Operational interpretation
Producer refetch
Post-prefix tool actions
Whether the agent re-invokes the producer of the affected value, regardless of what the new call returns.
Noticed
Reasoning plus remediation
Whether the agent explicitly identifies a related anomaly and follows it with a remedial action.
Observation state
Latest valid producer observation
Recovered matches gold, Residual matches the polluted value, and Neither matches neither reference value.
RSR
Complete final answer
Whether the answer resolves the original query using the available evidence; it is independent of observation-state attribution.
Appendix
Table 7: Objects evaluated by the process measures and judge outputs. These quantities are complementary rather than successive scoring rules.
Label
Majority agreement
Cohen’s κ
Human Fleiss’ κ
Precision
Recall
F1
Review agreement
Result
76/100 (76%)
0.318
0.477
97.1%
75.3%
84.8%
252/300 (84%)
Noticed
87/100 (87%)
0.739
0.569
87.0%
85.1%
86.0%
267/300 (89%)
Appendix
Table 8: Human validation of the automatic Result and Noticed labels over 100 rollouts. Each rollout received three independent annotations. Blind-stage statistics provide the primary validation; post-reveal agreement is complementary.
Figure 4: English translation of the original Chinese two-stage annotation interface used in the human-validation study. Blind labels were locked before the automatic evaluation labels and rationales were revealed.
Figure 5: Distribution of the 167 paired units. Left : units contributed by each workflow. Right : units per silent-error type; blue bars denote object or category errors and orange bars denote attribute or value errors. Numeric labels give exact counts, and temporal staleness has no retained instance.