cs.AISep 21, 2026

Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

Authors: Devansh Singh, Shreya Gopalan, Sundaraparipurnan Narayanan

Organizations: AI Tech Ethics

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research have studied task success and task completion of such agentic systems, research on agent to tool interaction chain via a wrapper and API, specifically in biological agentic workflows, remains limited. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful but some or all of the information or functionality from the tool via its API / wrapper is incomplete or missing and there are no communications to the user or the agent regarding the missing information. We call these 'silent failures' because the user or the agents are not aware that such a failure has occurred. For the purpose of this study, we developed an audit mechanism to identify such silent failures in agent to tool interaction chains, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within the ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus, characterising where the failure occurs in the chain. We observed 91 failures (manually validated post LLM based candidate discovery and automated testing), most frequent of them being missing data or fields and inconsistencies in search, filtering or ranking criteria. Most of the 91 failures occurred in API layer (52) or wrapper layer (25), with a potential of silent failure amplification downstream. The results show that silent failures originate upstream of the event and propagate downstream into apparently valid scientific outputs. We propose a concept of contextual reliability to handle such failures and suggest mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent-tool interaction pipeline.

Figures & tables

Figure 2: Flowchart of experimental methodology.
Figure 3: (A) Distribution of grades across each manually-validated tool. ’Failed’ cases are confirmed failures while the issue was not reproduced in ’False Positive’ cases. (B) Distribution of failures across each failure locus (failure locus of 3 cases could not be determined).
Figure 4: (A) Issue occurrences across the analysis dimensions. Individual test cases may contribute to more than one dimension. (B) Observed types of issues. Individual test cases may exhibit more than one issue.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

MeasureDimensionalityKey checks / verification requirements
1. CompletenessInformation, field, record, and qualifier coverageCheck whether all requested information is present; whether fields, records, references, qualifiers, alternatives, or pagination have been omitted; whether a response that claims completeness is actually complete; compare API/tool output →\rightarrow wrapper output →\rightarrow agent response.
2. CorrectnessFactual, identifier, logical, and semantic accuracyVerify values against authoritative sources; check identifiers, mappings, calculations, and claims; check whether the agent confused entities or interpreted a tool field incorrectly; distinguish retrieved facts from inferred facts.
3. AppropriatenessTool, endpoint, and parameter selection; query construction; scope fitCheck whether the correct tool/endpoint was selected; whether the tool can answer the question; whether parameters and syntax are appropriate; whether a narrow tool was incorrectly used as a comprehensive source; whether fallback tools change the intended scope.
4. ConsistencyRepeatability; cross-query, cross-source, and temporal consistencyRun equivalent queries repeatedly; compare alternative formulations and reverse-direction queries; test ordering, batching, and namespace transformations; determine whether differences are explainable by source semantics or execution conditions.
5. ReliabilityExecution stability, error handling, recovery, and dependency behaviorCheck for timeouts, errors, empty results, truncation, and partial responses; determine whether failures are surfaced; test whether retries/fallbacks preserve scope; check whether apparent success represents failed or partial execution.
6. AttributionSource and evidence provenance, citation completeness, and traceabilityCheck whether claims can be traced to sources; whether identifiers, references, evidence codes, and source records survive; whether the tool →\rightarrow wrapper →\rightarrow agent chain can be reconstructed; whether citations support the specific claims.
Appendix
Table 1: Dimensional classification of failures and rubrics.
DimensionAssociated Failure Loci (indicative)Instances
CompletenessL7, L5, L31. ClinicalTrials.gov (L7, CTG_TC_02): Agent misformulated an input query leading to missing information on FDAAA 801 violations. 2. AMPSphere (L5, AMPS_TC_01): Absence of co-prediction scores available in the API baseline. 3. BIGG Models (L3, BIGG_TC_7): The BiGG web search interface provides a functional ’multistrain=off’ filter, but this capability is missing from the API endpoints and documentation.
Interpretation VariabilitiesL7, L51. KEGG (L7, KEGG_TC_15): Agent correctly parses OR-logic within a module group but misapplies the AND-completeness rule across blocks. 2. BV-BRC (L5, BVBRC_TC_4): API returns the critical quality metrics (’checkm_completeness’, ’checkm_contamination’, ’genome_quality’), as confirmed by the tool_api_response. ’BVBRC_get_genome’ wrapper fails to include these fields in its return schema – omits them from the actual data returned to the agent. 3. KEGG (L7, KEGG_TC_15): The agent correctly identifies the boolean ’OR’ behavior of commas in the module definition but misinterprets the module completeness rule by stating that missing a single K-number within a group does not compromise functionality, ignoring the fact that module completeness requires at least one present K-number for every parenthetical group (step) in the module definition. This is a misinterpretation by the agent of the logic provided by the wrapper/API.
Result AppropriatenessL7, L5, L31. ChIP-Atlas (L7, CHIPATLAS_TC_08): Agent misreads MACS2’s -10 ×\times log10(Q-value) score as a raw/linear magnitude. 2. AMPSphere (L5, AMPS_TC_04): Undocumented whether exact-match search surfaces homology-based hits from the separate MMseqs/HMMER endpoints. 3. Expression Atlas (L3, GXA_TC_10): Missing data description critical for inferring values: The gene expression value is returned without mentioning the unit (TPM/FKPM).
Source and Data AttributionL3, L1, L51. BiGG Models (L3, BIGG_TC_04): PMID reference fields exist in schema but are empty for specific named models. 2. AMPSphere (L1, AMPS_TC_08): "Progenomes" — a genome database source, not an environmental location — is incorrectly classified as a habitat value alongside genuine habitats like "human gut." 3. BV-BRC (L5, BVBRC_TC_03): Data fields like assembly_accession, bioproject_accession and p2_genome_id are missing from the Wrapper.
Ranking and RelevanceL5, L31. AMPSphere (L5, AMPS_TC_04): MMseqs2/HMMER endpoints are present in the API baseline but missing from the wrapper. 2. ClinicalTrials.gov (L3, CTG_TC_10): Web UI defaults to "Relevance" sort; API default sort order differs/is unclear. 3. EMDB (L5, EMDB_TC_09): The API implementation enforces an undocumented or inconsistent result cap (e.g., 100 or 1000) that does not match the needs of a repository with 61,000+ entries, preventing effective data harvesting.
Reliability of InformationL5, L31. Expression Atlas (L3, GXA_TC_05): Accession-naming drift between E-MTAB and E-GTEX conventions over time creates evolutionary reliability risk for long-running studies. 2. BV-BRC (L5, BVBRC_TC_12): Legacy PATRIC-era p2_genome_id retained for cross-walks may drift from current BV-BRC taxonomy. 3. InterPro (L3, IPR_TC_09): The API (and thus the wrapper) lacks transparency regarding the underlying InterProScan version (5 or 6) which affects the fundamental reproducibility and meaning of sequence match data.
Appendix
Table 2: Dimensions, associated failure loci, and representative instances.
More in this topic

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 153 papers

Explore similar work

CardsList
Aug 24, 2026

Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses

Tool-using agents increasingly rely on external tools to complete multi-step tasks, but tool returns can fail in different ways and require different recovery actions. Existing robustness studies often use uncertainty-based measures to detect when an agent becomes unreliable. These measures can reveal that something has gone wrong, but they do not directly identify the type of tool failure or the appropriate response. We address this limitation by analyzing tool failures at the moment a return enters the agent context. Our approach combines two complementary signals. The first compares the likelihood of the returned content under the tool schema and under the full trajectory prefix. The second measures the agent's probability distribution over its legal next actions. We evaluate the approach by injecting incomplete and inconsistent returns into a retail customer-service benchmark. The results show that likelihood-based signals clearly capture incomplete returns and some direct inconsistencies, while action-based signals reveal how strongly a failure changes the next decision. Some failures that are weak under likelihood signals can still redirect the agent toward state-changing actions. These findings show that tool failures can be recognized at the return boundary, but reliable diagnosis requires combining multiple signals.

Jiachen Xu, Torben Bach Pedersen, Zhongming Yao +2

Department of Computer Science Aalborg University Aalborg, Denmark · College of Computer Science and Technology Zhejiang University Hangzhou, China · School of Information Science and Engineering Northeastern University Shenyang, China

Agent Failure AnalysisAI Agent ReliabilityTool-Using AgentsLLM Agent Reliability
AskPDF
Oct 20, 2026

RITE: Evaluating End-to-End Resilience to Silent Tool Errors in E-Commerce Agentic Systems

Tool-using LLM agents increasingly answer e-commerce queries through multi-step tool interactions. The expected output of a tool call can shift between agent development and execution, allowing a plausible but incorrect value to be returned without an explicit failure signal. These silent errors may propagate through subsequent decisions into final answers. To study agents' resilience across this end-to-end process, we formulate Silent-error Resilience Task, an interactive task that evaluates whether they complete the original query after a silent error corrupts an intermediate state. We introduce RITE, a benchmark built from human-authored industrial workflows and production-log-derived inputs. To enable controlled and reproducible evaluation, we use a semantic error taxonomy to inject errors into fixed workflow prefixes, freeze recorded tool behavior, and let agents continue freely. It contains 167 matched units across nine e-commerce workflows, yielding 501 silent-error, correct-value, and explicit-failure instances. Across 10 recent LLMs, silent errors roughly halve task success and are consistently harder than explicit failures. Recognition, use of an available verification path, and integration of the resulting evidence form successive bottlenecks. We also propose a light State Checker as an intervention to test whether resilience can be improved.

Wenshuo Zhang, Yuquan Lin, Yushi Sun +6

The Hong Kong University of Science and Technology · Tencent

LLM Agent EvaluationLLM Agent ReliabilityAI Agent BenchmarksE-Commerce
AskPDF
Sep 28, 2026

Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models

Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before generation, making post-failure claims directly auditable. FTA contains 100 tasks with deterministic failure traces spanning five failure families, a neutral control, and four user-pressure conditions, and evaluates unsupported claims alongside useful recovery. Across six models, three response policies, and 3,600 human-annotated responses, false-success rates are 22.8% under the baseline policy, 9.3% with a transparency instruction, and 0.8% with a structured evidence contract. Fabricated-detail rates decrease from 28.3% to 14.3% and 0.8%, while useful responses increase from 74.9% to 89.2% and 98.8%, respectively. The tested evidence-contract policy is associated with substantially lower post-failure reporting errors while useful-response rates remain high within this blocked-task benchmark.

Junru Zhu, Shiming Xie, Aime Lu Fan Chen +4

Independent Researcher · Ant Group · Tsinghua University +2

Agent Failure AnalysisAI Agent BenchmarksTool-Using AgentsLLM Agent Reliability
AskPDF