Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse
Organizations: AI Tech Ethics
Abstract
Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research have studied task success and task completion of such agentic systems, research on agent to tool interaction chain via a wrapper and API, specifically in biological agentic workflows, remains limited. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful but some or all of the information or functionality from the tool via its API / wrapper is incomplete or missing and there are no communications to the user or the agent regarding the missing information. We call these 'silent failures' because the user or the agents are not aware that such a failure has occurred. For the purpose of this study, we developed an audit mechanism to identify such silent failures in agent to tool interaction chains, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within the ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus, characterising where the failure occurs in the chain. We observed 91 failures (manually validated post LLM based candidate discovery and automated testing), most frequent of them being missing data or fields and inconsistencies in search, filtering or ranking criteria. Most of the 91 failures occurred in API layer (52) or wrapper layer (25), with a potential of silent failure amplification downstream. The results show that silent failures originate upstream of the event and propagate downstream into apparently valid scientific outputs. We propose a concept of contextual reliability to handle such failures and suggest mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent-tool interaction pipeline.
Figures & tables
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Measure | Dimensionality | Key checks / verification requirements |
|---|---|---|
| 1. Completeness | Information, field, record, and qualifier coverage | Check whether all requested information is present; whether fields, records, references, qualifiers, alternatives, or pagination have been omitted; whether a response that claims completeness is actually complete; compare API/tool output wrapper output agent response. |
| 2. Correctness | Factual, identifier, logical, and semantic accuracy | Verify values against authoritative sources; check identifiers, mappings, calculations, and claims; check whether the agent confused entities or interpreted a tool field incorrectly; distinguish retrieved facts from inferred facts. |
| 3. Appropriateness | Tool, endpoint, and parameter selection; query construction; scope fit | Check whether the correct tool/endpoint was selected; whether the tool can answer the question; whether parameters and syntax are appropriate; whether a narrow tool was incorrectly used as a comprehensive source; whether fallback tools change the intended scope. |
| 4. Consistency | Repeatability; cross-query, cross-source, and temporal consistency | Run equivalent queries repeatedly; compare alternative formulations and reverse-direction queries; test ordering, batching, and namespace transformations; determine whether differences are explainable by source semantics or execution conditions. |
| 5. Reliability | Execution stability, error handling, recovery, and dependency behavior | Check for timeouts, errors, empty results, truncation, and partial responses; determine whether failures are surfaced; test whether retries/fallbacks preserve scope; check whether apparent success represents failed or partial execution. |
| 6. Attribution | Source and evidence provenance, citation completeness, and traceability | Check whether claims can be traced to sources; whether identifiers, references, evidence codes, and source records survive; whether the tool wrapper agent chain can be reconstructed; whether citations support the specific claims. |
| Dimension | Associated Failure Loci (indicative) | Instances |
|---|---|---|
| Completeness | L7, L5, L3 | 1. ClinicalTrials.gov (L7, CTG_TC_02): Agent misformulated an input query leading to missing information on FDAAA 801 violations. 2. AMPSphere (L5, AMPS_TC_01): Absence of co-prediction scores available in the API baseline. 3. BIGG Models (L3, BIGG_TC_7): The BiGG web search interface provides a functional ’multistrain=off’ filter, but this capability is missing from the API endpoints and documentation. |
| Interpretation Variabilities | L7, L5 | 1. KEGG (L7, KEGG_TC_15): Agent correctly parses OR-logic within a module group but misapplies the AND-completeness rule across blocks. 2. BV-BRC (L5, BVBRC_TC_4): API returns the critical quality metrics (’checkm_completeness’, ’checkm_contamination’, ’genome_quality’), as confirmed by the tool_api_response. ’BVBRC_get_genome’ wrapper fails to include these fields in its return schema – omits them from the actual data returned to the agent. 3. KEGG (L7, KEGG_TC_15): The agent correctly identifies the boolean ’OR’ behavior of commas in the module definition but misinterprets the module completeness rule by stating that missing a single K-number within a group does not compromise functionality, ignoring the fact that module completeness requires at least one present K-number for every parenthetical group (step) in the module definition. This is a misinterpretation by the agent of the logic provided by the wrapper/API. |
| Result Appropriateness | L7, L5, L3 | 1. ChIP-Atlas (L7, CHIPATLAS_TC_08): Agent misreads MACS2’s -10 log10(Q-value) score as a raw/linear magnitude. 2. AMPSphere (L5, AMPS_TC_04): Undocumented whether exact-match search surfaces homology-based hits from the separate MMseqs/HMMER endpoints. 3. Expression Atlas (L3, GXA_TC_10): Missing data description critical for inferring values: The gene expression value is returned without mentioning the unit (TPM/FKPM). |
| Source and Data Attribution | L3, L1, L5 | 1. BiGG Models (L3, BIGG_TC_04): PMID reference fields exist in schema but are empty for specific named models. 2. AMPSphere (L1, AMPS_TC_08): "Progenomes" — a genome database source, not an environmental location — is incorrectly classified as a habitat value alongside genuine habitats like "human gut." 3. BV-BRC (L5, BVBRC_TC_03): Data fields like assembly_accession, bioproject_accession and p2_genome_id are missing from the Wrapper. |
| Ranking and Relevance | L5, L3 | 1. AMPSphere (L5, AMPS_TC_04): MMseqs2/HMMER endpoints are present in the API baseline but missing from the wrapper. 2. ClinicalTrials.gov (L3, CTG_TC_10): Web UI defaults to "Relevance" sort; API default sort order differs/is unclear. 3. EMDB (L5, EMDB_TC_09): The API implementation enforces an undocumented or inconsistent result cap (e.g., 100 or 1000) that does not match the needs of a repository with 61,000+ entries, preventing effective data harvesting. |
| Reliability of Information | L5, L3 | 1. Expression Atlas (L3, GXA_TC_05): Accession-naming drift between E-MTAB and E-GTEX conventions over time creates evolutionary reliability risk for long-running studies. 2. BV-BRC (L5, BVBRC_TC_12): Legacy PATRIC-era p2_genome_id retained for cross-walks may drift from current BV-BRC taxonomy. 3. InterPro (L3, IPR_TC_09): The API (and thus the wrapper) lacks transparency regarding the underlying InterProScan version (5 or 6) which affects the fundamental reproducibility and meaning of sequence match data. |
Agent Failure Analysis
Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.
See all 153 papers