AI systems fail silently far more often than they fail visibly. In an analysis of 100K human-AI interactions from the WildChat dataset, we find that 79% of AI failures are invisible: something went wrong but the user gave no overt indication that there was a problem. These invisible failures cluster into eight archetypes that help us characterize where and how AI systems are failing to meet users' needs. In addition, the archetypes show systematic co-occurrence patterns indicating higher-level failure types. To address the question of whether these archetypes will remain relevant as AI systems become more capable, we also created and annotated a counterfactual dataset in which WildChat's 2024-era responses are replaced by those from three present-day frontier LMs. This analysis indicates that failure rates have dropped substantially, but that the vast majority of failures remain invisible in our sense, and the distribution of failure archetypes seems stable. Finally, we illustrate how the archetypes help us to identify systematic and variable AI limitations across different usage domains. Overall, we argue that our invisible failure taxonomy can be a key component in reliable failure monitoring for product developers, scientists, and policy makers. Our code and data are available at https://github.com/bigspinai/bigspin-invisible-failure-archetypes
Figures & tables
Table 1: Annotation project results, based on a sample of 10K transcripts. The annotators are Opus 4.6 and GPT-5.4 . Micro- κ gives the global Cohen κ across all categories, Macro- κ is the mean of all the category-level κ values, and Agree. is the overall agreement rate between the two annotators. Appendix C gives complete per-category breakdowns. We chose the clearly superior ‘Signals-only’ approach, for which the annotators disagree on only 6% of cases.
Figure 1: Invisible failure archetype distribution. The bar sizes and counts indicate the frequency of the archetype. Since individual transcripts can manifest multiple archetypes, the percentages are the percent of failure transcripts labeled with that archetype.
Figure 2: Failure persistence in Future-2K . The GPT-4 cases are the original WildChat transcripts, included for comparison. Error rates for the newer models are substantially lower (‘none’ means ‘no failure detected’), but the vast majority of failures continue to be invisible.
Figure 3: Archetype distribution in Future-2K . The y-axis retains the frequency ordering of fig. 1 for comparison. The overall distribution is quite similar across models; many of the infrequent or unattested archetypes have almost no chance of occurring in the single-turn examples that make up Future-2K .
Original
Sonnet
Δ (pp)
% Eliminated
n
%
n
%
off_topic_drift
74
3.7
0
0.0
−3.7
100
plow_through
7
0.4
0
0.0
−0.4
100
repetition
7
0.4
0
0.0
−0.4
100
performative_hedge
84
4.2
3
0.2
−4.0
96
conversation_stalled
167
8.3
8
0.4
−7.9
95
Table 2: Problem signals most changed between GPT-4 and Sonnet. We restrict attention to signals with at least 5 occurrences in Future-2K . The top of the table shows the most reduced problem tags, and the bottom shows the most persistent problem tags.
Figure 4: Archetype–domain co-occurrence. The cell values give PPMI values, with darker blues indicating stronger associations. Different domains have different archetype associations, pointing to different underlying challenges for AI systems.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Failure annotation agreement reports.
Table 4: Archetype annotation agreement reports.
Table 5: Signal annotation agreement reports.
Figure 5: Distribution of domains.
Figure 6: Turn distribution in our sample of WildChat.
Figure 7: Archetype distribution for multi-turn interactions. The y-axis ordering corresponds to the frequency ordering in fig. 1 , to make it easier to track how the prevalence is different in this subset of cases. Though the rate of invisible failures is somewhat lower in this sample, they still account for the majority of failures (50% vs. 79% for the full dataset), and all the archetypes are still attested.
Figure 8: Archetype co-occurrence analyses.
Figure 9: Additional views on the domain–archetype relationship.
LLM agent systems increasingly run as long-lived autonomous runtimes: scheduling jobs, calling tools, maintaining memory, and pushing results to humans. We present a longitudinal study of silent failures in one such system: a personal-assistant agent runtime in continuous production since March 2026, with roughly 40 scheduled jobs, 8 LLM providers, a tool-governance proxy, and a knowledge-base memory plane, defended by 4,286 unit tests and 827 governance checks. Over eight weeks we documented 22 incidents with full root-cause postmortems, in which one meta-pattern -- a failure whose error signal never reaches a human in actionable form -- manifested at least 28 times. We derive a five-class, mechanism-oriented taxonomy: (A) environment and platform quirks, (B) design-assumption mismatches, (C) error swallowing and dilution, (D) chained hallucination and fabrication, (E) operational omission and forensic blind spots. Class D is unique to LLM systems and the most dangerous: the system does not merely fail to report an error -- the LLM transforms it into fluent, plausible narrative delivered to the user. We term this fail-plausible: gray failure's differential observability escalated -- the observer is not just blind, it is convincingly lied to by the failure itself. Three findings: about 70% of silent failures were caught by human user-view observation, not tests or audits; a retrospective audit of 15 incidents found 0% ex-ante prevention but 87% regression blocking -- audits are regression engines, not prediction engines; incident latency (13 hours to 60 days) tracks failure mechanism, not code complexity -- the longest-lived failures lived in the seams between components, where no test runs. We describe the resulting defense framework and distill design principles for agent systems whose failures are loud, attributable, and boring. All postmortems and artifacts are public.
Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research have studied task success and task completion of such agentic systems, research on agent to tool interaction chain via a wrapper and API, specifically in biological agentic workflows, remains limited. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful but some or all of the information or functionality from the tool via its API / wrapper is incomplete or missing and there are no communications to the user or the agent regarding the missing information. We call these 'silent failures' because the user or the agents are not aware that such a failure has occurred. For the purpose of this study, we developed an audit mechanism to identify such silent failures in agent to tool interaction chains, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within the ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus, characterising where the failure occurs in the chain. We observed 91 failures (manually validated post LLM based candidate discovery and automated testing), most frequent of them being missing data or fields and inconsistencies in search, filtering or ranking criteria. Most of the 91 failures occurred in API layer (52) or wrapper layer (25), with a potential of silent failure amplification downstream. The results show that silent failures originate upstream of the event and propagate downstream into apparently valid scientific outputs. We propose a concept of contextual reliability to handle such failures and suggest mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent-tool interaction pipeline.
AI systems can fail silently. The failure propagates through training loops, evaluation pipelines, and production monitoring stacks until downstream harm makes it visible. This paper introduces evaluation blindness: a measurement function M exhibits evaluation blindness with respect to failure class F when it produces readings indistinguishable from a healthy state while the system is actually failing, with no auxiliary signal flagging the gap. The problem surfaces at two lifecycle stages the literature has treated separately. At training time, reward models are gamed, importance-sampling corrections are silently miscalculated, and benchmark contamination inflates fine-tuning evaluations, all while loss curves look healthy and gradient updates proceed normally. At deployment time, monitoring fails to catch six classes of production failure, including an Operational category that is 100% silent by structural definition. We provide a formal detectability predicate unifying both stages. Four training-time case studies trace concrete breakdowns, including a real implementation bug in TRL PR #6594 where gradients are corrupted as loss decreases normally. A six-class taxonomy validated against 50 real-world incidents from court documents and regulatory filings finds that 53% of verifiable public failures were silent. A failure budget framework ties acceptable failure rates to use-case risk class. The implication is direct: measurement infrastructure is a correctness concern across the full AI lifecycle, not just at evaluation time. Data, code, and taxonomy schema are at https://github.com/priyanka25aug/llm-failure-taxonomy.