Agentic workflows combine large language models with planning, memory, tools, execution environments, and iterative feedback. Their failures are difficult to diagnose because the same user-visible symptom may originate from model behavior, workflow orchestration, external tools, infrastructure, or their interaction. This paper presents a source-aware taxonomy of agentic workflow failures and applies it to evidence from academic literature and GitHub. The taxonomy combines source attribution with three empirical dimensions: a concrete failure-type dictionary, workflow-stage localization, and agent-capability localization. Its central structure is a two-axis localization matrix: workflow stages form the horizontal process axis and agent capabilities form the vertical functional axis. This design moves beyond taxonomies that mainly classify final failure manifestations by localizing intermediate process states inside agent operation. Our empirical comparison shows that literature-derived failures emphasize planning, reasoning, learning, perception, and knowledge-related mechanisms, whereas GitHub-derived failures concentrate on execution-stage, actuation-related, tool, API, and infrastructure failures. The dominant hotspot is iterative execution and actuation. These findings indicate that agentic workflow reliability should be treated as a systems problem rather than only a model-output problem. The taxonomy provides a basis for structured evaluation, root cause analysis, failure injection, and engineering triage.
Figures & tables
Fig. 1: Overview of the methodology. Literature evidence is used to derive the initial taxonomy, GitHub evidence is used to extend and stress-test it, and reliability controls are used to adjudicate the final coded dataset.
Protocol item
Specification
Databases and expansion
IEEE Xplore, ACM Digital Library, and arXiv; backward and forward snowballing from relevant studies.
Search string
( agent OR agentic OR LLM agent OR autonomous agent OR tool-use agent OR multi-agent OR workflow orchestration OR reflection ) AND ( failure OR fault OR error OR bug OR breakdown OR invalid output ).
Inclusion criteria
I1: studies or evaluates an agentic, tool-using, or multi-agent workflow; I2: reports failure phenomena, causes, symptoms, error patterns, or evaluation breakdowns; I3: provides evidence concrete enough to extract at least one failure episode.
Exclusion criteria
E1: traditional software defects without agentic workflow behavior; E2: pure model risks without workflow or tool-use context; E3: method-only papers with no identifiable failure evidence; E4: duplicate or superseded versions; E5: inaccessible full text.
Screening procedure
Title/abstract screening followed by full-text screening. Ambiguous records were retained until full-text inspection to avoid premature exclusion.
TABLE I: Literature retrieval and screening protocol.
Dataset
n
Source
Stage
Cap.
Type
GitHub
150
0.71
0.69
0.67
0.65
Literature
80
0.79
0.61
0.63
0.72
TABLE II: Independent human–human annotation agreement (Cohen’s κ ). The 50 additional non-failure steps are not included in these failure-label rows.
Control
Implementation
Source attribution
Classify each episode as Agentic, Non-Agentic, or Mixed according to whether the primary responsibility lies in agent behavior, external conditions, or their interaction.
Failure-type coding
Select the most specific dictionary type supported by evidence; use general labels only when finer attribution is unsupported.
Matrix localization
Map each episode to one workflow stage and one agent capability to capture its intermediate process state.
Tool support
Use fuzzy keyword matching, dictionary matching, regular expressions, and structured LLM suggestions only as candidate labels; final statistics use adjudicated labels.
Reliability check
Report full-dataset human–LLM agreement as auxiliary evidence and independent human–human checks on 150 GitHub failures and 80 literature records (with 50 additional non-failure steps reviewed).
TABLE III: Annotation and reliability controls.
Fig. 2: Full taxonomy of agentic-workflow failures with three source-attribution categories, 16 failure families, 65 operational failure types, and a stage–capability matrix. Each paired count is (literature + GitHub), totaling 397 + 1,196 failure records. The top-level counts are Agentic (271+618), Non-Agentic (100+568), and Mixed (26+10); literature attribution in these displays includes auxiliary labels and is not a fully human-adjudicated source distribution. Family stage–capability tags describe dominant localization, not an additional type label. Matrix cells show primary stage (S0–S4) by capability (A–F) for 1,586 valid records (397 literature; 1,189 GitHub); the seven GitHub records without complete matrix labels remain in type-level counts. A dash denotes zero; the largest cell is S3 × D (827).
Decision point
Rule
Example consequence
Earliest responsible stage
Prefer the evidenced stage where the defect was introduced; otherwise use its first observable manifestation, not merely the final user-visible symptom.
Goal misunderstanding leading to wrong output is S1; wrong final synthesis after correct work is S4.
Stage boundary
Separate task understanding (S1), planning-time choices (S2), runtime action/feedback/recovery (S3), and final response construction (S4).
Bad tool choice in a plan is S2; correct tool with malformed runtime arguments is S3.
Capability boundary
Separate decisions and inferences from concrete actions, and separate knowledge/memory from monitoring and recovery.
Wrong algorithm choice is C; malformed shell command is D; context loss is B; repeated failed retry is E.
Source attribution
Use Agentic for avoidable agent behavior, Non-Agentic for external conditions outside direct control, and Mixed only when both sides materially contribute or evidence is underdetermined.
API outage is Non-Agentic; wrong arguments are Agentic; malformed tool response plus no validation is Mixed.
Specificity
Use the most specific supported failure type; use AF00, NF00, or MF00 only when the source is clear but evidence does not support a finer subtype.
Revisit general labels when logs, comments, or fixes reveal a concrete mechanism.
TABLE IV: Condensed boundary rules for applying the taxonomy.
TABLE V: Largest source differences by failure type. Percentages are normalized within each source; gaps are GitHub percentage minus literature percentage.
Fig. 3: Comparison of literature-derived and GitHub-derived failures across workflow stages and agent capabilities. Labels indicate the GitHub-minus-literature percentage-point gap. The main marginal gaps are statistically reliable: the stage distribution differs by source, χ2(4)=348.92 , p<0.001 , V=0.469 , with S3 showing a +46.6 pp gap, 95% CI [+39.4,+53.2], adjusted Fisher p<0.001 ; the capability distribution also differs, χ2(5)=334.19 , p<0.001 , V=0.458 , with D: Actuation showing a +48.1 pp gap, 95% CI [+41.3,+54.1], adjusted Fisher p<0.001 .
Fig. 4: Stage–capability heatmaps for literature (left), GitHub (center), and the combined dataset (right). Each cell shows the number of failures localized to a workflow stage and capability pair.
Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices like end-to-end outcome checks and ad-hoc trace inspection systematically mask the intermediate failures that dominate real-world error budgets. We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs), where each node carries typed quality metrics assessed by a calibrated LLM judge (GPT-4o), classified through a hierarchical failure taxonomy (3 levels, 21 subcategories), and linked to upstream dependencies for automated root cause attribution. An ablation study isolates the impact of DAG-based dependency modeling: it alone contributes +22 percentage points to failure detection recall and +34 pp to root cause accuracy over flat step-level evaluation with identical judges and rubrics. Across three production workflows (450 test cases, two agent model families, predominantly sequential architectures with a 12% non-DAG trace rate), AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation (0.89 vs. 0.41), Cohen's kappa = 0.84 agreement with human experts, and 72% root cause accuracy against an 81% human ceiling. Cross-system evaluation on tau-bench and SWE-bench traces confirms transferability (failure detection recall >= 0.78) without taxonomy or rubric modification. A 4-month pilot with 18 engineers detected 23 pre-release regressions through CI/CD-integrated regression testing, reducing median root-cause identification time from 4.2 hours to 22 minutes and driving measurable failure rate reductions in two workflows.
Dongxin Guo, Jikun Wu, Siu Ming Yiu
The University of Hong Kong · Stellaris AI Limited
Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.