XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights

Arun Joshi

cs.SEMar 6, 2026

XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights

Authors: Arun Joshi

Organizations: Independent Researcher Islington College (Alumini) Nepal

Abstract

Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows

Figures & tables

Category	Subcategory	Count
Planning Failure	Incorrect problem decomposition	1
Code Generation Failure	Logic errors (wrong implementation)	2
Testing/Validation Failure	Did not run validation tests	2
Understanding Failure	Misunderstood problem requirements	9
Iterative Refinement Failure	Exceeded iteration limit without progress	18

Table 1: Coding Agent Failure Taxonomy

Figure 1: System architecture showing the flow from raw trace to final explanation report. The system consists of three main components: automatic annotation, explanation generation, and report synthesis.

Metric	Raw	General Purpose LLMs	Our XAI
Technical Participants (n=10)
Time to understand (min)	8.4±2.1	5.2±1.3	3.0±0.8 **
Root cause accuracy (%)	42±15	68±12	89±8 **
Fix quality (1-5)	2.6±0.8	3.4±0.6	4.3±0.5 **
Confidence (1-7)	3.2±1.1	4.8±0.9	6.1±0.7 **
Non-Technical Participants (n=10)

Table 2: User Study Results (Mean ± SD)

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 152 papers

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Jul 10, 2026Xiangxin Zhao, Han Li, Shuaiting Li +4Agent Failure Analysis Coding Agents
University College London, London, United Kingdom · Nanjing University, Nanjing, China
Explaining AI Agents Through Execution Traces
Sep 5, 2026Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli +3Explainable Artificial Intelligence AI Agent Auditing
Sapienza University of Rome, Rome, Italy
CodeTracer: Towards Traceable Agent States
Apr 13, 2026Han Li, Yifan Yao, Letian Zhu +13AI Coding Agents Agent Trajectory Analysis
Nanjing University · Alibaba Group · Renmin University of China +3

Jul 10, 2026J/K move · Enter open · S save

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao+3

University College London, London, United Kingdom · Nanjing University, Nanjing, China

Agent Failure Analysis Coding Agents LLM Agent Reliability

Open paper AskPDF arXiv

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

Abstract

Agent Failure Analysis

Explore similar work

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Explaining AI Agents Through Execution Traces

CodeTracer: Towards Traceable Agent States