Authors: Han Li, Yifan Yao, Letian Zhu, Rili Feng, Hongyi Ye, Jiaming Wang, Yancheng He, Pengyu Zou, Lehan Zhang, Xinping Lei, Haoyang Huang, Ken Deng, Ming Sun, Zhaoxiang Zhang, He Ye, Jiaheng Liu
Organizations: Nanjing University · Alibaba Group · Renmin University of China · Kuaishou Technology · Institute of Automation, CAS · University College London
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains that make it hard to tell when the agent goes off track and why. Existing agent tracing analyses either focus on simple interaction or rely on small-scale manual inspection, which limits their scalability and usefulness for real coding workflows. We present CodeTracer, a tracing architecture that parses heterogeneous run artifacts through evolving extractors, reconstructs the full state transition history as a hierarchical trace tree with persistent memory, and performs failure onset localization to pinpoint the failure origin and its downstream chain. To enable systematic evaluation, we construct CodeTraceBench from a large collection of executed trajectories generated by four widely used code agent frameworks on diverse code tasks (e.g., bug fixing, refactoring, and terminal interaction), with supervision at both the stage and step levels for failure localization. Experiments show that CodeTracer substantially outperforms direct prompting and lightweight baselines, and that replaying its diagnostic signals consistently recovers originally failed runs under matched budgets. Our code and data are publicly available.
Figures & tables
Figure 1: Overview of the CodeTracer pipeline. Raw trajectories are standardized into hierarchical traces, curated into CodeTraceBench with step-level supervision, and diagnosed via failure onset localization with optional reflective replay.
Figure 2: Task categories solved per backbone. The central 66 categories are solved by all five models; 65 categories remain unsolved by any model.
Agent
Succ. (%)
Steps
Avg. Steps
Tok.(k)
MiniSWE-Agent
32.8
18,792
19.9
44.6
Terminus 2
35.2
25,381
24.7
51.3
SWE-Agent
37.5
43,216
41.2
86.7
OpenHands
38.3
46,754
43.9
91.4
Table 1: End-to-end success versus process cost on the no intersection setting. Tok.(k) is the average token consumption per task in thousands. SWE-Agent and OpenHands consume 2 × the tokens of MiniSWE-Agent for only modest success gains.
Figure 3: Distribution of error critical steps across stages, contrasting solved and unsolved runs.
Claude-sonnet-4
GPT-5
DeepSeek-V3.2
Qwen3-Coder-480B
Kimi-K2-Instruct
max_iter
Res.%
Tok.(k)
Res.%
Tok.(k)
Res.%
Tok.(k)
Res.%
Tok.(k)
Res.%
Tok.(k)
10
12.67
116.98
38.69
106.19
12.43
86.79
9.76
99.38
10.61
78.81
20
30.13
266.53
45.48
163.89
27.50
212.60
18.22
255.66
23.02
192.28
40
38.00
493.46
47.06
214.60
38.18
400.96
28.85
543.64
29.14
402.20
100
41.65
767.77
47.06
249.80
39.75
479.77
31.67
970.46
32.55
715.24
≥ 150
41.65
886.00
47.06
266.32
39.75
512.42
32.10
1182.06
33.27
805.38
Table 2: Resolved rate (%) and token consumption (in thousands) across iteration budgets and backbones . Tok.(k) reports average token usage within the first max_iter rounds over all valid samples with extractable prefix token traces. Token coverage by model and budget is reported separately for auditability.
Figure 4: Hierarchical trace tree. Exploration steps remain under the current state node, whereas state changing steps induce transitions to child states. CodeTracer builds this structure during tree indexing to provide compressed navigation of execution traces.
Overall
Easy
Medium
Hard
Backbone
P
R
F1
Tok
P
R
F1
Tok
P
R
F1
Tok
P
R
F1
Tok
Bare LLM
Claude-sonnet-4
16.64
15.82
16.22
105.1
18.31
17.08
17.67
68.5
16.07
15.14
15.59
97.8
13.52
12.83
13.17
155.6
GPT-5
16.69
21.46
18.78
58.5
18.42
23.54
20.65
38.2
16.14
20.73
18.13
54.5
16.94
17.42
17.18
86.6
DeepSeek-V3.2
13.11
21.66
16.33
83.4
14.37
23.72
17.87
54.6
12.58
20.74
15.67
77.5
10.28
17.95
12.89
123.6
Mini-CodeTracer
Table 3: Main localization results on CodeTraceBench (macro, step level) stratified by difficulty. P/R/F1 are reported as percentages (%). Tok denotes total LLM tokens (in thousands). Overall uses the full evaluation set; Easy/Medium/Hard split by trajectory difficulty.
Figure 5: Step budget decomposition per backbone (solved vs. unsolved) on the intersection subset. Each bar shows the fraction of total steps that are correct state changes, useful exploration, or ineffective. Hatched bars denote unsolved trajectories.
Claude-sonnet-4
GPT-5
DeepSeek-V3.2
Configuration
P
R
F1
Tok
P
R
F1
Tok
P
R
F1
Tok
Bare LLM
16.64
15.82
16.22
105.1
16.69
21.46
18.78
58.5
13.11
21.66
16.33
83.4
Mini-CodeTracer
20.50
26.24
19.17
82.4
26.03
21.39
19.33
44.8
24.08
20.98
19.24
63.8
+ Evolving Extraction
27.83
33.45
28.12
71.0
31.56
29.87
29.45
37.9
29.74
28.91
28.38
55.3
+ Tree Indexing
40.47
54.87
46.57
56.8
45.02
51.46
48.02
31.1
43.17
49.58
46.14
44.6
Table 4: Component ablation. Each row incrementally adds one component on top of Mini-CodeTracer (diagnosis loop). Evolving extraction adds layout discovery and parser adaptation with accumulating reuse; tree indexing builds a hierarchical trace tree for compressed navigation. P/R/F1 are step level macro metrics (%); Tok is average tokens per instance (k).
Figure 6: Reflective replay. Pass@1 on originally failed runs before and after injecting CodeTracer ’s diagnostic signals under matched budgets.
Figure 7: Effective action ratio. (a)–(e) per model histograms; (f) cross model violin and box summary.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12
Figure 10: Resolved rate vs. iteration budget for all 15 backbone–agent combinations. Rows correspond to backbones (Claude-sonnet-4, GPT-5, DeepSeek-V3.2, Qwen3-Coder-480B, Kimi-K2-Instruct); columns correspond to agent frameworks (MiniSWE-Agent, OpenHands, Terminus 2). Each panel sweeps max_iterations over {5,…,300} . The Qwen–Terminus 2 cell is a placeholder pending data availability.
Figure 11: Step-range error distribution for all 15 backbone–agent combinations. Rows correspond to backbones (Claude-sonnet-4, GPT-5, DeepSeek-V3.2, Qwen3-Coder-480B, Kimi-K2-Instruct); columns correspond to agent frameworks (MiniSWE-Agent, OpenHands, Terminus 2). Each panel shows the stacked area ratio of error critical steps across execution stages, illustrating where in the workflow failures tend to concentrate for each pairing.
Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).
Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows
Arun Joshi
Independent Researcher Islington College (Alumini) Nepal
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.
Yunjia Qi, Zehua Yin, Xintong Shi +10
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University