Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling
Authors: Christoph Daxerer, Johann Glock, Christian Macho, Rudolf Ramler, Martin Pinzger, Josef Pichler
Organizations: Software Competence Center Hagenberg GmbH (SCCH), Hagenberg, Austria · Department of Informatics Systems, University of Klagenfurt, Klagenfurt, Austria · School of Informatics, Communications and Media, University of Applied Sciences Upper Austria, Hagenberg, Austria
Background: To localize a fault, a coding agent can reason from source code alone or use debugger assistance by running the program and stepping through it in an interactive debugger. While earlier agents struggled with static execution tracing and debugger operation, frontier LLMs have improved at both. Objective: We evaluate how accurately coding agents localize faults without tools across original and transformed programs, how much debugger assistance improves accuracy and at what cost, and how agents use it when optional. Methods: We seed 610 single-line faults into 144 LeetCodeDataset solutions and rewrite each faulty program with five recipes of behavior-preserving transformations. Agents based on Sol 6.1, Luna 6, and Qwen 3.8 localize each fault with no, optional, or encouraged debugger assistance. Results: Without tools, the agents localized 98.2 to 99.9% of faults in original and 96.7 to 99.9% in transformed programs, losing accuracy mainly where recipes restructured loops and conditions. Encouraged assistance removed 63% and 61% of Luna 6's and Qwen 3.8's remaining transformed errors (+2.1 and +1.3 percentage points) at 14.7 to 27.0 times the token use, while Sol 6.1 stayed at 99.9%. With optional assistance, Luna 6 and Qwen 3.8 still gained +1.5 and +1.3 points, though the agents used the interactive debugger in only 0.2 to 6.2% of runs. Conclusions: On short, precisely specified programs, current coding agents localize nearly all faults from source code alone, and debugger assistance removes most errors that remain. Given the choice, agents executed the program far more often than they stepped through it, yet kept most of the benefit.
Figures & tables
Programs
LOC
CC
Kept
Reference programs
Easy
50 /
50
8
[4–11]
3
[2–4]
Medium
48 /
50
11
[7–14]
4
[3–5]
Hard
46 /
50
19
[14–24]
8
[5–11]
Faulty programs, by fault type
Off-by-one
125 /
150
13
[9–18]
5
[3–7]
TABLE I: Constructed programs. Programs gives admitted out of attempted. LOC and CC give medians with interquartile ranges, and for transformed programs the median change from the untransformed faulty program, in percent for LOC. Kept is the median share of preserved four-token sequences.
Model
Programs
No-tools
Discretionary
Encouraged
No-tools change ( G )
Effect ( Δ,Δ′ )
Difference ( I=Δ′−Δ )
Luna 6
Untransformed
98.2
98.1
98.9
+0.7 [+0.2, +1.4]
Transformed
96.7
98.2
98.8
− 1.5 [ − 2.3, − 0.6]
+2.1 [+1.4, +2.8]
+1.4 [+0.6, +2.1]
Sol 6.1
Untransformed
99.9
99.8
99.8
− 0.1 [ − 0.3, +0.0]
Transformed
99.9
99.9
99.9
+0.0 [ − 0.1, +0.0]
+0.0 [+0.0, +0.1]
+0.1 [+0.0, +0.3]
Qwen 3.8
Untransformed
98.8
98.9
99.1
+0.3 [ − 0.3, +1.1]
Transformed
97.9
99.1
99.2
− 0.9 [ − 1.6, − 0.1]
+1.3 [+0.8, +1.8]
+1.0 [+0.1, +1.7]
TABLE II: Top-1 localization accuracy in percent under statement-level grading. No-tools change is transformed minus untransformed static accuracy ( G ), Effect is encouraged-tools minus no-tools ( Δ and Δ′ on un-/transformed programs), and Difference is I=Δ′−Δ , all in percentage points. Untransformed accuracies weight each program by its number of matched transformed versions, as G and Δ do. Brackets give 95% intervals. Fig. 3 breaks G and Δ′ down by recipe and fault type.
Fig. 3: No-tools change G (left, transformed minus untransformed accuracy under no-tools) and assistance effect Δ′ on transformed programs (right, encouraged-tools minus no-tools) by recipe and fault type, per model, in percentage points, with 95% intervals. The rightmost column gives no-tools accuracy in percent on the matched untransformed (in gray) and transformed programs.
Untransformed
Transformed
Encouraged / no-tools
Model
Measure
No-tools
Discr.
Encour.
No-tools
Discr.
Encour.
Untr.
Transf.
Luna 6
Tokens (thousands)
1.4
2.2
41.8
2.0
3.3
55.0
29.7 ×
27.0 ×
Wall time (s)
5.9
4.5
41.2
7.5
6.4
47.0
7.0 ×
6.2 ×
Cost (US$ per 1,000 episodes)
0.22
0.25
1.26
0.32
0.38
1.60
5.7 ×
5.0 ×
Sol 6.1
Tokens (thousands)
1.3
3.6
25.0
1.7
4.9
32.2
19.8 ×
18.4 ×
Wall time (s)
4.9
14.0
37.8
5.2
15.0
39.9
7.7 ×
7.7 ×
TABLE III: Episode cost. Tokens and wall time are means per episode, and monetary cost is the mean cost of 1,000 episodes, estimated for Qwen 3.8 from a provider’s list prices. The last two columns divide encouraged-tools by no-tools.
Discretionary
Encouraged
Model
Measure
Untr.
Transf.
Untr.
Transf.
Luna 6
Any debugger action
0.0%
0.3%
100.0%
100.0%
Probe runs only
8.8%
11.8%
0.0%
0.0%
Debugger actions
0.0
0.0
11.6
12.5
Probe runs
0.1
0.2
1.1
1.2
Breakpoints
–
18.4%
14.6%
17.5%
TABLE IV: Tool use in the tool-enabled settings on untransformed and transformed programs. The first two rows per model give the shares of episodes with at least one debugger action and of episodes with probe runs but no debugger action. The next two give the mean numbers of debugger actions and probe runs per episode. The last three give the shares of debugger actions that manage breakpoints, step, and inspect state.
Discretionary minus
Slope β1 by
No-tools
Encouraged
Probe runs
Debugger
Luna 6
Untr.
− 0.1 [ − 1.0, +0.7]
− 0.8 [ − 1.6, − 0.2]
+3.2 †
–
Transf.
+1.5 [+0.9, +2.2]
− 0.6 [ − 1.0, − 0.2]
+15.0 †
+66.7 †
Sol 6.1
Untr.
− 0.2 [ − 0.4, +0.0]
− 0.1 [ − 0.2, +0.0]
− 1.1 †
–
TABLE V: Discretionary use. Mean per-case accuracy differences between discretionary-tools and the other two settings, and the slopes β1 of the per-case assistance effect on the rank of probe runs and of debugger actions under discretionary-tools (Section III-D ), all in percentage points. Brackets give 95% intervals. A dagger marks a slope without an interval and a dash a slope that is not reported.
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains that make it hard to tell when the agent goes off track and why. Existing agent tracing analyses either focus on simple interaction or rely on small-scale manual inspection, which limits their scalability and usefulness for real coding workflows. We present CodeTracer, a tracing architecture that parses heterogeneous run artifacts through evolving extractors, reconstructs the full state transition history as a hierarchical trace tree with persistent memory, and performs failure onset localization to pinpoint the failure origin and its downstream chain. To enable systematic evaluation, we construct CodeTraceBench from a large collection of executed trajectories generated by four widely used code agent frameworks on diverse code tasks (e.g., bug fixing, refactoring, and terminal interaction), with supervision at both the stage and step levels for failure localization. Experiments show that CodeTracer substantially outperforms direct prompting and lightweight baselines, and that replaying its diagnostic signals consistently recovers originally failed runs under matched budgets. Our code and data are publicly available.
Han Li, Yifan Yao, Letian Zhu +13
Nanjing University · Alibaba Group · Renmin University of China +3
While significant progress has been made in automating various aspects of software development through coding agents, there is still significant room for improvement in their bug fixing capabilities. Debugging and investigation of runtime behavior remains largely a manual, developer-driven process. Popular coding agents typically rely on either static analysis of the code or iterative test-fix cycles, which is akin to trial and error debugging. We posit that there is a wealth of rich runtime information that developers routinely access while debugging code, which agents are currently deprived of due to design limitations. Despite how prevalent debuggers are in modern IDEs and command-line tools, they have surprisingly not made their way into coding agents. In this work, we introduce Debug2Fix, a novel framework that incorporates interactive debugging as a core component of a software engineering agent via a subagent architecture. We incorporate debuggers for Java and Python into our agent framework and evaluate against GitBug-Java and SWE-Bench-Live and achieve >20% improvement in performance compared to the baseline for certain models. Furthermore, using our framework, we're able to make weaker models like GPT-5 and Claude Haiku 4.5 match or exceed the performances of stronger models like Claude Sonnet 4.5, showing that better tool design is often just as important as switching to a more expensive model. Finally, we conduct systematic ablations demonstrating the importance of both the subagent architecture and debugger integration.
LLM agents solve repository-level coding tasks through multi-turn tool use, but utilize half their budget on locating faults before editing. Dedicated localization frameworks have emerged, yet are still evaluated as file retrieval rather than actionable diagnosis, producing locations without the diagnostic context a repair agent needs. We introduce SHERLOC (Structured Hypothesis-driven Exploration and Reasoning for Localization), a training-free framework pairing a reasoning LLM with compact repository tools and self-recovery, without fine-tuning or multi-agent orchestration. SHERLOC reaches state-of-the-art localization across model scales: 84.33% accuracy@1 on SWE-Bench Lite and 81.27% recall@1 on SWE-Bench Verified; at ~30B parameters, it matches or outperforms other agentic methods. Injecting our locations and diagnostic findings into repair agents yields an average +5.95 pp resolve-rate gain from the best SHERLOC result per setting on SWE-Bench Verified. SHERLOC cuts localization and total tokens by 36.7% and 23.1% on average.
Hovhannes Tamoyan, Sean Narenthiran, Erik Arakelyan +2
NVIDIA, Santa Clara, CA 95051, USA · 2TU Darmstadt, Darmstadt, Germany · 3hessian.AI & National Research Center for Applied Cybersecurity ATHENE