Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling
Authors: Christoph Daxerer, Johann Glock, Christian Macho, Rudolf Ramler, Martin Pinzger, Josef Pichler
Organizations: Software Competence Center Hagenberg GmbH (SCCH), Hagenberg, Austria · Department of Informatics Systems, University of Klagenfurt, Klagenfurt, Austria · School of Informatics, Communications and Media, University of Applied Sciences Upper Austria, Hagenberg, Austria
Background: To localize a fault, a coding agent can reason from source code alone or use debugger assistance by running the program and stepping through it in an interactive debugger. While earlier agents struggled with static execution tracing and debugger operation, frontier LLMs have improved at both. Objective: We evaluate how accurately coding agents localize faults without tools across original and transformed programs, how much debugger assistance improves accuracy and at what cost, and how agents use it when optional. Methods: We seed 610 single-line faults into 144 LeetCodeDataset solutions and rewrite each faulty program with five recipes of behavior-preserving transformations. Agents based on Sol 6.1, Luna 6, and Qwen 3.8 localize each fault with no, optional, or encouraged debugger assistance. Results: Without tools, the agents localized 98.2 to 99.9% of faults in original and 96.7 to 99.9% in transformed programs, losing accuracy mainly where recipes restructured loops and conditions. Encouraged assistance removed 63% and 61% of Luna 6's and Qwen 3.8's remaining transformed errors (+2.1 and +1.3 percentage points) at 14.7 to 27.0 times the token use, while Sol 6.1 stayed at 99.9%. With optional assistance, Luna 6 and Qwen 3.8 still gained +1.5 and +1.3 points, though the agents used the interactive debugger in only 0.2 to 6.2% of runs. Conclusions: On short, precisely specified programs, current coding agents localize nearly all faults from source code alone, and debugger assistance removes most errors that remain. Given the choice, agents executed the program far more often than they stepped through it, yet kept most of the benefit.
Figures & tables
Programs
LOC
CC
Kept
Reference programs
Easy
50 /
50
8
[4–11]
3
[2–4]
Medium
48 /
50
11
[7–14]
4
[3–5]
Hard
46 /
50
19
[14–24]
8
[5–11]
Faulty programs, by fault type
Off-by-one
125 /
150
13
[9–18]
5
[3–7]
TABLE I: Constructed programs. Programs gives admitted out of attempted. LOC and CC give medians with interquartile ranges, and for transformed programs the median change from the untransformed faulty program, in percent for LOC. Kept is the median share of preserved four-token sequences.
Model
Programs
No-tools
Discretionary
Encouraged
No-tools change ( G )
Effect ( Δ,Δ′ )
Difference ( I=Δ′−Δ )
Luna 6
Untransformed
98.2
98.1
98.9
+0.7 [+0.2, +1.4]
Transformed
96.7
98.2
98.8
− 1.5 [ − 2.3, − 0.6]
+2.1 [+1.4, +2.8]
+1.4 [+0.6, +2.1]
Sol 6.1
Untransformed
99.9
99.8
99.8
− 0.1 [ − 0.3, +0.0]
Transformed
99.9
99.9
99.9
+0.0 [ − 0.1, +0.0]
+0.0 [+0.0, +0.1]
+0.1 [+0.0, +0.3]
Qwen 3.8
Untransformed
98.8
98.9
99.1
+0.3 [ − 0.3, +1.1]
Transformed
97.9
99.1
99.2
− 0.9 [ − 1.6, − 0.1]
+1.3 [+0.8, +1.8]
+1.0 [+0.1, +1.7]
TABLE II: Top-1 localization accuracy in percent under statement-level grading. No-tools change is transformed minus untransformed static accuracy ( G ), Effect is encouraged-tools minus no-tools ( Δ and Δ′ on un-/transformed programs), and Difference is I=Δ′−Δ , all in percentage points. Untransformed accuracies weight each program by its number of matched transformed versions, as G and Δ do. Brackets give 95% intervals. Fig. 3 breaks G and Δ′ down by recipe and fault type.
Fig. 3: No-tools change G (left, transformed minus untransformed accuracy under no-tools) and assistance effect Δ′ on transformed programs (right, encouraged-tools minus no-tools) by recipe and fault type, per model, in percentage points, with 95% intervals. The rightmost column gives no-tools accuracy in percent on the matched untransformed (in gray) and transformed programs.
Untransformed
Transformed
Encouraged / no-tools
Model
Measure
No-tools
Discr.
Encour.
No-tools
Discr.
Encour.
Untr.
Transf.
Luna 6
Tokens (thousands)
1.4
2.2
41.8
2.0
3.3
55.0
29.7 ×
27.0 ×
Wall time (s)
5.9
4.5
41.2
7.5
6.4
47.0
7.0 ×
6.2 ×
Cost (US$ per 1,000 episodes)
0.22
0.25
1.26
0.32
0.38
1.60
5.7 ×
5.0 ×
Sol 6.1
Tokens (thousands)
1.3
3.6
25.0
1.7
4.9
32.2
19.8 ×
18.4 ×
Wall time (s)
4.9
14.0
37.8
5.2
15.0
39.9
7.7 ×
7.7 ×
TABLE III: Episode cost. Tokens and wall time are means per episode, and monetary cost is the mean cost of 1,000 episodes, estimated for Qwen 3.8 from a provider’s list prices. The last two columns divide encouraged-tools by no-tools.
Discretionary
Encouraged
Model
Measure
Untr.
Transf.
Untr.
Transf.
Luna 6
Any debugger action
0.0%
0.3%
100.0%
100.0%
Probe runs only
8.8%
11.8%
0.0%
0.0%
Debugger actions
0.0
0.0
11.6
12.5
Probe runs
0.1
0.2
1.1
1.2
Breakpoints
–
18.4%
14.6%
17.5%
TABLE IV: Tool use in the tool-enabled settings on untransformed and transformed programs. The first two rows per model give the shares of episodes with at least one debugger action and of episodes with probe runs but no debugger action. The next two give the mean numbers of debugger actions and probe runs per episode. The last three give the shares of debugger actions that manage breakpoints, step, and inspect state.
Discretionary minus
Slope β1 by
No-tools
Encouraged
Probe runs
Debugger
Luna 6
Untr.
− 0.1 [ − 1.0, +0.7]
− 0.8 [ − 1.6, − 0.2]
+3.2 †
–
Transf.
+1.5 [+0.9, +2.2]
− 0.6 [ − 1.0, − 0.2]
+15.0 †
+66.7 †
Sol 6.1
Untr.
− 0.2 [ − 0.4, +0.0]
− 0.1 [ − 0.2, +0.0]
− 1.1 †
–
TABLE V: Discretionary use. Mean per-case accuracy differences between discretionary-tools and the other two settings, and the slopes β1 of the per-case assistance effect on the rank of probe runs and of debugger actions under discretionary-tools (Section III-D ), all in percentage points. Brackets give 95% intervals. A dagger marks a slope without an interval and a dash a slope that is not reported.