Iterative repair returns a failed program's error to a large language model (LLM) and asks it to fix it. After iterative repair, the remaining failures are mostly assertion failures: with four Qwen2.5 models, they make up 79% of residual failures on BigCodeBench, all on HumanEval and 93% on LiveCodeBench. We argue, through an information view of the feedback, that an assertion failure says little about where the fault lies. We propose Breakpoint Debugging (BPD), which returns the failed program annotated inline with its variables' runtime values. We take the tasks that iterative repair failed to fix with five repair attempts and compare two ways of spending five further generations on them: BPD, and resampling. On BigCodeBench, BPD repairs 26.7% of these tasks against 22.4%, and its advantage grows with model size. On HumanEval, BPD outperforms resampling but only slightly, and on LiveCodeBench, fresh sampling repairs more (6.5% against 4.9%). Furthermore, BPD reduces unchanged-code resubmissions from 28% to 8% on BigCodeBench. Using only the first failing test's trace, it outperforms tracing all failing tests while using fewer tokens than both all-trace BPD and plain iterative repair. On LiveCodeBench, BPD gives no gain even on the public tests, so a visible failing input is not enough for a trace to help. These results match the information view: a trace helps only when the test exposes a failing input, and the fault is an implementation slip; otherwise, a repair loop should resample rather than trace.
Figures & tables
Baseline
Fresh start
One trace
All traces
Benchmark
Model
BPD set
Acc. (%)
Acc. (%)
Repaired
Acc. (%)
Repaired
Acc. (%)
Repaired
BigCodeBench
7B
320
63.5
70.9
84
71.7
93
70.2
76
14B
258
71.3
76.9
64
78.8
85
77.3
68
32B
222
76.3
80.3
45
81.8
63
82.1
66
72B
214
76.3
79.3
34
81.4
58
80.8
51
Pooled
1,014
71.9
76.8
227 (22.4%)
78.4
299 (29.5%)
77.6
261 (25.8%)
TABLE I: Accuracy on all tasks (%) and tasks repaired on the BPD set (count). “Fresh” makes six plain calls per task. By construction, the baseline repairs none of the BPD set. Best performance in each row is shown in bold .
Benchmark
7B
14B
32B
72B
BigCodeBench
76.9
78.9
82.2
79.3
HumanEval
100
100
100
100
LiveCodeBench
95.3
96.0
94.0
86.8
TABLE II: Share of residual failures that are assertion failures (%)
Measure
Baseline
One trace
All traces
Identical resubmission (%)
28–23
8–6
9–5
Similarity to previous
.96–.90
.91–.79
.90–.80
More tests pass (%)
13–12
17–24
16–16
Mentions “trace” (%)
0
25–32
19–15
TABLE III: Repair attempts on the BPD set (BigCodeBench / HumanEval).
Model
Set
Baseline
One trace
All traces
7B
121
12.4
18.2 (12.4)
19.8 (10.7)
14B
120
30.0
29.2 (18.3)
23.3 (14.2)
32B
109
33.0
38.5 (15.6)
33.0 (15.6)
72B
99
38.4
29.3 (14.1)
31.3 (19.2)
Pooled
449
27.8
28.5 (15.1)
26.5 (14.7)
TABLE IV: LiveCodeBench BPD set judged on public tests only: share cleared (%), with the share cleared at attempt 0 in brackets.
Bench.
Run
Calls
Tok./call
Total (k)
Change
BCB
Baseline
6,086
1,795
10,925
One trace
5,095
2,061
10,502
−3.9%
All traces
5,203
2,692
14,007
+28.2%
HE
Baseline
240
943
226
One trace
167
1,139
190
−16.0%
All traces
168
1,604
270
+19.1%
TABLE V: Token usage on the BPD set, pooled over models (Qwen2.5 tokeniser). BPD is BigCodeBench, HE is HumanEval and LCB is LiveCodeBanch
Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.
When code-repair agents fail, final tests reveal that a patch is wrong but not which earlier commitment caused it or which software obligation failed. Poison Valleys traces this path in 1,844 model requests on two synthetic Python cases: PersistQueue adds atomic batch claiming to a disk-backed queue, while Dijkstra adds source-sensitive filtering to shortest-path search. We define plausible but incorrect software mechanisms, place each in a model's hypothesis, implementation plan, or both, and evaluate the generated code with an Oracle and targeted diagnostic coordinates. Moving the misconception from hypothesis to plan increased failure by 32.2% in PersistQueue and 28.9% in Dijkstra after matching the experimental mix. Different misconceptions produced distinct, model-dependent patterns of violated obligations, although several could converge on the same behavior. This creates a measured connection from upstream misconception to code mechanism and failure form. Code, data, and demo: https://poisonvalleys.org.
Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both open-weight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds.Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at ~45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM self-correction. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models.