How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

Johin Johny Arimbur

cs.SEApr 12, 2026

How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

Authors: Johin Johny Arimbur

Organizations: Independent Researcher

Abstract

Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both open-weight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds.Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at ~45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM self-correction. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models.

Figures & tables

Model	Family	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	$\Delta$
Llama 3.1 8B	Meta	67.1	73.2	75.6	76.2	76.8	+9.8
Llama 3.3 70B	Meta	82.9	89.6	90.9	92.7	93.3	+10.4
Scout 17B (16E)	Meta	75.6	84.8	87.2	89.0	89.6	+14.0
Maverick 17B (128E)	Meta	87.2	91.5	92.7	93.9	93.9	+6.7
Qwen3 32B	Alibaba	87.8	90.2	92.1	92.7	92.7	+4.9
Gemini 2.5 Flash	Google	86.6	95.7	95.7	96.3	96.3	+9.8

TABLE I: Cumulative pass@1 (%) on HumanEval (164 problems) by repair round.

R_{0}

is the initial attempt;

R_{1}

–

R_{4}

are repair rounds.

\Delta

denotes the total self-repair gain (

R_{4}-R_{0}

) in percentage points.

Fig. 1: Cumulative pass@1 rate on HumanEval across repair rounds for all seven models. Gemini 2.5 Flash achieves the highest final rate (96.3%), while most gains concentrate in the first two rounds.

Model	Family	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	$\Delta$
Llama 3.1 8B	Meta	55.6	66.9	69.6	70.8	71.6	+16.0
Llama 3.3 70B	Meta	67.7	86.4	89.5	89.9	90.7	+23.0
Scout 17B (16E)	Meta	65.4	77.4	80.5	82.9	83.3	+17.9
Maverick 17B (128E)	Meta	72.0	85.6	88.7	91.8	92.6	+20.6
Qwen3 32B	Alibaba	70.8	81.7	85.6	87.9	88.3	+17.5
Gemini 2.5 Flash	Google	63.8	86.8	91.8	93.4	93.8	+30.0

TABLE II: Cumulative pass@1 (%) on MBPP Sanitized (257 problems) by repair round.

Model	Family	HumanEval $\Delta$	MBPP $\Delta$
Llama 3.1 8B	Meta	+9.8	+16.0
Llama 3.3 70B	Meta	+10.4	+23.0
Scout 17B (16E)	Meta	+14.0	+17.9
Maverick 17B (128E)	Meta	+6.7	+20.6
Qwen3 32B	Alibaba	+4.9	+17.5
Gemini 2.5 Flash	Google	+9.8	+30.0

TABLE III: Cross-benchmark comparison of self-repair gains (

\Delta

, in percentage points) on HumanEval and MBPP Sanitized.

Fig. 2: Cross-benchmark comparison of self-repair gains for all seven models. MBPP consistently yields larger gains than HumanEval, with Gemini 2.5 Flash showing the largest MBPP gain (+30.0 pp).

Fig. 3: Distribution of error types at the initial attempt (

R_{0}

) on HumanEval for each model.

Fig. 4: Repair success rate by error type across all models. Assertion errors (logical mistakes) are the hardest to repair.

Fig. 5: Marginal improvement in pass@1 per repair round on HumanEval. The first repair round (

R_{0}\to R_{1}

) yields the largest gain for all models.

Model	Total Tokens	$\Delta$ (pp)	Tokens/pp
Llama 3.1 8B	195K	+9.8	19.9K
Llama 3.3 70B	112K	+10.4	10.8K
Scout 17B (16E)	155K	+14.0	11.1K
Maverick 17B (128E)	93K	+6.7	13.9K
Qwen3 32B	102K	+4.9	20.8K
Gemini 2.5 Flash	121K	+9.8	12.3K

TABLE IV: Total token usage (prompt

+

completion) and self-repair gain on HumanEval (164 problems). Gemini models’ internal reasoning tokens are excluded for comparability; see text.

Model	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	Never	Rep. %
Llama 3.1 8B	110	10	4	1	1	38	29.6
Llama 3.3 70B	136	11	2	3	1	11	60.7
Scout 17B (16E)	124	15	4	3	1	17	57.5
Maverick 17B (128E)	143	7	2	2	0	10	52.4
Qwen3 32B	144	4	3	1	0	12	40.0
Gemini 2.5 Flash	142	15	0	1	0	6	72.7

TABLE V: Per-round repair breakdown on HumanEval: number of problems first solved at each round, and the repair success rate (fraction of initially-failed problems eventually repaired).

Model	Strategy	$R_{0}$	$R_{1}$	$R_{2}$	$\Delta$
Llama 3.1 8B	Chain-of-thought	65.9	72.6	77.4	+11.6
	Explain-then-fix	65.9	72.0	76.2	+10.4
	Minimal	65.9	73.2	75.6	+9.8
Llama 3.3 70B	Chain-of-thought	84.1	93.9	96.3	+12.2
	Explain-then-fix	82.9	90.2	93.9	+11.0
	Minimal	81.1	86.6	87.8	+6.7

TABLE VI: Repair prompt ablation on HumanEval. Cumulative pass@1 (%) at each round for up to three prompt strategies on four models. Scout’s Minimal run is unavailable (see text). Deltas computed from raw problem counts before rounding.

R_{0}

and subsequent rounds may vary slightly across runs due to API-level non-determinism;

\Delta

computed from each run’s own

R_{0}

Fig. 6: Prompt strategy ablation on HumanEval. CoT consistently achieves the highest final pass rate across all models, with the benefit scaling with model capability.

Model	Self-Repair		Resampling
	Final %	Tokens	pass@5 %	Tokens
Llama 3.1 8B	76.8	195K	79.9	219K
Llama 3.3 70B	93.3	112K	90.9	231K
Scout 17B (16E)	89.6	155K	86.0	255K
Qwen3 32B	92.7	102K	92.7	223K

TABLE VII: Self-repair (greedy, 5 rounds) vs. independent resampling (temperature 0.8, 5 samples) on HumanEval (164 problems). Bold indicates higher pass rate. Note: repair uses greedy decoding (

T\!=\!0

) while resampling uses

T\!=\!0.8

; see text.

Fig. 7: Self-repair vs. independent resampling on HumanEval. Self-repair matches or exceeds resampling for all models except the 8B, while consistently using fewer tokens.

Model	$R_{0}$	Final	$\Delta$
Llama 3.1 8B	10.0	12.0	+2.0
Scout 17B (16E)	6.0	14.0	+8.0
Qwen3 32B	8.0	10.0	+2.0
Gemini 2.5 Flash	0.0	16.0	+16.0

TABLE VIII: LiveCodeBench results (50 problems). Self-repair improves all four models evaluated.

LLM Self-Correction

Large language model self-correction studies how language models detect and repair errors in their own answers, reasoning, and actions.

See all 159 papers

Jun 16, 2026

Unlocking LLM Code Correction with Iterative Feedback Loops

Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.

Le Zhang, Suresh Kothari

Iowa State University, Ames Iowa 50010, USA

LLM Self-Correction Execution-Guided Code Generation Code Generation Evaluation

Ask PDF

Oct 11, 2026

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.

Elad Sharony, Ido Greenberg, Jialin Song +3

Technion · NVIDIA

Automated Program Repair

Ask PDF

Jul 5, 2026

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.

Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3

IIIT Delhi · Microsoft Research India · IIT Kanpur

LLM Self-Correction Automated Program Repair Language Model Robustness Code Language Models

Ask PDF

Abstract

LLM Self-Correction

Explore similar work

Unlocking LLM Code Correction with Iterative Feedback Loops

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse