How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

Johin Johny Arimbur

cs.SEApr 12, 2026

How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

Authors: Johin Johny Arimbur

Organizations: Independent Researcher

Abstract

Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both open-weight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds.Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at ~45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM self-correction. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models.

Figures & tables

Model	Family	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	$\Delta$
Llama 3.1 8B	Meta	67.1	73.2	75.6	76.2	76.8	+9.8
Llama 3.3 70B	Meta	82.9	89.6	90.9	92.7	93.3	+10.4
Scout 17B (16E)	Meta	75.6	84.8	87.2	89.0	89.6	+14.0
Maverick 17B (128E)	Meta	87.2	91.5	92.7	93.9	93.9	+6.7
Qwen3 32B	Alibaba	87.8	90.2	92.1	92.7	92.7	+4.9
Gemini 2.5 Flash	Google	86.6	95.7	95.7	96.3	96.3	+9.8

TABLE I: Cumulative pass@1 (%) on HumanEval (164 problems) by repair round.

R_{0}

is the initial attempt;

R_{1}

–

R_{4}

are repair rounds.

\Delta

denotes the total self-repair gain (

R_{4}-R_{0}

) in percentage points.

Fig. 1: Cumulative pass@1 rate on HumanEval across repair rounds for all seven models. Gemini 2.5 Flash achieves the highest final rate (96.3%), while most gains concentrate in the first two rounds.

Model	Family	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	$\Delta$
Llama 3.1 8B	Meta	55.6	66.9	69.6	70.8	71.6	+16.0
Llama 3.3 70B	Meta	67.7	86.4	89.5	89.9	90.7	+23.0
Scout 17B (16E)	Meta	65.4	77.4	80.5	82.9	83.3	+17.9
Maverick 17B (128E)	Meta	72.0	85.6	88.7	91.8	92.6	+20.6
Qwen3 32B	Alibaba	70.8	81.7	85.6	87.9	88.3	+17.5
Gemini 2.5 Flash	Google	63.8	86.8	91.8	93.4	93.8	+30.0

TABLE II: Cumulative pass@1 (%) on MBPP Sanitized (257 problems) by repair round.

Model	Family	HumanEval $\Delta$	MBPP $\Delta$
Llama 3.1 8B	Meta	+9.8	+16.0
Llama 3.3 70B	Meta	+10.4	+23.0
Scout 17B (16E)	Meta	+14.0	+17.9
Maverick 17B (128E)	Meta	+6.7	+20.6
Qwen3 32B	Alibaba	+4.9	+17.5
Gemini 2.5 Flash	Google	+9.8	+30.0

TABLE III: Cross-benchmark comparison of self-repair gains (

\Delta

, in percentage points) on HumanEval and MBPP Sanitized.

Fig. 2: Cross-benchmark comparison of self-repair gains for all seven models. MBPP consistently yields larger gains than HumanEval, with Gemini 2.5 Flash showing the largest MBPP gain (+30.0 pp).

Fig. 3: Distribution of error types at the initial attempt (

R_{0}

) on HumanEval for each model.

Fig. 4: Repair success rate by error type across all models. Assertion errors (logical mistakes) are the hardest to repair.

Fig. 5: Marginal improvement in pass@1 per repair round on HumanEval. The first repair round (

R_{0}\to R_{1}

) yields the largest gain for all models.

Model	Total Tokens	$\Delta$ (pp)	Tokens/pp
Llama 3.1 8B	195K	+9.8	19.9K
Llama 3.3 70B	112K	+10.4	10.8K
Scout 17B (16E)	155K	+14.0	11.1K
Maverick 17B (128E)	93K	+6.7	13.9K
Qwen3 32B	102K	+4.9	20.8K
Gemini 2.5 Flash	121K	+9.8	12.3K

TABLE IV: Total token usage (prompt

+

completion) and self-repair gain on HumanEval (164 problems). Gemini models’ internal reasoning tokens are excluded for comparability; see text.

Model	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	$R_{4}$	Never	Rep. %
Llama 3.1 8B	110	10	4	1	1	38	29.6
Llama 3.3 70B	136	11	2	3	1	11	60.7
Scout 17B (16E)	124	15	4	3	1	17	57.5
Maverick 17B (128E)	143	7	2	2	0	10	52.4
Qwen3 32B	144	4	3	1	0	12	40.0
Gemini 2.5 Flash	142	15	0	1	0	6	72.7

TABLE V: Per-round repair breakdown on HumanEval: number of problems first solved at each round, and the repair success rate (fraction of initially-failed problems eventually repaired).

Model	Strategy	$R_{0}$	$R_{1}$	$R_{2}$	$\Delta$
Llama 3.1 8B	Chain-of-thought	65.9	72.6	77.4	+11.6
	Explain-then-fix	65.9	72.0	76.2	+10.4
	Minimal	65.9	73.2	75.6	+9.8
Llama 3.3 70B	Chain-of-thought	84.1	93.9	96.3	+12.2
	Explain-then-fix	82.9	90.2	93.9	+11.0
	Minimal	81.1	86.6	87.8	+6.7

TABLE VI: Repair prompt ablation on HumanEval. Cumulative pass@1 (%) at each round for up to three prompt strategies on four models. Scout’s Minimal run is unavailable (see text). Deltas computed from raw problem counts before rounding.

R_{0}

and subsequent rounds may vary slightly across runs due to API-level non-determinism;

\Delta

computed from each run’s own

R_{0}

Fig. 6: Prompt strategy ablation on HumanEval. CoT consistently achieves the highest final pass rate across all models, with the benefit scaling with model capability.

Model	Self-Repair		Resampling
	Final %	Tokens	pass@5 %	Tokens
Llama 3.1 8B	76.8	195K	79.9	219K
Llama 3.3 70B	93.3	112K	90.9	231K
Scout 17B (16E)	89.6	155K	86.0	255K
Qwen3 32B	92.7	102K	92.7	223K

TABLE VII: Self-repair (greedy, 5 rounds) vs. independent resampling (temperature 0.8, 5 samples) on HumanEval (164 problems). Bold indicates higher pass rate. Note: repair uses greedy decoding (

T\!=\!0

) while resampling uses

T\!=\!0.8

; see text.

Fig. 7: Self-repair vs. independent resampling on HumanEval. Self-repair matches or exceeds resampling for all models except the 8B, while consistently using fewer tokens.

Model	$R_{0}$	Final	$\Delta$
Llama 3.1 8B	10.0	12.0	+2.0
Scout 17B (16E)	6.0	14.0	+8.0
Qwen3 32B	8.0	10.0	+2.0
Gemini 2.5 Flash	0.0	16.0	+16.0

TABLE VIII: LiveCodeBench results (50 problems). Self-repair improves all four models evaluated.

LLM Self-Correction

Large language model self-correction studies how language models detect and repair errors in their own answers, reasoning, and actions.

See all 159 papers

Unlocking LLM Code Correction with Iterative Feedback Loops
Jun 16, 2026Le Zhang, Suresh KothariLLM Self-Correction Execution-Guided Code Generation
Iowa State University, Ames Iowa 50010, USA
Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair
Oct 11, 2026Elad Sharony, Ido Greenberg, Jialin Song +3Automated Program Repair
Technion · NVIDIA
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Jul 5, 2026Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3LLM Self-Correction Automated Program Repair
IIIT Delhi · Microsoft Research India · IIT Kanpur

Jun 16, 2026J/K move · Enter open · S save

Unlocking LLM Code Correction with Iterative Feedback Loops

Le Zhang, Suresh Kothari

Iowa State University, Ames Iowa 50010, USA

LLM Self-Correction Execution-Guided Code Generation Code Generation Evaluation

Open paper AskPDF arXiv

Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.

Abstract

LLM Self-Correction

Explore similar work

Unlocking LLM Code Correction with Iterative Feedback Loops

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse