LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops

Ravin Ravi; Dylan Bradshaw; Stefano Ruberto; Gunel Jahangirova; Valerio Terragni

cs.SEMar 24, 2026

LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops

Authors: Ravin Ravi, Dylan Bradshaw, Stefano Ruberto, Gunel Jahangirova, Valerio Terragni

Organizations: University of Auckland, Auckland, New Zealand · JRC European Commission, Ispra, Italy · King’s College London, London, United Kingdom

Abstract

Large Language Models (LLMs) are showing remarkable performance in generating source code, yet the generated code often has issues like compilation errors or incorrect code. Researchers and developers often face wasted effort in implementing checks and refining LLM-generated code, frequently duplicating their efforts. This paper presents LLMLOOP, a framework that automates the refinement of both source code and test cases produced by LLMs. LLMLOOP employs five iterative loops: resolving compilation errors, addressing static analysis issues, fixing test case failures, and improving test quality through mutation analysis. These loops ensure the generation of high-quality test cases that serve as both a validation mechanism and a regression test suite for the generated code. We evaluated LLMLOOP on HUMANEVAL-X, a recent benchmark of programming tasks. Results demonstrate the tool's effectiveness in refining LLM-generated outputs.

Figures & tables

Flag	Additional Argument	Description
-e	-	Enable EvoSuite
-p	Path to Java project (String)	Path to an existing Java project
-d	-	Enable debug logging
-t	-	Enable test generation using LLMs
-r	-	Enable code coverage report
-n	Number of retries (Int)	Set the number of attempts in the feedback loop

TABLE I: Command-line flags of LLMloop

	LLMloop stage	pass@1	pass@1
	(mean $\pm$ std)	(mean %)
#1	Baseline (no feedback loop)	117.50 $\pm$ 1.20	71.65%
#2	Compilation (Loop 1)	125.30 $\pm$ 1.55	76.40%
#3	HumanEval-X Given Tests (Loop 2)	130.40 $\pm$ 2.91	79.51%
#4	Static Analysis (Loop 3)	130.50 $\pm$ 3.11	79.57%
Loops 4 and 5 repeated for LLM and EvoSuite -generated test cases

TABLE II: Pass@1 of HumanEval-X problems (mean over 10 runs) at each step of LLMloop .

Fig. 2: Pass@K for the ten runs, baseline vs LLMloop

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 309 papers

Unlocking LLM Code Correction with Iterative Feedback Loops
Jun 16, 2026Le Zhang, Suresh KothariLLM Self-Correction Execution-Guided Code Generation
Iowa State University, Ames Iowa 50010, USA
MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation
Jul 24, 2026Zhen Zhao, Qihang Yang, Feifei Dai +2Execution-Guided Code Generation Code Generation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China
AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation
Jul 24, 2026Jie "JW" WuCode Generation Code Generation Evaluation
Michigan Technological University, Houghton, MI, USA

Jun 16, 2026J/K move · Enter open · S save

Unlocking LLM Code Correction with Iterative Feedback Loops

Le Zhang, Suresh Kothari

Iowa State University, Ames Iowa 50010, USA

LLM Self-Correction Execution-Guided Code Generation Code Generation Evaluation

Open paper AskPDF arXiv

Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.

Abstract

LLM Code Generation

Explore similar work

Unlocking LLM Code Correction with Iterative Feedback Loops

MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation