REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Muhammad Waseem; Aakash Ahmad; Pekka Abrahamsson

cs.SEAug 21, 2026

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Authors: Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

Organizations: Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Computing, University of Derby, Derby, United Kingdom

Abstract

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

Figures & tables

Fig. 1 : Overview of REFINE for Java file-level refactoring-candidate generation, verification, and trace-based analysis.

#	Project name	# Java files	# Classes	# Methods	KLOC
1	JHotDraw	685	773	5,897	122
2	GanttProject	570	837	4,727	86
3	ArgoUML	1,687	2,017	11,846	332
4	Apache Xerces-J	1,109	1,308	8,004	297
5	Apache Ant	993	1,365	10,184	239
6	JEdit	615	1,143	5,883	193

TABLE I : Dataset overview

Group	Metric	Dir.	$\Sigma$ Before	$\Sigma$ After			$\Sigma\Delta=\Sigma$ Before $-\Sigma$ After			% $\Delta$			Holm $p$ / $r$
				GPT-5.5	Gemini 3.1	Opus 4.8	GPT-5.5	Gemini 3.1	Opus 4.8	GPT-5.5	Gemini 3.1	Opus 4.8	GPT-5.5	Gemini 3.1	Opus 4.8
SMELLS	Total Code Smells	$\downarrow$	3 021	959	822	952	2 062	2 199	2 069	68.26	72.79	68.49	$\mathbf{p<.001,\ r=.986}$	$\mathbf{p<.001,\ r=1.000}$	$\mathbf{p<.001,\ r=.992}$
	Critical Code Smells	$\downarrow$	130	42	28	64	88	102	66	67.69	78.46	50.77	$p<.001,\ r=.600$	$p<.001,\ r=.627$	$p<.001,\ r=.602$
	Major Code Smells	$\downarrow$	2 106	272	177	284	1 834	1 929	1 822	87.08	91.60	86.51	$\mathbf{p<.001,\ r=.979}$	$\mathbf{p<.001,\ r=.998}$	$\mathbf{p<.001,\ r=.990}$
	Minor Code Smells	$\downarrow$	785	239	84	213	546	701	572	69.55	89.30	72.87	$p<.001,\ r=.962$	$p<.001,\ r=.992$	$p<.001,\ r=.966$
QUALITY	Cyclomatic Complexity	$\downarrow$	4 631	4 606	3 654	4 301	25	977	330	0.54	21.10	7.13	$p=1.000,\ r=-.010$	$\mathbf{p<.001,\ r=.709}$	$p=1.000,\ r=.128$

TABLE II : Aggregate before–after results for code-smell reduction, quality indicators, and structural metrics.

Fig. 2 : System-level profile for RQ1. Cells show beneficial-direction percentage change for each system, model, and indicator. Positive values are beneficial; negative values are adverse. Total Code Smells, CC, LOC, LCOM, and MML are lower-is-better; MI is higher-is-better. Warmer colours indicate stronger beneficial movement.

Indicator	LLM configuration result, $n/450$ (%)			Model comparison and statistical evidence
	GPT-5.5	Gemini 3.1	Opus 4.8	Pooled Wilson 95% CI	Cochran $Q$	$p$	Best model	Pairwise (Holm–McNemar)
Public method signatures preserved	420/450 (93.3)	388/450 (86.2)	424/450 (94.2)	[89.6, 92.7]	44.08	$<.001$	Opus 4.8	Gemini 3.1 lower than GPT-5.5 and Opus 4.8
Exception-handling constructs preserved	431/450 (95.8)	416/450 (92.4)	433/450 (96.2)	[93.5, 95.9]	15.70	$<.001$	Opus 4.8	Gemini 3.1 lower than GPT-5.5 and Opus 4.8
Framework contracts preserved	425/450 (94.4)	407/450 (90.4)	420/450 (93.3)	[91.2, 94.0]	21.58	$<.001$	GPT-5.5	Gemini 3.1 lower than GPT-5.5 and Opus 4.8
Conditional constructs preserved	405/450 (90.0)	387/450 (86.0)	395/450 (87.8)	[86.1, 89.6]	9.57	.008	GPT-5.5	Gemini 3.1 lower than GPT-5.5
Critical assert/fail-call preservation	257/450 (57.1)	257/450 (57.1)	257/450 (57.1)	[54.5, 59.7]	0.00	1.000	–	No model difference

TABLE III : Automated preservation and verification pass rates across LLM configurations.

Diagnostic	Overall	Model signal
Per-indicator failures
Critical assert/fail-call failures	579 (42.9%)	193 per model
Conditional-construct failures	163 (12.1%)	Gemini 3.1 highest: 63/450
Public-method signature failures	118 (8.7%)	Gemini 3.1 highest: 62/450
Framework-contract failures	98 (7.3%)	Gemini 3.1 highest: 43/450
Exception-handling failures	70 (5.2%)	Gemini 3.1 highest: 34/450

TABLE IV : Failure diagnostics for behavioural-risk preservation indicators.

	Model-level refactoring behaviour					Rank assoc.
Refactoring behaviour	GPT-5.5	Gemini 3.1	Opus 4.8	KW $H$	$p$	$\rho_{R}/\rho_{\%}$
Churn metrics (per-file means)
Total churn (lines)	47.0	69.0	75.2	23.929	$<.001$	+0.695 / -0.278
Lines added	21.2	14.3	31.6	46.969	$<.001$	+0.646 / -0.351
Lines removed	25.8	54.6	43.6	12.107	.002	+0.623 / -0.194
Net LOC change	-4.7	-40.3	-11.9	55.745	$<.001$	+0.327 / -0.343

TABLE V : Refactoring behaviour and code-smell reduction associations.

Outcome	REF.	Dir.	$\Delta$	$p$
Valid/analyzable outputs (%)	100.0	100.0	0.0	1.000
Total smell reduction, median (%)	100.0	20.8	79.2	$<.001$
Major smell reduction, median (%)	100.0	100.0	0.0	$<.001$
Critical smell reduction, median (%)	100.0	100.0	0.0	.006
Public signatures preserved (%)	92.0	75.1	16.9	$<.001$
Critical assert/fail preserved (%)	58.0	100.0	-42.0	$<.001$

TABLE VI : Contextual comparison between REFINE and direct prompting.

Automated Code Refactoring

See all 16 papers

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Mar 4, 2026Alex Thillen, Niels Mündler, Veselin Raychev +1Automated Code Refactoring Automated Software Engineering
Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland
LintEval: Benchmarking LLMs on Linter-Guided File Refactoring
Oct 14, 2026Tongwei Deng, Jinyu Zhu, Yue Cao +3Automated Code Refactoring Code Language Models
School of Computer Science, Peking University · Key Lab of HCST (PKU), MOE; SCS, Peking University · School of Software Engineering, Beijing Jiaotong University
RefactorAssist: Agentic Refinement for Reliable Code Refactoring
Aug 2, 2026Jonathan Cordeiro, Shayan Noei, Ying ZouLLM Self-Refinement Automated Program Repair
Queen’s University, Kingston, Canada

Mar 4, 2026J/K move · Enter open · S save

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev

Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland

Automated Code Refactoring Automated Software Engineering LLM Agent Evaluation Code Generation Benchmarks Repository-Level Code Generation

Open paper AskPDF arXiv

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.

Abstract

Automated Code Refactoring

Explore similar work

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

RefactorAssist: Agentic Refinement for Reliable Code Refactoring