When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness

Muhammad Waseem; Aakash Ahmad; Pekka Abrahamsson

cs.SEOct 17, 2026

When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness

Authors: Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

Organizations: Tampere University, Finland. · the School of Computing and Engineering, University of Derby, UK.

Abstract

Large Language Models (LLMs) are increasingly used to refactor source files, but file-local evaluations do not reveal changes to contracts, dependencies, or caller assumptions affecting repository adoption. To address this gap, we conduct 1,500 generation attempts across 150 production files from 15 open-source Java repositories, five LLM configurations, and two dependency-aware generation conditions: single-prompt refactoring and LangGraph-based multi-agent refactoring. Of these attempts, 1,455 yield complete candidates eligible for analysis. We evaluate these candidates using local changes, repository-facing structural indicators, detected caller impact, the behavioural-equivalence indicator, and available test evidence, and categorize them as Low-risk, Review-needed, High Contract/Dependency Risk, or Inconclusive. The results show that 1,365 candidates (93.8%) are classified as High Contract/Dependency Risk, whereas 106 (7.3%) have at least one detected broken downstream caller and 290 (19.9%) have a false behavioural-equivalence indicator; only eight satisfy the Low-risk criteria. Within the reason-covered subset, the High-risk rate decreases under more conservative dependency-delta thresholds, while 60 of the 65 candidates with conclusive targeted-test outcomes pass, including 50 classified as High risk. Paired analyses of 705 complete pairs identify no statistically significant generation-condition difference in the principal risk indicators, although model configurations differ in specific detected risk channels. These findings suggest that practitioners should treat LLM-generated refactorings as candidates requiring staged repository-level verification, while future research should validate screening and extraction precision and strengthen compilation, behavioural, and coordinated multi-file verification.

Figures & tables

Fig. 1: Local improvement and repository-level verification of an LLM-generated refactoring candidate. A candidate may improve the edited file while changing a public contract, caller-used API, dependency relation, framework-sensitive construct, or downstream caller assumption.

Fig. 2: Overview of the study methodology. The study selects production Java files with local refactoring need and repository-level exposure, constructs dependency context, generates candidates under dependency-aware single-prompt and LangGraph-based multi-agent conditions, and evaluates eligible candidates using local change measures, repository-level risk indicators, staged verification evidence, and statistical analyses. Generation completeness is assessed before candidate-level analysis.

#	Project	Java files	Classes	Methods	KLOC
1	Ant	993	1,365	10,184	239
2	ArgoUML	1,687	2,017	11,846	332
3	Checkstyle	2,273	2,839	7,215	188
4	Eclipse Collections	997	1,360	16,502	214
5	GanttProject	570	837	4,727	86
6	Guava	1,995	4,139	21,018	475

TABLE I: Subject systems used in the study.

Stage group	Nodes	Responsibility
Preparation	load , analyze , dependency_context	Load the fixed baseline file, detect PMD smells, and construct dependency context.
Planning	plan , llm_plan , feasibility	Map smells to refactoring techniques, produce a structured plan, and check size or feasibility constraints.
Generation	refactor	Generate the candidate Java file.
Verification	verify , ripple_verify , llm_verify	Run static quality checks, apply ripple-risk verification, and record an LLM-based assessment.
Reporting	report	Persist metrics, traces, and reports.

TABLE II: Stages of the LangGraph-based multi-agent generation condition.

Model	Configured model identifier
Gemini	google/gemini-3.1-pro-preview
GPT	openai/gpt-5.5
Claude	anthropic/claude-opus-4.8
DeepSeek	deepseek/deepseek-v4-pro
Qwen	qwen/qwen3.6-plus

TABLE III: Model configurations used in the study.

Category	Setting	Value
Temperature	Refactoring call	0.4 for B0-V2 and the MA refactoring agent
	MA planning call	0.2
	MA verification call	0.1
Output budget	Refactoring output	LOC-scaled, with a default maximum of 2,500 tokens
	MA planning / verification	4,096 / 2,048 tokens
Timeout	Refactoring call	LOC-banded, typically 480–3,600 seconds

TABLE IV: Model invocation settings used in the study.

Project	Files	LOC		Smells		Fan-in		Fan-out		APIs		Caller APIs		Callers		Dep. risk		Structural
Project	Files	Tot.	Med.	Tot.	Med.	Tot.	Med.	Tot.	Med.	Tot.	Med.	Tot.	Med.	Tot.	Med.	Tot.	Med.	Inh.	Iface
Ant	10	3258	254.0	48	3.0	108	1.5	156	15.0	145	10.5	44	1.0	108	1.5	37.7	2.65	7	4
ArgoUML	10	4709	460.5	53	5.0	206	0.0	325	16.0	89	4.5	24	0.0	206	0.0	22.3	0.98	8	4
Checkstyle	10	3849	369.0	12	1.0	93	1.5	160	17.0	137	5.5	12	1.0	93	1.5	25.2	1.05	7	2
Eclipse Collections	10	5793	261.5	95	2.5	44	1.0	345	29.5	383	17.0	198	2.0	44	1.0	127.0	3.30	0	5
GanttProject	10	2353	200.0	41	2.0	91	2.0	148	7.5	119	7.5	59	1.5	91	2.0	38.9	1.73	4	4

TABLE V: Project-level characteristics of the selected files before LLM-based refactoring.

Metric	Min	Q1	Med.	Mean	Q3	Max	% $>0$
LOC	18	88.5	212.5	342.7	375.5	3182	100.0
Smells	1	1.0	3.0	8.7	11.8	60	100.0
Fan-in	0	0.0	1.0	8.3	5.8	181	68.7
Fan-out	0	4.0	10.0	16.7	21.8	119	98.0
APIs	0	2.0	7.0	14.5	18.0	135	92.7
Caller APIs	0	0.0	1.5	5.7	6.0	86	68.7

TABLE VI: Distribution of exposure indicators across the 150 selected files.

Metric	$H$	Raw $p$	Holm-adjusted $p$
LOC	23.50	0.0526	0.1863
Baseline smells	41.91	0.0001	0.0010
Fan-in	25.21	0.0325	0.1863
Fan-out	28.56	0.0120	0.0839
Public/protected APIs	15.30	0.3578	0.3578
Caller-used APIs	25.40	0.0308	0.1863

TABLE VII: Differences among projects in the file-level exposure indicators.

Detected outcome	Count	$N$	Rate	Wilson 95% CI	Single-prompt	Multi-agent
High contract/dependency risk	1,365	1,455	93.8%	92.5–94.9%	661/706 (93.6%)	704/749 (94.0%)
Medium contract/dependency risk	90	1,455	6.2%	5.1–7.5%	45/706 (6.4%)	45/749 (6.0%)
At least one broken caller	106	1,455	7.3%	6.1–8.7%	57/706 (8.1%)	49/749 (6.5%)
Behavioural-equivalence indicator false	290	1,455	19.9%	18.0–22.1%	138/706 (19.5%)	152/749 (20.3%)

TABLE VIII: Core repository-level risk indicators detected in the eligible candidates.

Category	Detected mechanism	Count	Rate	Single-prompt	Multi-agent
Dependency	Any dependency change	716	82.9%	94/115 (81.7%)	622/749 (83.0%)
	Large dependency delta	662	76.6%	85/115 (73.9%)	577/749 (77.0%)
	Internal import/dependency change	54	6.3%	9/115 (7.8%)	45/749 (6.0%)
Caller/API	Caller-used API changed or removed	597	69.1%	83/115 (72.2%)	514/749 (68.6%)
Contract/API	Any contract/API change	333	38.5%	40/115 (34.8%)	293/749 (39.1%)
	Public/protected signature changed	231	26.7%	27/115 (23.5%)	204/749 (27.2%)

TABLE IX: Detailed risk mechanisms among candidates with usable risk-reason information.

Threshold	High	$N$	Rate
5	810	864	93.8%
7	790	864	91.4%
10	765	864	88.5%
15	745	864	86.2%
20	714	864	82.6%

TABLE X: Sensitivity of the High-risk classification to the dependency-delta threshold.

Fig. 3: Distribution and concentration of detected downstream caller impact among the 1,455 eligible candidates. Panel (a) presents the distribution of broken-caller counts overall and by generation condition. Panel (b) presents the files associated with the largest eligible broken-caller counts. The distribution is strongly zero-inflated, with the largest effects concentrated in a small number of files.

Outcome	Paired $N$	Single-prompt	Multi-agent	MA–SP	Discordant pairs	$p$	Test
High contract/dependency risk	705	660 (93.6%)	660 (93.6%)	0.0 pp	2 / 2	1.000	Exact McNemar
Broken caller detected	705	57 (8.1%)	49 (7.0%)	$-1.1$ pp	23 / 15	0.256	Exact McNemar
Behavioural-equivalence indicator false	705	138 (19.6%)	152 (21.6%)	$+2.0$ pp	55 / 69	0.243	Exact McNemar
Broken-caller count	705	0 [0, 0]	0 [0, 0]	0	47 non-zero differences	1.000	Wilcoxon signed-rank

TABLE XI: Paired comparison of single-prompt and multi-agent generation.

Detected mechanism	Single-prompt	Multi-agent	MA–SP	Raw $p$	Holm $p$	Adjusted result
Caller-used API affected	83 (72.2%)	83 (72.2%)	0.0 pp	–	1.000	Not significant
Public/protected signature changed	27 (23.5%)	25 (21.7%)	$-1.7$ pp	0.500	1.000	Not significant
Public API removed	6 (5.2%)	2 (1.7%)	$-3.5$ pp	0.289	1.000	Not significant
Constructor signature changed	19 (16.5%)	18 (15.7%)	$-0.9$ pp	1.000	1.000	Not significant
Package declaration changed	0 (0.0%)	0 (0.0%)	0.0 pp	–	1.000	Not significant
Large dependency delta	85 (73.9%)	77 (67.0%)	$-7.0$ pp	0.0078	0.0859	Not significant

TABLE XII: Paired comparison of detailed risk mechanisms among the 115 pairs with usable reason information in both conditions.

LLM configuration	Pair $N$	High risk		Broken caller		Behavioural-equivalence indicator false
LLM configuration	Pair $N$	SP	MA	SP	MA	SP	MA
Claude Opus 4.8	150	94.0%	94.0%	8.7%	8.0%	26.0%	30.0%
DeepSeek v4 Pro	148	93.9%	93.9%	6.8%	8.1%	18.9%	21.6%
Gemini 3.1 Pro Preview	149	94.0%	94.6%	12.1%	9.4%	25.5%	23.5%
GPT-5.5	149	94.0%	94.6%	8.1%	7.4%	15.4%	26.8%
Qwen 3.6 Plus	109	91.7%	89.9%	3.7%	0.0%	9.2%	0.0%

TABLE XIII: Repository-level risk indicators by model and generation condition.

Condition	Outcome	Complete blocks	Statistic	$p$	Post-hoc status
Single-prompt	High contract/dependency risk	108	Not defined	–	No within-block variation
	Broken-caller occurrence	108	$Q=18.70$	$<0.001$	Exact McNemar with Holm
	Behavioural-equivalence indicator false	108	$Q=29.20$	$<0.001$	Exact McNemar with Holm
	Broken-caller count	108	$\chi^{2}_{F}=21.16$	$<0.001$	Pairwise tests not executed
Multi-agent	High contract/dependency risk	149	$Q=8.57$	0.073	Not applied
	Broken-caller occurrence	149	$Q=21.89$	$<0.001$	Exact McNemar with Holm

TABLE XIV: Model-level omnibus tests within each generation condition.

Reporting category	Count	Rate	Wilson 95% CI	Single-prompt	Multi-agent
Low-risk	8	0.5%	0.3–1.1%	0/706 (0.0%)	8/749 (1.1%)
Review-needed	55	3.8%	2.9–4.9%	43/706 (6.1%)	12/749 (1.6%)
High Contract/Dependency Risk	1,365	93.8%	92.5–94.9%	661/706 (93.6%)	704/749 (94.0%)
Inconclusive	27	1.9%	1.3–2.7%	2/706 (0.3%)	25/749 (3.3%)

TABLE XV: Adoption-readiness distribution under the available verification evidence.

Evidence item	Count	Rate
Structural and behavioural evidence ( $n=1{,}455$ eligible candidates)
High contract risk	1,365	93.8%
Medium contract risk	90	6.2%
Broken caller detected	106	7.3%
Behavioural-equivalence indicator: true	1,165	80.1%
Behavioural-equivalence indicator: false	290	19.9%

TABLE XVI: Coverage and outcomes of the adoption-readiness evidence.

LLM configuration	Eligible $n$	Low-risk	Review-needed	High risk	Inconclusive
Claude Opus 4.8	300	1 (0.3%)	9 (3.0%)	282 (94.0%)	8 (2.7%)
DeepSeek v4 Pro	298	2 (0.7%)	11 (3.7%)	280 (94.0%)	5 (1.7%)
Gemini 3.1 Pro Preview	299	2 (0.7%)	11 (3.7%)	282 (94.3%)	4 (1.3%)
GPT-5.5	299	1 (0.3%)	11 (3.7%)	282 (94.3%)	5 (1.7%)
Qwen 3.6 Plus	259	2 (0.8%)	13 (5.0%)	239 (92.3%)	5 (1.9%)

TABLE XVII: Adoption-readiness distribution by LLM configuration.

Source-file characteristic	High-risk share	Broken-caller share	False behavioural-equivalence share
LOC	$\rho=0.317$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.263$ , $p_{\mathrm{Holm}}=0.006$	$\rho=0.369$ , $p_{\mathrm{Holm}}<0.001$
Baseline smell count	$\rho=0.154$ , $p_{\mathrm{Holm}}=0.119$	$\rho=0.261$ , $p_{\mathrm{Holm}}=0.006$	$\rho=0.446$ , $p_{\mathrm{Holm}}<0.001$
Fan-in	$\rho=0.341$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.324$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.112$ , $p_{\mathrm{Holm}}=0.521$
Fan-out	$\rho=0.361$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.181$ , $p_{\mathrm{Holm}}=0.080$	$\rho=0.224$ , $p_{\mathrm{Holm}}=0.029$
Public/protected API count	$\rho=0.305$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.282$ , $p_{\mathrm{Holm}}=0.003$	$\rho=0.310$ , $p_{\mathrm{Holm}}<0.001$
Caller-used API count	$\rho=0.341$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.404$ , $p_{\mathrm{Holm}}<0.001$	$\rho=0.232$ , $p_{\mathrm{Holm}}=0.026$

TABLE XVIII: File-level associations between pre-generation source-file characteristics and the three RQ5 outcomes (

N=150

files).

Risk-rule mechanism	High-risk candidates	Share of 810 High-risk candidates	Sole High-producing mechanism
Caller-used API affected	597	73.7%	114
Public/protected signature changed	231	28.5%	0
Public API removed	82	10.1%	0
Constructor signature changed	138	17.0%	6
Package changed	0	0.0%	0
Large dependency delta ( $\mathit{depDelta}\geq 5$ )	662	81.7%	150

TABLE XIX: Risk-rule mechanisms among High-risk candidates with machine-readable risk reasons.

Automated Code Refactoring

See all 16 papers

Aug 21, 2026

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Computing, University of Derby, Derby, United Kingdom

Automated Code Refactoring Automated Software Engineering Agentic Code Generation

Ask PDF

Aug 2, 2026

RefactorAssist: Agentic Refinement for Reliable Code Refactoring

Code refactoring aims to enhance the internal structure of source code without affecting its functional behavior. The recent advancements of Large Language Models (LLMs) have demonstrated potential for automating software engineering tasks, such as code refactoring. However, the refactorings produced by LLMs often introduce subtle errors, leading to functional behavior changes and failed unit tests, which limit their practical adoption. To address the limitations of LLM-generated refactorings, we analyze the root causes of their failures and develop the RefactorAssist agent to improve the functional correctness of LLM-generated refactorings. To this end, we use 10 open-source Java projects with their native test suites and manually evaluate why LLM-generated refactorings fail unit tests. We then design an agentic approach that leverages unit-test logs, error explanations, project context retrieval, and code diffs to guide the iterative refactoring. Our findings show that the main reasons for failure are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). To make our approach cost-effective, RefactorAssist first applies a static repair step for missing imports, unbalanced brackets, and compilation errors without LLMs. For remaining failures, RefactorAssist incorporates error logs and code diffs, achieving up to a 70.8% repair rate on the remaining failures and a 94.2% cumulative pass rate under the best-performing configuration. These results indicate that static checks and test-guided, context-aware agentic repair can increase the reliability of LLM-generated refactorings, bringing them closer to practical integration within developer workflows.

Jonathan Cordeiro, Shayan Noei, Ying Zou

Queen’s University, Kingston, Canada

LLM Self-Refinement Automated Program Repair Agentic Code Generation Automated Software Engineering Automated Code Refactoring

Ask PDF

Mar 4, 2026

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.

Alex Thillen, Niels Mündler, Veselin Raychev +1

Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland

Automated Code Refactoring Automated Software Engineering LLM Agent Evaluation Code Generation Benchmarks Repository-Level Code Generation

Ask PDF

Abstract

Automated Code Refactoring

Explore similar work

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

RefactorAssist: Agentic Refinement for Reliable Code Refactoring

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?