CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Alex Thillen; Niels Mündler; Veselin Raychev; Martin Vechev

cs.SEMar 4, 2026

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Authors: Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev

Organizations: Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland

Abstract

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.

Figures & tables

Figure 1: Overview of the CodeTaste benchmark pipeline. The workflow follows six key stages: (1) Commit Discovery from GitHub; (2) Generation of Task Descriptions; (3) Build Environment, which creates a reproducible execution environment for tests and agent inference; (4) Rule Generation of static analysis rules that capture refactoring intent; (5) Inference, where the agent generates a patch; and (6) Evaluation of generated patches against static analysis rules and test suites.

Figure 2: Data funnel for obtaining the CodeTaste benchmark instances. In each step of the main commit pipeline, we filter for the top

n

entries using (1) keywords, (2) complexity checks, (3) LLM assessment. From the resulting entries, we extract 100 instances for which our agentic setup produces (4) a working execution environment and (5) valid static analysis rules.

Panel of Figure 3 — Figure 3: Example additive rule $\gamma^{+}$ and reductive rule $\gamma^{-}$ from a migration away from a centralized selector helper toward domain-specific store hooks. Lower panels show two distinct candidate patches, $\hat{X}_{1}$ and $\hat{X}_{2}$ , for the same initial repository state. Colored tokens show metavariable bindings, e.g., $HOOK denotes the bound store-hook name, while red / green line highlights show line coverage$ \mathcal{C} $. In the shown excerpts,$ \Omega(\gamma^{-},R)={\omega^{-}*{1}} $and$ \Omega(\gamma^{+},R\circ\hat{X}*{i})={\omega^{+}*{1}} $for$ i\in{1,2} $, so each displayed rule application has match count$ \mathcal{M}=1 $; the additive witness in$ \hat{X}*{1}$ spans several added lines.

Figure 4: The 100 instances of CodeTaste span 6 different programming languages, focusing on JavaScript, TypeScript, and Go.

		Mean	Max
Task Prompts	Instructed Track	1921.02	3451.0
(# Characters)	Open Track	38.00	68.0
Evaluation	Total Tests	1638.53	12449
	Additive Rules	17.94	70
	Reductive Rules	63.41	120
	Total Rules	81.35	160

Figure 5: Summary statistics for CodeTaste instance metrics. Each instance is assessed by a significant number of rules and requires considerable file changes.

Figure 6: The Open Track of CodeTaste has three distinct inference modes: Direct , in which the agent directly performs the refactoring, Plan , in which the agent drafts a plan before execution, and Multiplan , in which an oracle picks the best-aligned self-drafted plan of the model before execution.

Figure 7: Sketch of the evaluation metrics for CodeTaste .

Panel of Figure 8 — Figure 8: (Left) On the Instructed Track, GPT-5.2 significantly outperforms other models across Ifr , Pass , and $\mathcal{A}$ . In terms of $\mathcal{A}$ and Pass , Qwen3 significantly underperforms. All models achieve roughly human-level Prec . (Right) In the Open Track, the propose-then-implement strategy (Plan) and oracle selection (Oracle Multiplan) improve Ifr and $\mathcal{A}$ for GPT-5.2 . Functional correctness Pass is rarely impaired, resulting in similar $\mathcal{A}$ as Ifr . Across all agents, performance is far from human refactoring choices.

Benchmark	# Languages	Mean $\|F_{X^{*}}\|$	Mean $\|L_{X^{*}}\|$	Intent Alignment	Tests	Static Checks
MiniCode [ 20 ]	$1$	– 1	– 1	✗	✓	✗
Aider refactor benchmark [ 1 ]	$1$	$1.0$	– 2	✗	✗	✓
Can It Edit? [ 6 ]	$1$	$1.0$		✗	✓	✗
RefactorMirror [ 25 ]	$1$	$1.0$		✓	✓	✗
SWE-Refactor [ 40 ]	$1$			✗	✓	✓
SWE-bench [ 19 ]	$1$	$1.7$	$32.8$	✗	✓	✗

Table 1: Comparison of related benchmarks by scope and evaluation. Intent Alignment denotes whether the benchmark evaluates how well a generated patch recovers a human refactoring intent that is not provided to the agent. Tests denote execution-based behavior-preservation checks. Static checks denote task-specific structural or refactoring oracles, excluding generic metrics, parse checks, and coverage diagnostics. Λ Statistics computed by us. 1 Data is unavailable. 2 Lacks reference solution

X^{*}

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Human label	Gemini keeps rule	Gemini filters rule
Non-restrictive	14	1
Restrictive	5	10
Agreement	24/30 (80%)
Cohen’s $\kappa$	0.60

Appendix

Table 2: Human audit of restrictive additive rules.

99designs/gqlgen	Automattic/wp-calypso	DataDog/datadog-agent
NationalSecurityAgency/ghidra	PrestaShop/PrestaShop	TanStack/router
actualbudget/actual	ansible/ansible	antvis/G6
apache/flink	apache/hadoop	apache/nuttx
apache/pinot	apache/pulsar	apache/shardingsphere
apache/shenyu	aquasecurity/tracee	aws/aws-cli
badges/shields	bentoml/BentoML	bevyengine/bevy

Appendix

Table 3: Complete list of used repositories.

		GPT-5.2			GPT-5.1 M			Sonnet 4.5			MiniMax M2.7			Qwen3
		$\textsc{Ifr}^{+}$	$\textsc{Ifr}^{-}$	Pass	$\textsc{Ifr}^{+}$	$\textsc{Ifr}^{-}$	Pass	$\textsc{Ifr}^{+}$	$\textsc{Ifr}^{-}$	Pass	$\textsc{Ifr}^{+}$	$\textsc{Ifr}^{-}$	Pass	$\textsc{Ifr}^{+}$	$\textsc{Ifr}^{-}$	Pass
Instructed Track		89.6	88.7	76.0	75.6	69.0	47.0	79.0	62.6	43.0	75.2	58.9	49.0	56.6	30.6	30.0
Open Track	Direct	9.7	10.3	87.0	7.5	9.7	79.0	10.8	12.1	66.0	7.3	11.5	71.0	4.4	6.7	58.0
Open Track	Plan	11.2	17.2	87.0	9.8	15.1	66.0	16.8	21.7	57.0	10.5	10.8	63.0	4.1	8.8	35.0
Open Track	Multiplan	17.8	26.7	81.0	9.7	14.2	58.0	17.4	18.0	50.0	10.4	15.9	58.0	5.3	7.0	43.0

Appendix

Table 4: Full experimental results across all tracks and models in percents on

\textsc{Ifr}^{+}

(additive instruction-following rate),

\textsc{Ifr}^{-}

(reductive instruction-following rate), and Pass (test pass rate).

Pass (%)

\mathcal{A}

(%)

\mathcal{A}^{+}

(%)

\mathcal{A}^{-}

(%)

Ifr (%)

Ifr + (%)

Ifr Γ (%)

GPT-5.2

Instructed

76.0

\pm

8.5

69.6

\pm

8.2

70.7

\pm

8.6

69.6

\pm

8.4

89.0

\pm

3.3

89.6

\pm

4.0

88.7

\pm

4.0

Open

Direct

87.0

\pm

6.7

7.9

\pm

4.0

6.1

\pm

3.2

7.9

\pm

4.2

10.6

\pm

4.5

9.7

\pm

4.3

10.3

\pm

4.7

Open

Plan

87.0

\pm

6.7

15.1

\pm

5.6

8.9

\pm

4.5

16.1

\pm

6.0

16.4

\pm

5.6

11.2

\pm

5.0

17.2

\pm

6.0

Open

Multiplan

81.0

\pm

7.8

20.6

\pm

6.1

12.9

\pm

4.9

22.2

\pm

6.7

25.5

\pm

6.3

17.8

\pm

5.7

26.7

\pm

6.8

GPT-5.1 Codex Mini

Appendix

Table 5: Full experimental results across all tracks and models for Pass ,

\mathcal{A}

, Ifr in percents with 95 % confidence intervals.

		Prec (%)		Prec + (%)		Prec Γ (%)		$\|L_{\hat{X}}\|$	$\|L_{\hat{X}}^{+}\|$	$\|L_{\hat{X}}^{-}\|$
GPT-5.2
Instructed		48.4	$\pm$ 5.0	41.2	$\pm$ 6.3	56.0	$\pm$ 6.2	1978.4	974.0	1004.5
Open	Direct	16.7	$\pm$ 5.0	7.6	$\pm$ 4.1	27.0	$\pm$ 7.6	632.0	303.8	328.2
Open	Plan	15.0	$\pm$ 4.4	6.1	$\pm$ 3.6	27.1	$\pm$ 7.0	1671.5	718.9	952.7
Open	Multiplan	19.8	$\pm$ 4.8	8.8	$\pm$ 4.0	36.8	$\pm$ 7.6	1539.1	811.4	727.8
GPT-5.1 Codex Mini

Appendix

Table 6: Full experimental results across all tracks and models for Prec in percents with 95 % confidence intervals. We also report average filtered LoC edited

|L_{\hat{X}}|

in patches.

Setting	C	Go	Java	JavaScript	Python	Rust
Instructed	$57.5\pm 11.8$	$34.4\pm 3.4$	$34.7\pm 5.0$	$35.6\pm 3.6$	$30.2\pm 4.9$	$44.4\pm 7.5$
Open / Direct	$29.7\pm 11.2$	$6.4\pm 1.4$	$3.7\pm 1.3$	$5.9\pm 1.5$	$5.2\pm 2.2$	$1.5\pm 1.1$
Open / Plan	$29.0\pm 10.6$	$7.8\pm 1.5$	$1.8\pm 1.1$	$9.6\pm 1.9$	$8.4\pm 2.8$	$10.4\pm 4.3$
Open / Multiplan	$18.9\pm 9.7$	$8.7\pm 1.7$	$5.8\pm 1.9$	$9.8\pm 2.0$	$8.7\pm 2.8$	$9.8\pm 3.6$

Appendix

Table 7: Mean alignment score by primary programming language. We report mean

\pm

standard error over model-instance runs; lower scores indicate more difficult alignment. C includes C/C++ instances and has 3 benchmark instances.

Metric	Track	Mode	GPT-5.2	GPT-5.1 M	Sonnet 4.5	MiniMax M2.7	Qwen3
Ifr	Instructed		4/0	2/1	1/1	1/2	0/4
	Open	Direct	0/0	0/0	1/0	0/0	0/1
	Open	Plan	1/0	1/1	3/0	0/1	0/3
	Open	Multiplan	4/0	1/1	1/1	1/1	0/4
Pass	Instructed		4/0	1/1	1/1	1/1	0/4
	Open	Direct	3/0	2/0	0/2	0/1	0/2

Appendix

Table 8: Pairwise significance summary. Each model cell reports corrected-significant wins/losses against the other models in the same metric, track, and mode.

Panel of Figure 9 — Figure 9: Example OpenGrep rule $\gamma$ with symbolic propagation on repository $R=\{f\}$ . Colored tokens show metavariable bindings, e.g., $THEME denotes the bound theme object, while green line highlights show line coverage$ \mathcal{C} $. The witnesses are$ \Omega(\gamma,R)={\omega_{1},\omega_{2},\omega_{3}} $, so the match count is$ \mathcal{M}(\gamma,R)=3 $; the coverage contains two source lines because$ \omega_{2} $and$ \omega_{3}$ occur on the same line.

Panel of Figure 10 — Figure 10: Commit message scoring prompt used as a late-stage filter in the data scraping funnel.

Panel of Figure 11 — Figure 11: Diff scoring prompt used to rank candidate commits by refactoring quality.

Panel of Figure 12 — Figure 12: Setup Prompt for Environment Configuration

Panel of Figure 13 — Figure 13: Excerpt of the standardized scripts generated during the setup phase for the boa-dev/boa instance (base commit ac9eb4bc ).

Panel of Figure 14 — Figure 14: Task description generation prompt used to synthesize an issue-style refactoring task from commit associated metadata.

Panel of Figure 15 — Figure 15: Prompt used to derive the open-track instruction by abstracting away actionable detail from an instructed task description.

Panel of Figure 16 — Figure 16: Prompt used to generate a single concrete refactoring plan from an open-track instruction.

Panel of Figure 17 — Figure 17: Prompt used to generate multiple distinct candidate refactoring plans from an open-track instruction.

Panel of Figure 18 — Figure 18: Prompt used by the oracle judge to select the candidate plan that best matches the original refactoring description.

Panel of Figure 19 — Figure 19: Default Prompt used to run inference on an agent.

Panel of Figure 20 — Figure 20: Prompt used to run claude-code to assist evaluation of the different tracks in the case studies.

Automated Code Refactoring

See all 16 papers

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring
Aug 21, 2026Muhammad Waseem, Aakash Ahmad, Pekka AbrahamssonAutomated Code Refactoring Automated Software Engineering
Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Computing, University of Derby, Derby, United Kingdom
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Aug 10, 2026Yuling Shi, Jinghan Xu, Kelin Fu +12AI Coding Agents Software Engineering Benchmarks
Shanghai Jiao Tong University · Peking University · Douyin Group +4
RefactorAssist: Agentic Refinement for Reliable Code Refactoring
Aug 2, 2026Jonathan Cordeiro, Shayan Noei, Ying ZouLLM Self-Refinement Automated Program Repair
Queen’s University, Kingston, Canada

Aug 21, 2026J/K move · Enter open · S save

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Computing, University of Derby, Derby, United Kingdom

Automated Code Refactoring Automated Software Engineering Agentic Code Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

Abstract

Automated Code Refactoring

Explore similar work

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

RefactorAssist: Agentic Refinement for Reliable Code Refactoring