LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

Tongwei Deng; Jinyu Zhu; Yue Cao; Chenxu Wang; Yanzhen Zou; Bing Xie

cs.SEOct 14, 2026

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

Authors: Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie

Organizations: School of Computer Science, Peking University · Key Lab of HCST (PKU), MOE; SCS, Peking University · School of Software Engineering, Beijing Jiaotong University

Abstract

Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.

Figures & tables

Figure 1: The LintEval framework for linter-guided file refactoring. The pipeline systematically extracts lint alerts ( $L$ ) from functionally correct programs ( $A$ ) across multiple programming languages to evaluate LLM-driven revision. The resulting refactored file ( $A^{\prime}$ ) is evaluated across four dimensions, where quality metrics are strictly conditionalized on functionality preservation to ensure new alert avoidance and edit minimality.

Metric	Value
Unique problems	3,999
Samples	13,061
Total lint alerts	171,662
Avg. alerts per sample	13.1
Avg. LOC per sample	61.0
Unique alert types	396

Table 2: Dataset statistics for LintEval . The dataset contains 13,061 competitive-programming samples across three languages, each annotated with lint alerts.

Figure 2: Distribution of lint alert categories (

N{=}171{,}662

alerts) and programming languages (

N{=}13{,}061

samples) in LintEval.

Model	Pass@1 ( $\uparrow$ )	Reduction@P ( $\uparrow$ )	New@P ( $\downarrow$ )	$\Delta$ Line@P ( $\downarrow$ )	LinePerFix@P ( $\downarrow$ )
Copy-Original	100.0%	0.0%	0.0%	0.0	–
Rule-Based Fixers	76.3%	25.9%	0.6%	71.4	51.7
GPT-5.2	67.1%	87.8%	7.3%	71.7	16.9
GPT-5.4-mini	68.3%	90.4%	5.6%	60.2	14.2
DeepSeek-v3.2	57.6%	73.0%	6.7%	45.4	13.8
Qwen2.5-Coder-32B	61.7%	57.1%	8.1%	33.3	12.6

Table 3: Overall results on LintEval . Reduction@P, New@P,

\Delta

Line@P, and LinePerFix@P are computed only over passed instances. Best results per column are highlighted in boldface .

	Pass Cases				Failure Cases
Model	Reduction@P	New@P	$\Delta$ Line@P	LinePerFix@P	Reduction@F	New@F	$\Delta$ Line@F	LinePerFix@F
GPT-5.2	87.8%	7.3%	71.7	16.9	86.8% $\downarrow$	6.3% $\downarrow$	65.5 $\downarrow$	17.1 $\uparrow$
GPT-5.4-mini	90.4%	5.6%	60.2	14.2	89.2% $\downarrow$	4.6% $\downarrow$	69.0 $\uparrow$	13.9 $\downarrow$
DeepSeek-v3.2	73.0%	6.7%	45.4	13.8	41.6% $\downarrow$	6.5% $\downarrow$	23.6 $\downarrow$	12.4 $\downarrow$
Qwen2.5-Coder-32B	57.1%	8.1%	33.3	12.6	45.3% $\downarrow$	13.7% $\uparrow$	30.4 $\downarrow$	10.4 $\downarrow$
Qwen2.5-Coder-0.5B	-1.2%	10.5%	17.2	9.5	-169.1% $\downarrow$	11.7% $\uparrow$	13.4 $\downarrow$	131.7 $\uparrow$

Table 4: Editing behavior of passed and failed refactoring attempts. @P metrics are computed over

S_{\text{pass}}

; @F metrics are computed over failed instances for diagnostic analysis only. Arrows indicate whether the failure-case value is higher or lower than the pass-case value. Bold marks the best value per column.

Model	Pass@1	Reduction@P
Qwen3-4B	45.8	32.0
Qwen3-8B	33.3	70.7
Qwen3.5-9B	37.5	31.3
Qwen3.6-27B	64.3	62.0
Qwen3.6-35B-A3B	65.3	61.5

Table 6: Newer Qwen checkpoints under the same single-turn, zero-shot protocol without fine-tuning. No checkpoint dominates both functionality preservation and lint alert reduction.

Figure 3: Category-wise Reduction@P for selected models. Additional category-wise results, including negative values for smaller models, are reported in Table 18 .

	With Alerts					No Alerts
Model	Pass@1	Reduction@P	New@P	$\Delta$ Line@P	LinePerFix@P	Pass@1	Reduction@P	New@P	$\Delta$ Line@P	LinePerFix@P
GPT-5.4-mini	68.3	90.4	5.6	60.2	14.2	95.4 $\uparrow$	45.4 $\downarrow$	46.8 $\uparrow$	69.0 $\uparrow$	15.5 $\uparrow$
DeepSeek-v3.2	57.6	73.0	6.7	45.4	13.8	62.5 $\uparrow$	49.9 $\downarrow$	10.1 $\uparrow$	81.6 $\uparrow$	40.5 $\uparrow$
Qwen2.5-Coder-0.5B	48.8	-1.2	10.5	17.2	9.5	60.0 $\uparrow$	-1.0 $\uparrow$	3.4 $\downarrow$	10.5 $\downarrow$	74.4 $\uparrow$
Qwen2.5-Coder-7B	53.0	24.7	12.7	32.5	12.7	64.5 $\uparrow$	9.2 $\downarrow$	3.6 $\downarrow$	31.0 $\downarrow$	42.9 $\uparrow$

Table 7: Ablation study on the impact of lint alert conditioning. With Alerts : full input with lint diagnostics; No Alerts : input without lint alert messages. Arrows in the No Alerts columns compare against the With Alerts baseline:

\uparrow

\downarrow

= improvement;

\uparrow

\downarrow

= degradation.

Model	Format	Pass@1	Reduction@P	New@P	$\Delta$ Line@P	LinePerFix@P
0.5B	Base	48.8%	-1.2%	10.5%	17.2	9.5
	A+L $\rightarrow$ A ′	37.8%	79.9%	8.4%	42.0	15.9
	Q $\rightarrow$ A	45.2%	-1.0%	2.5%	2.6	30.8
	Q $\rightarrow$ A ′	40.8%	9.1%	3.7%	11.9	16.3
3B	Base	52.0%	9.5%	11.2%	27.9	12.8
	A+L $\rightarrow$ A ′	60.4%	85.0%	6.6%	56.1	16.0

Table 8: Effect of training data format on LintEval . Three Qwen2.5-Coder sizes (0.5B, 3B, 7B) are fine-tuned with formats A+L

\rightarrow

A ′ , Q

\rightarrow

A, and Q

\rightarrow

A ′ . Bold marks the best value per model group.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Round 1 P / R	Round 3 P / R	Change R / P
GPT-5.4-mini	67.5 / 77.9	65.9 / 86.9	+9.0 / -1.6
DeepSeek-v3.2	67.1 / 71.3	63.3 / 81.3	+10.0 / -3.8
Qwen2.5-Coder-7B	56.4 / 18.7	55.2 / 20.6	+1.9 / -1.2

Appendix

Table 9: Three-round mini-swe-agent diagnostic on the same LintEval instances. P and R denote Pass@1 and Reduction@P. Each Reduction@P conditions on that round’s passing outputs.

Model	Build	Run	TO	Wrong	$\geq$ 90%	0%
GPT-5.4-mini	10.5	5.3	1.8	82.5	19.1	2.1
DeepSeek-v3.2	28.1	6.2	1.6	64.1	22.0	4.9
Qwen2.5-Coder-7B	26.3	13.2	1.3	59.2	17.8	6.7
Qwen2.5-Coder-0.5B	40.8	9.7	0.0	49.5	15.7	13.7

Appendix

Table 10: Camera-ready diagnostics (continued): failure types among non-passing revisions (%). Run and TO denote runtime errors and timeouts; the last two columns split wrong answers by test pass rate. Percentages may not sum to 100% because of rounding.

Size	Language	Pass@1 diff.	Reduction@P diff.
Qwen2.5-Coder-0.5B	C++	-22.5	+89.6
Qwen2.5-Coder-0.5B	Python	+1.5	+78.9
Qwen2.5-Coder-0.5B	Java	-5.9	+56.6
Qwen2.5-Coder-3B	C++	-4.5	+81.2
Qwen2.5-Coder-3B	Python	+23.1	+73.7
Qwen2.5-Coder-3B	Java	+11.8	+63.3

Appendix

Table 11: Camera-ready diagnostics (continued): language–size effects of A+L

\to

A ′ supervision relative to Base. Reduction@P improves in all nine cells, while Pass@1 changes are mixed.

Setting	Pass@1	Reduction@P
Qwen2.5-Coder-3B LoRA	59.0	84.0
Qwen2.5-Coder-3B full-parameter	63.7	85.8
DeepSeek-Coder-6.7B Base	25.3	19.0
DeepSeek-Coder-6.7B A+L $\to$ A ′	61.2	79.0
DeepSeek-Coder-6.7B Q $\to$ A	24.9	13.0
DeepSeek-Coder-6.7B Q $\to$ A ′	8.6	18.4

Appendix

Table 12: Camera-ready diagnostics (continued): SFT robustness across training regimes and backbones. Reduction@P conditions on each setting’s passing outputs.

Language	Linter	Rule Family	Linter Version	Compiler/Runtime
C++	clang-tidy	bugprone, performance, modernize, readability	22.1.0	g++ 13.3.0
Python	Pylint	error, performance, refactor, convention	4.0.5	Python 3.12.10
Java	PMD	Error Prone, Performance, Design, Code Style	7.12.0	OpenJDK 21.0.10

Appendix

Table 13: Language-specific linter configurations and runtime environments.

Category	Lint Alerts	C++ (clang-tidy)	Python (Pylint)	Java (PMD)
Error-prone	18,434	bugprone-*	error	Error Prone , Security
Performance	17,776	performance-*	performance	Performance , Multithreading
Refactor	17,977	modernize-*	refactor	Design
Style	115,458	readability-*	convention	Code Style , Best Practices
Other	2,017	other	other	other

Appendix

Table 14: Cross-language taxonomy of lint alert categories in LintEval. Counts show the number of alerts in each diagnostic category. Representative rule families illustrate how each category maps to language-specific linter conventions.

	C++				Python
Model	Pass@1	Reduction@P	New@P	LinePerFix@P	Pass@1	Reduction@P	New@P	LinePerFix@P
GPT-5.2	67.2%	94.9%	1.3%	23.4	64.5%	84.4%	9.7%	11.6
GPT-5.4-mini	69.7%	95.6%	0.9%	20.7	65.5%	86.5%	8.4%	9.8
DeepSeek-v3.2	68.0%	76.6%	1.7%	10.7	46.8%	70.6%	9.7%	14.5
Qwen2.5-Coder-32B	64.3%	58.6%	2.1%	12.9	58.1%	63.5%	16.2%	10.1
Qwen2.5-Coder-0.5B	63.9%	1.5%	0.1%	3.2	32.5%	-7.2%	28.2%	14.9

Appendix

Table 16: Full language-wise performance on LintEval for C++ and Python. All four metrics are reported; best results per column are in bold .

Model	Pass@1	Reduction@P	New@P	LinePerFix@P
GPT-5.2	76.5%	77.5%	17.1%	18.8
GPT-5.4-mini	72.5%	81.7%	15.1%	13.4
DeepSeek-v3.2	51.0%	63.7%	20.5%	20.0
Qwen2.5-Coder-32B	62.7%	37.2%	14.7%	22.1
Qwen2.5-Coder-0.5B	41.2%	-3.6%	31.3%	16.7
Qwen2.5-Coder-3B	49.0%	8.5%	29.6%	15.3

Appendix

Table 17: Full language-wise performance on LintEval (continued): Java. All four metrics are reported; best results per column are in bold .

	Error-prone			Performance			Refactor
Model	Pass@1	Reduction@P	New@P	Pass@1	Reduction@P	New@P	Pass@1	Reduction@P	New@P
GPT-5.2	63.3%	83.0%	0.7%	65.8%	88.9%	1.3%	62.0%	59.3%	10.5%
GPT-5.4-mini	64.9%	84.4%	0.0%	66.7%	90.3%	0.2%	60.8%	72.5%	11.0%
DeepSeek-v3.2	59.6%	46.1%	0.0%	53.9%	77.6%	0.7%	53.0%	53.6%	10.7%
Qwen2.5-Coder-32B	61.3%	33.3%	1.3%	60.2%	73.5%	4.2%	51.9%	55.6%	3.7%
Qwen2.5-Coder-0.5B	54.7%	3.5%	0.0%	52.1%	-0.4%	0.8%	49.4%	0.6%	1.8%

Appendix

Table 18: Category-wise performance for Error-prone, Performance, and Refactor alerts. Pass@1 is computed over samples having

\geq 1

alert in the category; Reduction@P and New@P aggregate over passed instances.

	Style			Other
Model	Pass@1	Reduction@P	New@P	Pass@1	Reduction@P	New@P
GPT-5.2	65.8%	91.4%	0.0%	58.0%	100.0%	0.0%
GPT-5.4-mini	66.9%	93.0%	0.0%	62.0%	100.0%	0.0%
DeepSeek-v3.2	55.5%	79.7%	1.0%	64.0%	98.4%	0.0%
Qwen2.5-Coder-32B	61.3%	59.0%	0.5%	61.7%	95.3%	0.0%
Qwen2.5-Coder-0.5B	46.5%	-0.4%	2.2%	61.7%	3.0%	0.0%

Appendix

Table 19: Category-wise performance (continued): Style and Other alerts under the same aggregation protocol as the first part.

	1 alert				2–5 alerts
Model	Pass@1	Reduction@P	New@P	LinePerFix@P	Pass@1	Reduction@P	New@P	LinePerFix@P
GPT-5.2	79.5%	64.5%	35.5%	21.9	71.7%	82.5%	10.0%	13.1
GPT-5.4-mini	82.1%	62.5%	34.4%	12.9	75.1%	83.0%	8.1%	9.1
DeepSeek-v3.2	61.5%	69.4%	13.9%	17.1	63.0%	68.1%	9.5%	9.4
Qwen2.5-Coder-32B	82.1%	46.9%	40.6%	11.2	70.5%	61.4%	16.6%	9.0
Qwen2.5-Coder-0.5B	59.0%	22.9%	11.4%	17.2	55.5%	8.7%	12.7%	8.9

Appendix

Table 20: Alert-count breakdown for files with 1 or 2–5 alerts. All four metrics are reported; best results per column are in bold .

	6–10 alerts				$>$ 10 alerts
Model	Pass@1	Reduction@P	New@P	LinePerFix@P	Pass@1	Reduction@P	New@P	LinePerFix@P
GPT-5.2	60.0%	87.2%	8.7%	13.7	64.5%	89.0%	6.3%	19.3
GPT-5.4-mini	60.0%	88.7%	6.9%	10.1	63.9%	92.1%	4.7%	17.5
DeepSeek-v3.2	54.2%	75.5%	4.6%	12.7	53.6%	73.6%	6.4%	16.3
Qwen2.5-Coder-32B	55.0%	69.4%	8.3%	11.1	52.4%	53.1%	5.7%	16.1
Qwen2.5-Coder-0.5B	46.7%	-2.8%	13.6%	11.1	41.0%	-4.1%	8.9%	8.2

Appendix

Table 21: Alert-count breakdown (continued): files with 6–10 or more than 10 alerts. The same metric definitions apply; best results per column are in bold .

	Fixed alerts by category				#Fix	#New	Fix/New
Format (3B)	E-prone	Perf	Refac	Style	Total	Actual	Ratio
Base	8	47	39	254	360	261	1.4 $\times$
A+L $\rightarrow$ A ′	229	241	138	1803	2453	189	13.0 $\times$
Q $\rightarrow$ A	6	20	14	237	281	98	2.9 $\times$
Q $\rightarrow$ A ′	12	104	14	167	304	585	0.5 $\times$

Appendix

Table 22: Per-category alert breakdown for 3B SFT models (Qwen2.5-Coder-3B). Fixed counts denote original alerts removed per category. #New is the total newly introduced alerts from the evaluation pipeline. The “Other” category is omitted (

\leq 3.3\%

of total fixed alerts). Fix/New Ratio shows how many alerts are fixed per new alert introduced.

Language	#Samples	Avg. Messages	Rule Types	Linter
C++	538	10.4	89	clang-tidy
Python	416	8.9	72	Pylint
Java	242	18.8	63	PMD
Total	1,196	11.6	224	–

Appendix

Table 23: Statistics of SFT training data. This table summarizes the revision-format training split only; it is kept in the appendix because the main text now focuses on the benchmark and evaluation tables.

Automated Code Refactoring

See all 16 papers

Aug 21, 2026

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson

Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Computing, University of Derby, Derby, United Kingdom

Automated Code Refactoring Automated Software Engineering Agentic Code Generation

Ask PDF

Mar 4, 2026

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.

Alex Thillen, Niels Mündler, Veselin Raychev +1

Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland

Automated Code Refactoring Automated Software Engineering LLM Agent Evaluation Code Generation Benchmarks Repository-Level Code Generation

Ask PDF

Oct 11, 2026

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.

Elad Sharony, Ido Greenberg, Jialin Song +3

Technion · NVIDIA

Automated Program Repair

Ask PDF

Abstract

Automated Code Refactoring

Explore similar work

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair