SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

Deyao Hong; Yizhe Chi; Wenyi Li; Xiaoqiu Wang; Mingju Gao; Kaisen Yang; Bingxiang He; Youjie Zheng

cs.CLAug 24, 2026

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

Authors: Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng,

Organizations: Navers Lab, Einsia.AI · Tsinghua University

Abstract

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ( $5.4\%$ ) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$ . Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$ . Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

Figures & tables

Figure 1: SWE Refactor Bench at a glance. Top: how a migration gets scored. Left, prior benchmarks: one stage, a behavioural test suite. With no migration audit in front of it, a submission that never migrated anything still turns every check green, so the suite cannot tell a completed migration from an untouched repository. Right, SWE Refactor Bench: three stages in series—Migration Audit asks whether the migration happened at all, Behavioural Tests is that same fixed suite, and Agentic Verification sends six coding agents, one hour each, after the differences the suite was never written to catch. Bottom left: the 20 tasks span four kinds of technical debt—language, framework, platform, and build toolchain. Bottom right: composite scores of 8 frontier models, each in its strongest configuration (out of 100; defined in ( 4 )).

Debt class	Representative migrations	Tasks	Source LoC	Budget	Criteria	Modules	Checks
				(h)	(Stage I)	(Stage II)	(Stage II)
Language	C $\to$ Rust, C $\to$ Java, Go $\to$ Zig	7	4.4 k–39.8 k	12–30	60	117	59,771
Framework	Flask $\to$ Starlette, Gin $\to$ chi, Vue $\to$ React	7	0.8 k–94.8 k	6–16	39	79	55,852
Platform	POSIX $\to$ wasm32-wasi , Node $\to$ V8 realm	3	16.0 k–358.0 k	6–10	19	31	6,725
Build toolchain	Autotools $\to$ CMake, Maven $\to$ Gradle	3	19.0 k–78.8 k	6	18	37	7,770
Total	20 upstream projects, 20 tasks	20	867,062	262	136	264	130,118

Table 1: Composition of the SWE Refactor Bench task set. Every task requires a real open-source repository to be migrated in its entirety onto another stack, with its interface and observable behaviour unchanged. Budget is the time limit given to the agent; criteria are the Stage I migration questions, modules are the independent test modules of Stage II, and checks are the behavioural cases those modules hold.

Figure 2: On the left, what the agent sees : a real repository on the source stack, an instruction, an offline image, and the process it works through on its own—read the repository and the requirements, rewrite onto the target stack, get it to build, then check itself against the original. On the right, the three-stage evaluation it never touches : Migration Audit checks whether the migration actually happened, Behavioural Tests checks whether the fixed behaviour is preserved exactly, and Agentic Verification goes looking for whatever differences remain beyond the fixed tests.

Model	Harness	Effort	Behavioural pass $\uparrow$	Score $\uparrow$	Cost $\downarrow$
Model	Harness	Effort	%	out of 100	USD / task
Claude Opus 5	Claude Code	low	92.5	20.5	34.8
Claude Opus 5	Claude Code	medium	92.7	28.5	38.4
Claude Opus 5	Claude Code	high	92.8	34.5	55.7
Claude Opus 5	Claude Code	xhigh	92.8	47.0	74.9
Claude Opus 5	Claude Code	max	91.9	31.0	72.4

Table 3: Behavioural pass rate, score and cost for the same runs. Behavioural pass rate is the row mean of the fraction of Behavioural Tests fixed checks passed (a perfect score is not required), score is defined in ( 4 ), and cost is the average API spend per task. Harness identifies the coding-agent client: GPT-series models use Codex, and all others use Claude Code. Shading encodes only the magnitude of a value within its column; each model’s best-scoring row is marked with a shaded background .

			Migrated $\uparrow$		All tests pass $\uparrow$	Accepted $\uparrow$	Broken $\downarrow$	Blindness $\downarrow$	Score $\uparrow$
Class	Tasks	Runs	$n$	%	$n$	$n$	%	$n$	mean
Build toolchain	3	78	63	80.8	36	6	82.4	2	31.4
Platform port	3	78	45	57.7	21	4	76.5	4	17.2
Framework rewrite	7	182	132	72.5	40	14	44.0	15	12.0
Language rewrite	7	182	100	54.9	21	4	66.7	9	5.6

Table 4: Summary by debt class, pooling all

26

configurations. Column meanings follow the metric definitions of Section 3.1 ; shading follows Table 3 . Stage I pass rate and acceptance count do not move together : build toolchain is the easiest class to get past Stage I and the one that loses most at Stage III; framework rewrites contribute

14

of the

28

acceptances; language rewrites score lowest.

Figure 3: (a) Where each model sits on two axes, both of them shares of the same runs: horizontally “does it still behave like the original” (a perfect Stage II), vertically “was the migration done at all” (Stage I). (b) The same two axes, per task, with marker area the number of accepted submissions. A migration has to satisfy both conditions, but they are satisfied by different runs, and most runs manage at most one.

Figure 4: The four kinds of submission Stage I has to tell apart. Top left, nothing rewritten : the new file is the C copied over verbatim, only the suffix changed. Top right, wrapped : the Rust side is an extern "C" forwarding shim and the original C still does the work. Bottom left, half done : some functions are genuinely rewritten while others still call back into the old implementation. Bottom right, rewritten wrongly : really rewritten, but the behaviour changed. The first two pass every behavioural test, and only an instrument that reads mechanism can reject them.

Figure 5: (a) Among the

340

runs that did complete the migration, the share reaching each level on the fixed suite. (b) Among the

88

submissions that passed every fixed check, how many of the six independent verifiers (one hour each) failed to find a behavioural difference. Only the rightmost bar is an accepted migration.

		Migration Audit $\uparrow$		Behavioural Tests $\uparrow$		Agentic Verification $\uparrow$		Final $\uparrow$
Category	Runs	$n$	%	$n$	%	$n$	%	Acc.	Score
Build toolchain	78	63/78	80.8	34/63	54.0	6/34	17.6	6	31.4
Platform port	78	45/78	57.7	17/45	37.8	4/17	23.5	4	17.2
Framework	182	132/182	72.5	25/132	18.9	14/25	56.0	14	12.0
Language	182	100/182	54.9	12/100	12.0	4/12	33.3	4	5.6
All	520	340/520	65.4	88/340	25.9	28/88	31.8	28	13.4

Table 5: Agent success varies across stages and migration categories. Every cell is the share of the runs that entered that stage and passed it, so the three multiply out to the category’s final acceptance rate. The Stage II column is taken over the runs that cleared Stage I (a blindness run passes every fixed test but is already out at Stage I, so it does not count); the Stage III column is taken over the runs that cleared both earlier stages and actually met the verifiers. Agents do not perform best on one category throughout the pipeline : they achieve their highest pass rates on build toolchain rewrites at the first two stages but their lowest at the last; framework rewrites show the reverse pattern.

			Called a real migration		Judge against human
Task	Class	runs	by human	by judge	agree $\uparrow$	too strict $\downarrow$	too lenient $\downarrow$
lang01 (cmark)	Language	26	7	6	25	1	0
lang03 (sqlparse)	Language	26	15	14	23	2	1
fw02 (json-server)	Framework	26	15	13	24	2	0
fw06 (uploadserver)	Framework	26	15	13	22	3	1
pf02 (Stylus)	Platform	26	14	12	24	2	0

Table 6: Task-by-Task Comparison ofMigration Audit and Independent Human Annotation. Each row is one task, attempted once by every one of the

26

model–effort configurations. Human and judge are how many of those runs each side called a genuine migration; agree , too strict and too lenient sort the same runs by whether the two verdicts matched, the judge zeroed a run the human accepted, or the reverse.

Figure 6: Break rate of each of the six verifiers : the share of its rounds in which it handed in an executable counterexample that passes on the original and fails on the submission, coloured by the model behind it and annotated with its reasoning-effort setting. The dashed line marks the lower of the two claude-opus-5 verifiers.

Benchmark	Unit of work	Starts failing	Stack changes	Migration criterion	Agentic verifier
SWE-bench [ 33 ]	one issue	$\checkmark$	$\times$	$\times$	$\times$
SWE-EVO [ 38 ]	one feature	$\checkmark$	$\times$	$\times$	$\times$
SWE-CI [ 10 ]	CI history	$\checkmark$	$\times$	$\times$	$\times$
Terminal-Bench [ 47 ]	one task	$\checkmark$	$\times$	$\times$	$\times$
TransCoder [ 53 ]	one function	$\checkmark$	$\checkmark$	$\times$	$\times$
MigrationBench [ 44 ]	whole repository	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 7: Positioning against related benchmarks. Starts failing means whether the starting state already fails some test in the suite being scored; that is what makes “red to green” an informative verdict—and it is exactly the property SWE Refactor Bench cannot have. Migration criterion means whether a non-behavioural instrument can reject a submission that passed every behavioural check; agentic verifier means whether, at scoring time, further models actively search for differences beyond the fixed tests that the authors did not anticipate.

\checkmark

is yes,

\checkmark

is partial—MigrationBench moves a language version, Java 8 to 17/21, while the language itself stays—and

\times

marks a dimension the benchmark does not aim at, not a defect.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage I — migration criteria				Stage II — test modules
Criterion	What it asks	Required	Verdict	Module	What it measures	Pass rate
no-c-sources	has C left the build	$\checkmark$	pass	build	two link configurations	$1.00$
no-foreign-headers	is only cmark.h left	$\checkmark$	pass	structure	install tree, ABI	$1.00$
no-c-in-build	is no C compiled	$\checkmark$	pass	conformance	CommonMark corpus	$1.00$
rust-present	is Rust really there	$\checkmark$	pass	renderers	five output renderers	$1.00$
rust-is-primary	is Rust the implementation	$\checkmark$	fail	data-tables	entities, case folding	$1.00$

Appendix

Table 8: One task and one submission in full: claude-opus-5 at high effort on lang01-cmark-c-to-rust . All

4{,}184

checks across

15

modules pass, the highest mark Stage II can give; seven of the eight criteria pass, and the eighth, rust-is-primary , fails unanimously because the Rust reproduces the C implementation layout field by field— “the same hand-managed pointer stack, the same names and the same order … this is the original transliterated into unsafe Rust” , cited to src/inlines.rs:55 and neighbouring lines. Final score:

0

			Migrated $\uparrow$	All tests pass $\uparrow$	Blindness $\downarrow$	Accepted $\uparrow$	Score $\uparrow$
			Stage I passed	Stage II perfect	passed, not migrated	all three passed	out of 100
Task	Migration	$n$	$n$	$n$	$n$	$n$	mean
Language rewrites
lang01	C $\to$ Rust	26	6	5	5	0	0.00
lang02	C $\to$ Java	26	18	4	0	0	12.69
lang03	Python $\to$ Go	26	14	0	0	0	0.00

Appendix

Table 9: The

520

runs broken down by task. Every task is run once by each of the

26

configurations (

n=26

). Column meanings follow the metric definitions of Section 3.1 : migrated is the number of runs passing Migration Audit, all tests pass is the number passing every fixed check in Behavioural Tests ( regardless of the Stage I verdict), blindness is the subset of those that failed Stage I, accepted is the number passing all three stages, and score is defined in ( 4 ). All tests pass

-

blindness is therefore the number of runs that reached Stage III. Only

7

of the

20

tasks ever produced an accepted submission .

Task	Upstream project	Source stack	Target stack	LoC	$B$ (h)	Criteria	Modules	Checks
Language rewrites—the implementation language moves, the artifact does not
lang01	cmark 0.31.1	C11	Rust 1.90	22,619	30	8	15	4,184
lang02	zlib 1.3.1	C89	Java 17	22,732	20	10	18	4,162
lang03	sqlparse 0.5.3	Python	Go (stdlib)	4,393	15	8	15	8,530
lang04	acorn 8.14.0	JavaScript	Rust 1.90	9,571	30	7	13	16,039
lang05	go-yaml v3.0.1	Go	Zig 0.14.1	11,967	20	8	22	10,271

Appendix

Table 10: The task set, task by task. LoC counts newlines in the version-controlled implementation source of State A, excluding tests, vendored trees and documentation, measured the same way for every task rather than transcribed from each project’s own description; together these repositories hold

10{,}594

version-controlled files.

B

is the agent’s time budget in hours. Criteria is the number of prompt-form migration criteria, each of them required: a single failure zeroes the submission. Modules is the number of independent test modules and checks the number of fixed behavioural cases they hold, recorded from a reference build of State A. The size of a test suite is set by how much of the product is mechanically observable, not by the size of the repository. Every task has six verifiers of one hour each. Table 1 summarises these rows by class. Each row ships an upstream release verbatim, at the version shown and under its own licence: cmark [ 45 ] , zlib [ 21 ] , sqlparse [ 2 ] , Acorn [ 55 ] , go-yaml [ 9 ] , Jsonnet [ 25 ] , JSONata [ 30 ] , httpbin [ 52 ] , json-server [ 60 ] , the RealWorld Conduit app [ 58 ] , ChartMuseum [ 56 ] , miniserve [ 29 ] , go-simple-upload-server [ 1 ] , GraphHopper [ 27 ] , SQLite [ 59 ] , Stylus [ 3 ] , QuickJS [ 5 ] , libsodium [ 17 ] , Gson [ 26 ] and PyCryptodome [ 57 ] .

Automated Software Engineering

Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.

See all 81 papers

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Aug 10, 2026Yuling Shi, Jinghan Xu, Kelin Fu +12AI Coding Agents Software Engineering Benchmarks
Shanghai Jiao Tong University · Peking University · Douyin Group +4
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Jul 8, 2026Wenqi Huang, Charley Lee, Leonard Tng +1AI Coding Agents Software Engineering Benchmarks
1Datacurve
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI

Aug 10, 2026J/K move · Enter open · S save

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng+11

Shanghai Jiao Tong University · Peking University · Douyin Group+4

AI Coding Agents Software Engineering Benchmarks Automated Code Refactoring

Open paper AskPDF arXiv

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

Abstract

Automated Software Engineering

Explore similar work

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark