Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58% reach 99% of the fixed checks, yet only 26% reach 100%. Agent capability differs across migration categories: agents score 31.4 on build toolchain rewrites but only 5.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.
Figures & tables
Figure 1: SWE Refactor Bench at a glance. Top: how a migration gets scored. Left, prior benchmarks: one stage, a behavioural test suite. With no migration audit in front of it, a submission that never migrated anything still turns every check green, so the suite cannot tell a completed migration from an untouched repository. Right, SWE Refactor Bench: three stages in series—Migration Audit asks whether the migration happened at all, Behavioural Tests is that same fixed suite, and Agentic Verification sends six coding agents, one hour each, after the differences the suite was never written to catch. Bottom left: the 20 tasks span four kinds of technical debt—language, framework, platform, and build toolchain. Bottom right: composite scores of 8 frontier models, each in its strongest configuration (out of 100; defined in ( 4 )).
Debt class
Representative migrations
Tasks
Source LoC
Budget
Criteria
Modules
Checks
(h)
(Stage I)
(Stage II)
(Stage II)
Language
C → Rust, C → Java, Go → Zig
7
4.4 k–39.8 k
12–30
60
117
59,771
Framework
Flask → Starlette, Gin → chi, Vue → React
7
0.8 k–94.8 k
6–16
39
79
55,852
Platform
POSIX → wasm32-wasi , Node → V8 realm
3
16.0 k–358.0 k
6–10
19
31
6,725
Build toolchain
Autotools → CMake, Maven → Gradle
3
19.0 k–78.8 k
6
18
37
7,770
Total
20 upstream projects, 20 tasks
20
867,062
262
136
264
130,118
Table 1: Composition of the SWE Refactor Bench task set. Every task requires a real open-source repository to be migrated in its entirety onto another stack, with its interface and observable behaviour unchanged. Budget is the time limit given to the agent; criteria are the Stage I migration questions, modules are the independent test modules of Stage II, and checks are the behavioural cases those modules hold.
Figure 2: On the left, what the agent sees : a real repository on the source stack, an instruction, an offline image, and the process it works through on its own—read the repository and the requirements, rewrite onto the target stack, get it to build, then check itself against the original. On the right, the three-stage evaluation it never touches : Migration Audit checks whether the migration actually happened, Behavioural Tests checks whether the fixed behaviour is preserved exactly, and Agentic Verification goes looking for whatever differences remain beyond the fixed tests.
Model
Harness
Effort
Behavioural pass ↑
Score ↑
Cost ↓
%
out of 100
USD / task
Claude Opus 5
Claude Code
low
92.5
20.5
34.8
Claude Opus 5
Claude Code
medium
92.7
28.5
38.4
Claude Opus 5
Claude Code
high
92.8
34.5
55.7
Claude Opus 5
Claude Code
xhigh
92.8
47.0
74.9
Claude Opus 5
Claude Code
max
91.9
31.0
72.4
Table 3: Behavioural pass rate, score and cost for the same runs. Behavioural pass rate is the row mean of the fraction of Behavioural Tests fixed checks passed (a perfect score is not required), score is defined in ( 4 ), and cost is the average API spend per task. Harness identifies the coding-agent client: GPT-series models use Codex, and all others use Claude Code. Shading encodes only the magnitude of a value within its column; each model’s best-scoring row is marked with a shaded background .
Migrated ↑
All tests pass ↑
Accepted ↑
Broken ↓
Blindness ↓
Score ↑
Class
Tasks
Runs
n
%
n
n
%
n
mean
Build toolchain
3
78
63
80.8
36
6
82.4
2
31.4
Platform port
3
78
45
57.7
21
4
76.5
4
17.2
Framework rewrite
7
182
132
72.5
40
14
44.0
15
12.0
Language rewrite
7
182
100
54.9
21
4
66.7
9
5.6
Table 4: Summary by debt class, pooling all 26 configurations. Column meanings follow the metric definitions of Section 3.1 ; shading follows Table 3 . Stage I pass rate and acceptance count do not move together : build toolchain is the easiest class to get past Stage I and the one that loses most at Stage III; framework rewrites contribute 14 of the 28 acceptances; language rewrites score lowest.
Figure 3: (a) Where each model sits on two axes, both of them shares of the same runs: horizontally “does it still behave like the original” (a perfect Stage II), vertically “was the migration done at all” (Stage I). (b) The same two axes, per task, with marker area the number of accepted submissions. A migration has to satisfy both conditions, but they are satisfied by different runs, and most runs manage at most one.
Figure 4: The four kinds of submission Stage I has to tell apart. Top left, nothing rewritten : the new file is the C copied over verbatim, only the suffix changed. Top right, wrapped : the Rust side is an extern "C" forwarding shim and the original C still does the work. Bottom left, half done : some functions are genuinely rewritten while others still call back into the old implementation. Bottom right, rewritten wrongly : really rewritten, but the behaviour changed. The first two pass every behavioural test, and only an instrument that reads mechanism can reject them.
Figure 5: (a) Among the 340 runs that did complete the migration, the share reaching each level on the fixed suite. (b) Among the 88 submissions that passed every fixed check, how many of the six independent verifiers (one hour each) failed to find a behavioural difference. Only the rightmost bar is an accepted migration.
Migration Audit ↑
Behavioural Tests ↑
Agentic Verification ↑
Final ↑
Category
Runs
n
%
n
%
n
%
Acc.
Score
Build toolchain
78
63/78
80.8
34/63
54.0
6/34
17.6
6
31.4
Platform port
78
45/78
57.7
17/45
37.8
4/17
23.5
4
17.2
Framework
182
132/182
72.5
25/132
18.9
14/25
56.0
14
12.0
Language
182
100/182
54.9
12/100
12.0
4/12
33.3
4
5.6
All
520
340/520
65.4
88/340
25.9
28/88
31.8
28
13.4
Table 5: Agent success varies across stages and migration categories. Every cell is the share of the runs that entered that stage and passed it, so the three multiply out to the category’s final acceptance rate. The Stage II column is taken over the runs that cleared Stage I (a blindness run passes every fixed test but is already out at Stage I, so it does not count); the Stage III column is taken over the runs that cleared both earlier stages and actually met the verifiers. Agents do not perform best on one category throughout the pipeline : they achieve their highest pass rates on build toolchain rewrites at the first two stages but their lowest at the last; framework rewrites show the reverse pattern.
Called a real migration
Judge against human
Task
Class
runs
by human
by judge
agree ↑
too strict ↓
too lenient ↓
lang01 (cmark)
Language
26
7
6
25
1
0
lang03 (sqlparse)
Language
26
15
14
23
2
1
fw02 (json-server)
Framework
26
15
13
24
2
0
fw06 (uploadserver)
Framework
26
15
13
22
3
1
pf02 (Stylus)
Platform
26
14
12
24
2
0
Table 6: Task-by-Task Comparison ofMigration Audit and Independent Human Annotation. Each row is one task, attempted once by every one of the 26 model–effort configurations. Human and judge are how many of those runs each side called a genuine migration; agree , too strict and too lenient sort the same runs by whether the two verdicts matched, the judge zeroed a run the human accepted, or the reverse.
Figure 6: Break rate of each of the six verifiers : the share of its rounds in which it handed in an executable counterexample that passes on the original and fails on the submission, coloured by the model behind it and annotated with its reasoning-effort setting. The dashed line marks the lower of the two claude-opus-5 verifiers.
Benchmark
Unit of work
Starts failing
Stack changes
Migration criterion
Agentic verifier
SWE-bench [ 33 ]
one issue
✓
×
×
×
SWE-EVO [ 38 ]
one feature
✓
×
×
×
SWE-CI [ 10 ]
CI history
✓
×
×
×
Terminal-Bench [ 47 ]
one task
✓
×
×
×
TransCoder [ 53 ]
one function
✓
✓
×
×
MigrationBench [ 44 ]
whole repository
✓
✓
×
×
Table 7: Positioning against related benchmarks. Starts failing means whether the starting state already fails some test in the suite being scored; that is what makes “red to green” an informative verdict—and it is exactly the property SWE Refactor Bench cannot have. Migration criterion means whether a non-behavioural instrument can reject a submission that passed every behavioural check; agentic verifier means whether, at scoring time, further models actively search for differences beyond the fixed tests that the authors did not anticipate. ✓ is yes, ✓ is partial—MigrationBench moves a language version, Java 8 to 17/21, while the language itself stays—and × marks a dimension the benchmark does not aim at, not a defect.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Stage I — migration criteria
Stage II — test modules
Criterion
What it asks
Required
Verdict
Module
What it measures
Pass rate
no-c-sources
has C left the build
✓
pass
build
two link configurations
1.00
no-foreign-headers
is only cmark.h left
✓
pass
structure
install tree, ABI
1.00
no-c-in-build
is no C compiled
✓
pass
conformance
CommonMark corpus
1.00
rust-present
is Rust really there
✓
pass
renderers
five output renderers
1.00
rust-is-primary
is Rust the implementation
✓
fail
data-tables
entities, case folding
1.00
Appendix
Table 8: One task and one submission in full: claude-opus-5 at high effort on lang01-cmark-c-to-rust . All 4,184 checks across 15 modules pass, the highest mark Stage II can give; seven of the eight criteria pass, and the eighth, rust-is-primary , fails unanimously because the Rust reproduces the C implementation layout field by field— “the same hand-managed pointer stack, the same names and the same order … this is the original transliterated into unsafe Rust” , cited to src/inlines.rs:55 and neighbouring lines. Final score: 0 .
Migrated ↑
All tests pass ↑
Blindness ↓
Accepted ↑
Score ↑
Stage I passed
Stage II perfect
passed, not migrated
all three passed
out of 100
Task
Migration
n
n
n
n
n
mean
Language rewrites
lang01
C → Rust
26
6
5
5
0
0.00
lang02
C → Java
26
18
4
0
0
12.69
lang03
Python → Go
26
14
0
0
0
0.00
Appendix
Table 9: The 520 runs broken down by task. Every task is run once by each of the 26 configurations ( n=26 ). Column meanings follow the metric definitions of Section 3.1 : migrated is the number of runs passing Migration Audit, all tests pass is the number passing every fixed check in Behavioural Tests ( regardless of the Stage I verdict), blindness is the subset of those that failed Stage I, accepted is the number passing all three stages, and score is defined in ( 4 ). All tests pass − blindness is therefore the number of runs that reached Stage III. Only 7 of the 20 tasks ever produced an accepted submission .
Task
Upstream project
Source stack
Target stack
LoC
B (h)
Criteria
Modules
Checks
Language rewrites—the implementation language moves, the artifact does not
lang01
cmark 0.31.1
C11
Rust 1.90
22,619
30
8
15
4,184
lang02
zlib 1.3.1
C89
Java 17
22,732
20
10
18
4,162
lang03
sqlparse 0.5.3
Python
Go (stdlib)
4,393
15
8
15
8,530
lang04
acorn 8.14.0
JavaScript
Rust 1.90
9,571
30
7
13
16,039
lang05
go-yaml v3.0.1
Go
Zig 0.14.1
11,967
20
8
22
10,271
Appendix
Table 10: The task set, task by task. LoC counts newlines in the version-controlled implementation source of State A, excluding tests, vendored trees and documentation, measured the same way for every task rather than transcribed from each project’s own description; together these repositories hold 10,594 version-controlled files. B is the agent’s time budget in hours. Criteria is the number of prompt-form migration criteria, each of them required: a single failure zeroes the submission. Modules is the number of independent test modules and checks the number of fixed behavioural cases they hold, recorded from a reference build of State A. The size of a test suite is set by how much of the product is mechanically observable, not by the size of the repository. Every task has six verifiers of one hour each. Table 1 summarises these rows by class. Each row ships an upstream release verbatim, at the version shown and under its own licence: cmark [ 45 ] , zlib [ 21 ] , sqlparse [ 2 ] , Acorn [ 55 ] , go-yaml [ 9 ] , Jsonnet [ 25 ] , JSONata [ 30 ] , httpbin [ 52 ] , json-server [ 60 ] , the RealWorld Conduit app [ 58 ] , ChartMuseum [ 56 ] , miniserve [ 29 ] , go-simple-upload-server [ 1 ] , GraphHopper [ 27 ] , SQLite [ 59 ] , Stylus [ 3 ] , QuickJS [ 5 ] , libsodium [ 17 ] , Gson [ 26 ] and PyCryptodome [ 57 ] .
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.
Yuling Shi, Jinghan Xu, Kelin Fu +12
Shanghai Jiao Tong University · Peking University · Douyin Group +4
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.