Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?
Organizations: Klavis AI
Abstract
Developers rarely give a coding agent a single request: they ask it to build a component and then to extend, upgrade, and optimize it, each time on code the agent itself wrote. Most benchmarks, however, score one task at a time. We introduce Multi-step SWE, a long-horizon benchmark of 69 software engineering tasks, each a chain of 5-15 dependent steps on a single codebase. Every step is a complete, multi-hour engineering request that starts from the code the agent produced in the preceding steps, so early design decisions and errors carry forward. The tasks span four families of engineering work and nine languages, include both from-scratch builds and existing open-source codebases, and ground every step in an existing source of truth that defines both the instruction and its verifier. Across nine frontier models, the strongest model reaches a solve rate of only 18.5%, and 38 of the 69 tasks are not solved by any model. We release Multi-step SWE publicly at https://multistepswe.com.
Figures & tables
| Principle | Realization in Multi-step SWE |
|---|---|
| Multi-step | Each task chains dependent steps on one codebase, and each step starts from the code the agent produced in the preceding steps. |
| Long horizon | Each step is a complete engineering request with a budget of several hours; reference solutions reach tens of thousands of lines of code. |
| Well defined | Each step is grounded in an existing source of truth that is provided in the workspace or cited in the instruction, and its verifier uses the field’s own test suites and reference tools; every task passed two rounds of expert review. |
| Diverse | Four task families, nine languages, 34 tasks built from scratch, and 35 tasks on existing open-source codebases. |
| Family | Starting workspace | Each step provides | Each step must | Example task |
|---|---|---|---|---|
| Optimization (18) | a real library with the optimized code of the scored kernels removed | the scored functions and a guide to build, check, and time them | make the next kernel family faster, with unchanged output | BoringSSL, one cryptographic kernel family per step |
| Feature implementation (18) | empty, a skeleton, or an earlier release of the project | an interface contract and, usually, the release’s specification | implement the capabilities of a later upstream release | a Tor client in Rust across 12 releases |
| Version upgrade (20) | the first version’s specification, with no implementation | the next version’s specification and, usually, sample tests | implement the next version, changing behavior where it changes | a Lua interpreter in Java, from Lua 5.1 to 5.5 |
| Migration (13) | a legacy codebase at a pinned revision | an instruction naming the target stack and the required changes | move to the next stack release, keeping behavior unchanged | CityVoice, from Rails 4.0 to 8.1 |
| Task | Attack | Result before the fix | Countermeasure |
|---|---|---|---|
| SMB server | A background process rewrites the reward file with a reward of 1 | step passed | the score is passed through a channel the agent cannot write |
| Arquillian | A planted JUnitCore class reports every test as passing | step passed | the scorer’s own test runner comes first on the classpath |
| Go toolchain | A C program reads the expected outputs by absolute path | 62 of 133 cases pass | graded code runs unprivileged; outputs are readable only by root |
| Protocol Buffers | A C wrapper runs the official protoc , shipped as a compressed file and unpacked during the build | step passed | every file the agent can write is scanned for upstream code, including inside compressed files |
| CityVoice (Rails) | Deprecation warnings are turned off through configuration | deprecation check passes | warnings are captured inside the running process |
| BoringSSL | A cache returns stored outputs for repeated inputs | speedup on an AES function (reference solution: ) | inputs are perturbed on every timed call |
| SCALE | |
| Tasks | 69 : optimization (18) feature implementation (18) version upgrade (20) migration (13) |
| Steps | 574 in total; 5–15 per task (median 8, mean 8.3) |
| BUDGET | |
| Agent budget per step | 2 h (471 steps) 3 h (56 steps) 4 h (47 steps) |
| Agent budget per task | 10–60 h (median 16 h) |
| SETTING | |
| Tasks solved | |||||||
|---|---|---|---|---|---|---|---|
| Model | Overall | Optimization | Feature | Upgrade | Migration | Pass@4 | Steps passed |
| Claude Opus 5.5 | 18.5 [11.6, 25.7] | 34.7 | 12.5 | 0 8.8 | 19.2 | 31.9 | 56.8 |
| Claude Fable 5.1 | 17.8 [ 0 9.8, 26.1] | 45.8 | 0 8.3 | 0 1.3 | 17.3 | 24.6 | 49.1 |
| Gemini 4 Argon | 10.1 [ 0 5.8, 15.2] | 16.7 | 0 6.9 | 0 3.8 | 15.4 | 27.5 | 40.1 |
| GPT-6.1 Sol | 10.1 [ 0 5.1, 15.9] | 12.5 | 0 6.9 | 0 5.0 | 19.2 | 21.7 | 37.7 |
| GPT-6 Astra | 0 6.9 [ 0 2.9, 12.0] | 0 5.6 | 0 5.6 | 0 2.5 | 17.3 | 15.9 | 36.7 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Step and instruction | What the verifier checks |
|---|---|
| Lua interpreter in Java, Lua 5.3 5.4. Implement Lua 5.4 as its reference manual specifies, keeping the build and command-line interface and matching the reference interpreter’s output and errors. | Runs the official Lua 5.4 test suite and conformance programs recorded from the reference interpreter, including programs that a correct interpreter must reject. |
| turso database engine in Rust, a later release. Add ALTER TABLE column operations, checkpoint modes, and the other numbered requirements of the next upstream release. | Runs the engine’s shell and compares its output and database files with those of stock sqlite3 . |
| simdutf in C++, UTF-8 decoding. Make UTF-8 validation and transcoding faster without changing any output byte or reported error position. | Checks for identical output on hidden inputs against an unmodified reference implementation, then times each function on separate hidden inputs; the reward is 1 only if the output is exact and the speedup exceeds that of the reference solution, . |
| CityVoice Rails application, Rails 4.0 4.2. Move to Rails 4.2 and Ruby 2.3, retire deprecated APIs rather than silence them, and keep behavior unchanged apart from documented Rails changes. | Replays behavior recorded from the unmodified original (HTTP endpoints, webhooks, rake tasks, and mail) and checks that the migration is genuine, for example that the application actually boots on Rails 4.2. |
| Step | Fork | New in this fork | Cases | Full credit |
|---|---|---|---|---|
| 1 | Frontier | the full EVM: instruction set, gas schedule, tries, four precompiles | 1,166 | 15 |
| 2 | Homestead | DELEGATECALL ; contract-creation and signature rules | 2,606 | 16 |
| 3 | Tangerine Whistle | repricing of I/O-heavy operations; 63/64 call-gas rule | 1,390 | 15 |
| 4 | Spurious Dragon | replay protection, state clearing, code-size limit | 1,044 | 13 |
| 5 | Byzantium | REVERT , STATICCALL , return data, alt_bn128 and modexp precompiles | 2,493 | 14 |
| 6 | Constantinople | bitwise shifts, CREATE2 , EXTCODEHASH | 3,122 | 14 |
| Task | Progression across steps | Language | Steps | Start |
|---|---|---|---|---|
| Migration (13 tasks) | ||||
| Bento packaging | setup.py to PEP 517/621, uv lockfile, abi3 and free-threaded wheels, PEP 751 | Python | 6 | E |
| Arquillian | Maven and Java 6 to Gradle and Java 25 | Java | 13 | E |
| CityVoice | Rails 4.0 on Ruby 2.1 to Rails 8.1 on Ruby 3.4 | Ruby | 13 | E |
| hedis (Redis fork) | IPv4-first networking to IPv6-only | C | 6 | E |
| ISE server framework | C++98 to C++26 | C++ | 11 | E |
| Model | Harness | Reasoning effort |
|---|---|---|
| Claude Opus 5.5 | Claude Code | max |
| Claude Fable 5.1 | Claude Code | max |
| Gemini 4 Argon | mini-swe-agent | high |
| GPT-6.1 Sol | Codex CLI | max |
| GPT-6 Astra | Codex CLI | max |
| DeepSeek V4.1 Flash | Claude Code | max |
| Model | Correct | P50 | P75 | P90 | Geomean | Max |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 98.7 | 4.7 | 9.3 | 18.2 | 5.1 | 215 |
| Claude Fable 5.1 | 99.6 | 4.7 | 10.2 | 16.0 | 5.5 | 208 |
| Gemini 4 Argon | 92.5 | 3.5 | 7.9 | 14.9 | 3.7 | 205 |
| GPT-6.1 Sol | 93.1 | 3.4 | 6.0 | 12.6 | 3.7 | 95.6 |
| GPT-6 Astra | 91.4 | 3.6 | 7.0 | 12.2 | 4.1 | 55.5 |
| DeepSeek V4.1 Flash | 97.6 | 3.8 | 7.4 | 12.2 | 4.4 | 123 |
| Task | per step ( ) | Correct | P50 | P90 | Geomean | Max |
|---|---|---|---|---|---|---|
| Apache Arrow | 2.8, 2.3, 1.8, 1.7, 1.2 | 96 | 2.8 | 8.5 | 3.1 | 10.7 |
| AviSynth+ | 2.8, 8.6, 2.8, 8.5, 2.9, 4.1, 7.0 | 96 | 4.4 | 9.5 | 4.2 | 16.5 |
| BitMagic | 1.9, 7.8, 3.2, 3.1, 5.0, 1.6 | 99 | 3.7 | 11.7 | 4.1 | 21.1 |
| BoringSSL | 24.7, 11.3, 5.6, 1.9, 2.3 | 89 | 5.8 | 25.6 | 5.1 | 51.3 |
| FFmpeg libswscale | 9.0, 11.3, 4.9, 7.1, 6.9, 3.4 | 82 | 7.2 | 15.4 | 5.8 | 27.4 |
| ISA-L crypto | 9.8, 3.7, 3.0, 6.0, 10.2, 5.0, 3.0 | 96 | 5.0 | 11.5 | 4.8 | 15.6 |
AI Coding Agents
AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.
See all 347 papers