SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Organizations: Nebius, Amsterdam, Netherlands.
Abstract
Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.
Figures & tables
| Stage | PRs | Repos |
|---|---|---|
| PRs | 29,511,758 | 145,306 |
| PRs with tests | 8,593,722 | 101,958 |
| PR linked with issue and test | 805,598 | 50,797 |
| Repo based filtering | 583,809 | 21,692 |
| Successful tasks w/ F2P | 41,349 | 4,006 |
| Issue text based filtering | 33,049 | 3,701 |
| Model | pass@1 | pass@3 | pass@5 | pass@10 |
|---|---|---|---|---|
| Non-interactive setup | 12.1 | 14.2 | 14.9 | 15.7 |
| mini-SWE-agent (Qwen3-30B, 32k) | 17.4 | 30.5 | 36.9 | 46.1 |
| mini-SWE-agent (DeepSeek-V3.2, 32k) | 20.3 | 37.3 | 46.6 | 59.8 |
| mini-SWE-agent (Qwen3-480B, 32k) | 25.8 | 42.4 | 50.0 | 58.8 |
| mini-SWE-agent (Qwen3-480B, 64k) | 26.8 | 43.1 | 49.2 | 55.9 |
| mini-SWE-agent (Qwen3-480B, 128k) | 27.1 | 44.4 | 52.2 | 62.7 |
| Prompt | Acc | Prec | Rec | F1 |
|---|---|---|---|---|
| Rebench V1 | 0.67 | 0.76 | 0.22 | 0.34 |
| SPICE | 0.66 | 0.59 | 0.34 | 0.43 |
| Verified | 0.68 | 0.75 | 0.24 | 0.36 |
| Verified+ | 0.69 | 0.66 | 0.40 | 0.50 |
| Verified-E | 0.65 | 0.83 | 0.10 | 0.17 |
| Model | Acc | Prec | Rec | F1 |
|---|---|---|---|---|
| gpt-oss-120b | 0.68 | 0.75 | 0.24 | 0.36 |
| gpt-oss-120b (high) | 0.66 | 0.81 | 0.16 | 0.26 |
| Deepseek v3.2 | 0.66 | 0.76 | 0.18 | 0.29 |
| GLM 4.7 | 0.64 | 0.81 | 0.10 | 0.18 |
| MiniMax M2.1 | 0.59 | 0.59 | 0.16 | 0.25 |
| GPT 5.2 | 0.67 | 0.80 | 0.18 | 0.30 |
| Setup | Acc | Prec | Rec | F1 |
|---|---|---|---|---|
| Single Model | 0.67 | 0.76 | 0.22 | 0.34 |
| Mixed (Avg) | 0.69 | 0.73 | 0.31 | 0.43 |
| Mixed (Consensus) | 0.64 | 0.88 | 0.06 | 0.11 |
| Model | Py | JS | Go | Rust | Scala | pass@1 | pass@3 |
|---|---|---|---|---|---|---|---|
| Claude Opus-4.5 | 36.1 | 26.7 | 15.0 | 28.9 | 19.4 | 25.2 | 32.7 |
| GLM 4.7 | 27.2 | 26.1 | 17.2 | 24.4 | 11.7 | 21.3 | 26.7 |
| MiniMax M2.1 | 26.1 | 26.1 | 15.6 | 20.6 | 7.8 | 19.2 | 27.0 |
| Gemini 3 Flash | 25.6 | 25.0 | 11.7 | 21.1 | 7.2 | 18.1 | 27.7 |
| DeepSeek v3.2 | 23.3 | 24.4 | 12.2 | 21.7 | 5.6 | 17.4 | 25.0 |
| GPT 5.2 | 20.6 | 24.4 | 11.1 | 21.7 | 7.2 | 17.0 | 25.0 |
| Model | A@1 | A@3 | B*@1 | B*@3 |
|---|---|---|---|---|
| DeepSeek V3.2 | 22.0 | 26.0 | 4.0 | 4.0 |
| Gemini | 26.0 | 34.0 | 0.0 | 4.0 |
| GLM 4.7 | 28.0 | 34.0 | 4.0 | 6.0 |
| GPT 5.2 | 14.0 | 26.0 | 4.0 | 6.0 |
| Opus 4.5 | 22.0 | 28.0 | 6.0 | 8.0 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Lang. | Tasks | Repos | Kept R. | Kept T. | Task Sh. | Repo Sh. |
|---|---|---|---|---|---|---|
| python | 184,558 | 10,548 | 2,144 | 152,161 | 0.82 | 0.20 |
| go | 123,424 | 6,168 | 1,345 | 104,115 | 0.84 | 0.22 |
| typescript | 114,164 | 7,356 | 1,300 | 92,292 | 0.81 | 0.18 |
| javascript | 80,262 | 7,650 | 915 | 58,349 | 0.73 | 0.12 |
| java | 75,114 | 3,574 | 795 | 64,805 | 0.86 | 0.22 |
| rust | 49,703 | 3,325 | 530 | 39,397 | 0.79 | 0.16 |
| Language | python | js | js/ts | ruby | go | c | rust | java | php | cpp |
|---|---|---|---|---|---|---|---|---|---|---|
| Tasks/Repos | 55 | 11 | 6 | 6 | 5 | 5 | 5 | 4 | 4 | 2 |
Software Engineering Benchmarks
Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.
See all 155 papers