Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Organizations: Nebius
Abstract
Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver-judge-repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.
Figures & tables
| Model | Context limit | Pass@1 SEM | Pass@3 | All-3 | Mean total tokens | Mean steps |
| Opus 5.5 (high) | 1M | 81.52 1.16 | 89.11 | 73.76 | 4.48M | 53.2 |
| GPT 6.1 Sol (high) | 1M | 78.38 0.92 | 82.18 | 73.27 | 1.75M | 27.4 |
| GPT 5.6 Sol (high) | 1M | 75.74 1.25 | 87.13 | 63.37 | 2.84M | 34.3 |
| GPT 5.6 Luna (xhigh) | 1M | 71.45 0.44 | 84.16 | 56.93 | 13.01M | 76.5 |
| Kimi K3 (max) | 262K | 63.70 2.43 | 76.24 | 48.51 | 16.84M | 151.4 |
| DeepSeek V4 Flash (high) | 1M | 56.11 1.47 | 71.78 | 39.11 | 26.50M | 199.5 |
| Language | Opus 5.5 (high) | GPT 6.1 Sol (high) | GPT 5.6 Sol (high) | GPT 5.6 Luna (xhigh) | Kimi K3 (max) | DeepSeek V4 Flash (high) | GLM-5.3-Flash (max) | Qwen 3.8 27B (xhigh) | Qwen 3.6 27B (thinking) | |
| Clojure | 13 | 82.1 | 74.4 | 51.3 | 46.2 | 64.1 | 35.9 | 30.8 | 38.5 | 10.3 |
| Dart | 14 | 71.4 | 61.9 | 81.0 | 81.0 | 66.7 | 61.9 | 40.5 | 38.1 | 11.9 |
| Elixir | 23 | 75.4 | 81.2 | 72.5 | 65.2 | 53.6 | 34.8 | 31.9 | 30.4 | 8.7 |
| Erlang | 12 | 100.0 | 100.0 | 86.1 | 83.3 | 83.3 | 75.0 | 50.0 | 58.3 | 25.0 |
| F# | 16 | 66.7 | 62.5 | 81.3 | 68.8 | 47.9 | 41.7 | 43.8 | 29.2 | 10.4 |
| Haskell | 14 | 92.9 | 95.2 | 83.3 | 83.3 | 76.2 | 78.6 | 66.7 | 52.4 | 26.2 |
| Pipeline configuration | Candidates | Strict EV passes | Pass rate (%) |
| SWE-rebench V2 install agent, A only | 1,096 | 377 | 34.4 |
| Combined auditor, A only | 1,096 | 539 | 49.2 |
| Combined auditor, A / B2 / B5 | 2,640 | 1,247 | 47.2 |
| Stage | SWE-rebench V2 path | Longtail-SWE |
| Before semantic annotation | 3,261 (100.0%) | 3,261 (100.0%) |
| After semantic screening | 1,096 (33.6%) | 2,640 (81.0%) |
| Strict execution validation | 377 (11.6%) | 1,247 (38.2%) |
| Image build and revalidation | 364 (11.2%) | 1,201 (36.8%) |
| After repair quality review | 197 (6.0%) | 533 (16.3%) |
| Difficulty-selected pool | 80 (2.5%) | 237 (7.3%) |
| Evaluation tests | Pass@1 SEM | Pass@3 | All-3 |
| Pre-repair | 36.5 | 20.6 | |
| Post-repair | 77.8 | 54.0 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | SWE-rebench V2 | Longtail-SWE |
| Clojure | 21 | 55 |
| Dart | 20 | 122 |
| Elixir | 59 | 109 |
| Erlang | 36 | 57 |
| F# | 62 | 74 |
| Haskell | 49 | 126 |
| Dataset | Tasks | Repos. | Langs. | Words | Files | Reference additions | Test additions | F2P | P2P |
| Longtail-SWE | 202 | 129 | 13 | 513.25 | 13.14 | 414.82 | 645.87 | 23.97 | 42.91 |
| SWE-Bench ProMax | 170 | 70 | 7 | 481.45 | 12.38 | 342.05 | 236.52 | — | — |
| SWE-Bench Pro (Public) | 731 | 11 | 4 | 178.88 | 5.07 | 120.34 | 105.73 | 14.43 | 57.46 |
| SWE-bench Multilingual | 300 | 41 | 9 | 234.83 | 1.74 | 28.87 | 39.35 | 1.44 | 34.02 |
| Task type | Tasks | Share (%) |
| Core features | 136 | 67.3 |
| Major bugs | 55 | 27.2 |
| Integration features | 25 | 12.4 |
| Edge-case bugs | 20 | 9.9 |
| Minor bugs | 19 | 9.4 |
| UI/UX features | 11 | 5.4 |
| Language | Tasks | Repos | Easy | Medium | Hard |
| Clojure | 13 | 10 | 0 | 3 | 10 |
| Dart | 14 | 11 | 1 | 3 | 10 |
| Elixir | 23 | 13 | 0 | 7 | 16 |
| Erlang | 12 | 7 | 2 | 4 | 6 |
| F# | 16 | 8 | 0 | 13 | 3 |
| Haskell | 14 | 8 | 1 | 5 | 8 |
Software Engineering Benchmarks
Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.
See all 170 papers