cs.SENov 6, 2026

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

Authors: Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev

Organizations: Nebius

Abstract

Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver-judge-repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

    Feb 27, 2026Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov +1Software Engineering BenchmarksSoftware Engineering Agents