SWE-Next: Scalable Real-World Software Engineering Tasks for Agents

Jiarong Liang; Zhiheng Lyu; Zijie Liu; Xiangchao Chen; Ping Nie; Kai Zou; Wenhu Chen

cs.SEMar 21, 2026

SWE-Next: Scalable Real-World Software Engineering Tasks for Agents

Authors: Jiarong Liang, Zhiheng Lyu, Zijie Liu, Xiangchao Chen, Ping Nie, Kai Zou, Wenhu Chen

Organizations: University of Waterloo · University of North Carolina at Chapel Hill · Independent

Abstract

Executable software engineering data is valuable for training SWE agents, but scaling it remains difficult for two reasons: only a small fraction of real repository changes yield verifiable, high-signal task instances, and naively building repository-specific environments quickly becomes the dominant systems cost. We present SWE-Next, an execution-grounded framework for scalable SWE task and trajectory collection. On the data side, SWE-Next mines real merged pull requests, executes candidate base/merged commit pairs, and retains only those that produce strict test improvements without regressions, yielding self-verifying instances. It also applies strict submission gating so that collected trajectories remain evidence-driven rather than speculative. On the systems side, SWE-Next introduces reusable repo-quarter profiles, which reuse the same environment across nearby commits in time while keeping each task run separate and reproducible. Using only 30 hours and 639GB of environment storage, SWE-Next processes 3,971 seed repositories and 102,582 candidate commit pairs mined from real merged PRs to construct a dataset of 2,308 self-verifying instances. Experiments show that SWE-Next improves downstream pass@1 with fewer or comparable training trajectories, indicating that its gains come not from a stronger trajectory generator, but from higher-signal execution-grounded supervision and more efficient data collection.

Figures & tables

Figure 1: End-to-end overview of SWE-Next. Starting from real repositories and merged pull requests, we execute tests on base/merged commit pairs to retain verifiable instances, amortize environment setup via reusable quarter profiles, and package validated tasks for trajectory collection and post-training.

Dataset	# Instances	# Repos	Exec?	Source	Env. Size
R2E ( Jain et al., 2024 )	0.25k	137	✓	Synth	270 GB
R2E-gym (Subset) ( Jain et al., 2025 )	4.6k	10	✓	Synth	4 TB
SWE-bench-extra ( Badertdinov et al., 2024 )	6.38k	2k	$\times$	Real	–
SWE-bench-train ( Jimenez et al., 2024 )	19k	37	$\times$	Real	–
SWE-fixer ( Xie et al., 2025 )	115k	856	$\times$	Real	–
SWE-Gym ( Pan et al., 2025 )	2.4k	11	✓	Real	6 TB

Table 1: Dataset-scale comparison and environment size for open-source SWE training datasets. Most projects do not report their Env. Size. As a coarse lower-bound estimate, if a pipeline requires at least 100 distinct environments per repository, then scaling to thousands of repositories implies at least 10TB. SWE-Next substantially reduces environment storage via reusable profiles, making large-scale task collection practical without requiring TB infrastructure. This improvement also greatly accelerates data collection: we processed all 3.97k repositories and constructed the dataset in just 30 hours .

Figure 2: Quarter profile mechanism for reusable environments. We map each instance to a deterministic (repo, quarter) profile, build a shared quarter-env image that caches a virtual environment (without bundling repository source), and run per-commit repo checkouts in isolated workspaces while reusing the shared environment. When the quarter-env image build fails, we fall back to per-commit environment builds.

Pipeline	Backbone	System	# Train Trajs	Verified	Lite
Comparable trajectory-SFT baselines
Qwen2.5-Coder-Instruct	7B	OpenHands	–	1.8	1.0
R2E-Gym	7B	OpenHands	3.3k	14.4	11.3
SWE-smith	7B	SWE-agent	2k	15.2	11.7
SWE-Next (ours)	7B	OpenHands	3.7k	17.4	13.7
Qwen2.5-Coder-Instruct	14B	OpenHands	–	4.0	2.7

Table 2: Downstream agent pass@1 resolve rates on SWE-Bench-Verified and SWE-Bench-Lite, with the number of training trajectories used for SFT. We do not compare against systems that use verifiers or multiple attempts at test time.

SFT data	SWE-Bench Verified	SWE-Bench Lite
R2E-Gym	7.8%	5%
SWE-Next (ours)	8.2%	6.67%

Table 3: Performance gains are not driven by a stronger trajectory generator, but by SWE-Next’s execution-grounded task synthesis and supervision.

Failure Type	% of failures
Residual env/harness	2.4%
Search/localization	8.7%
Patch quality	75.7%
Submit gating / no valid diff	13.2%

Table 4: Failure-type decomposition over failed SWE-Next trajectories. With reusable repo-quarter profiles and runner fixups, residual env or harness failures are rare.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Candidates executed	102582
Final instances	2308 (2.2% yield)
Candidates with unchanged test behavior	74.5%
Candidates failing during setup	2.5%
Candidates failing during test execution	20.8%
Unique repos in final	311

Appendix

Table 5: Additional system statistics for the SWE-Next run described in this paper.

Field	Meaning
Identity and provenance
repo	Canonical repository name in owner/name format.
repo_name	Normalized repository identifier used by the pipeline for local naming and artifact organization.
repo_key	Filesystem-safe repository key used in image and artifact naming.
instance_id	Unique task identifier for the commit pair.
base_commit	Pre-fix commit SHA used as the buggy starting point.

Appendix

Table 6: Top-level fields in the released SWE-Next JSONL dataset.

Figure 3: Domain composition of the 3,971 seeded repositories, inferred from GitHub repository topics and descriptions. The corpus spans diverse software domains.

Repository	Description
Dev Tools
avocado-framework/avocado	Avocado is a set of tools and libraries to help with automated testing.
Blazemeter/taurus	Automation-friendly framework for Continuous Testing by.
etianen/logot	Test whether your code is logging correctly .
pylint-dev/astroid	A common base representation of python source code for pylint and other projects.
pytest-dev/pyfakefs	Provides a fake file system that mocks the Python file system modules.

Appendix

Table 7: Repositories that contribute NewCommitBetter instances in our dataset, grouped by coarse domain. Descriptions are derived from GitHub repository metadata.

Automated Software Engineering

Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.

See all 80 papers

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Feb 27, 2026Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov +1Software Engineering Benchmarks Software Engineering Agents
Nebius, Amsterdam, Netherlands.
daVinci-Env: Open SWE Environment Synthesis at Scale
Mar 13, 2026Dayuan Fu, Shenyu Wu, Yunze Wu +11Software Engineering LLM Agent Training
SII · GAIR · SJTU
SWE-Stream: Scaling Executable Training Data for SWE Agents via Progressive Construction and Elastic Streaming
Oct 14, 2026Chengyang Liu, Wen Yang, Shifeng Chen +5Software Engineering Agents LLM Agent Training
Xiamen University · Unisound · Institute of Automation, Chinese Academy of Sciences

Feb 27, 2026J/K move · Enter open · S save

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev

Nebius, Amsterdam, Netherlands.

Software Engineering Benchmarks Software Engineering Agents LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.

Abstract

Automated Software Engineering

Explore similar work

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

daVinci-Env: Open SWE Environment Synthesis at Scale

SWE-Stream: Scaling Executable Training Data for SWE Agents via Progressive Construction and Elastic Streaming