Organizations: The University of Tokyo · Shanghai Jiao Tong University · Shanghai Innovation Institute · Huazhong University of Science and Technology · Nankai University · Singapore University of Technology and Design · Queen’s University · Fudan University · Zhejiang University
The Agentic Web is emerging as a paradigm in which autonomous software agents interact with online resources and with each other to accomplish user goals. However, the capacity of Agentic Web is still limited by insufficient autonomous software agent population, which has become a crucial challenge for scaling Agentic Web. In order to alleviate this, we study the task of automatically converting existing code repositories into autonomous software agents via coding agents, decompose the process into critical stages, and identify key technical hurdles. To systematically evaluate this capability, we propose SoftWare Agent generation for Agentic Web Bench (SW-A2-Bench), the first benchmark designed for software agent generation. SW-A2-Bench evaluates not only whether software agents can be generated, but also whether generated software agents are faithful to the source repositories and interoperable with other agents in multi-agent workflows. Our experiments demonstrate that our approach effectively activates the functional capabilities of code repositories and enables interoperable multi-agent collaboration in Agentic Web. We believe that this work will provide a standardized evaluation for software agent generation and will contribute to the future of scaling the capacity of Agentic Web.
Figures & tables
Figure 1: Conceptual illustration of software agent generation for Agentic Web. Through agent generation, code repositories are transformed into software agents that can be integrated into the Agentic Web, enabling interactions and collaborations for real-world tasks.
Figure 2: Execution task diversity analysis. Single-repo tasks are distributed across 9 application domains, illustrating broad coverage of domain-specific execution scenarios. Multi-repo tasks are grouped by the number of distinct domains involved in each workflow (cross- k domains), predominantly highlighting the prevalence of cross-domain interactions and increasing interoperability requirements.
Figure 3: Execution task complexity distribution. For single-repo tasks, difficulty is decomposed into several dimensions, measured by corresponding indicators. For multi-repo tasks, difficulty is determined by orchestration complexity, measured by the length of the sequential multi-agent chain (i.e., the number of sequentially invoked repositories). Tasks are grouped into easy, medium, and hard tiers based on these respective criteria.
Figure 4: The orchestration mechanism instantiated in our experiments for SW- A2 -Bench Stage 3. Multi-repo tasks are provided together with their oracle subtask decompositions obtained during benchmark construction, which serve as the input to the orchestration process. In parallel, Agent Cards produced by software agent generation agent are converted into agent skills. The coordinator relies solely on these skills to select and bind appropriate software agents for each subtask, orchestrates their execution and obtains the result.
Framework
Pass@1 ↑
Pass@3 ↑
Tries ↓
Tokens ↓
Claude Code
100.00
100.00
1.000
3374449
Codex
94.28
100.00
1.086
2321397
OpenHands
94.28
100.00
1.057
3254978
EnvX
100.00
100.00
1.000
4215051
Table 1: Stage 1 Results: Agent Generation Process Assessment. We report Pass@1 and Pass@3 in percentage (%).
Framework
Execution SR (%) ↑
Easy
Medium
Hard
Overall
Claude Code
57.9
38.5
10.4
36.9
Codex
45.2
42.3
14.2
34.5
OpenHands
54.8
32.7
10.4
33.9
EnvX
53.2
38.5
10.4
35.1
Table 2: Stage 2 Results: Capability Inheritance. We report Execution Success Rate (SR, %) for task execution.
Framework
Avg-Precision ↑
Avg-Recall ↑
Claude Code
70.4
70.4
Codex
50.0
47.8
OpenHands
65.2
69.7
EnvX
69.8
74.0
Table 3: Skill Specification Quality: Avg-Precision and Avg-Recall (in %).
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Pujun Zheng, Zixin Shang, Shufan Jiang +5
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.