As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
Figures & tables
Figure 1: AgentSWE at a glance. (a) Existing benchmarks grade the patch a coding agent writes, or the performance of an agent someone else built; neither asks whether a coding agent delivers the agent a requirement describes. (b) AgentSWE commissions an agent from a natural-language requirement and grades what the delivered agent does on held-out cases, over creation, editing and optimization. (c) Mean held-out Result ( S for Optimization) of five builders in the Codex harness ( Table 1 ).
Figure 2: AgentSWE task families and benchmark construction. (a) Task objectives and construction processes. (b) Builder-visible inputs, evaluator-owned scoring contracts and held-out cases, supported by Harbor environments.
Figure 3: AgentSWE evaluation workflow: development with public feedback, delivery selection and freezing, and held-out evaluation using task-specific Result rubrics or native benchmark objectives.
Task
GPT-5.6 Sol
GPT-5.5
DS-V4-Pro
DS-V4-Flash
Qwen3.6
Creation
Authorized vulnerability validation
0.0
0.0
14.0
14.0
0.0
Database analytics
3.0
9.8
14.3
0.0
0.0
Desktop GUI automation
18.0
3.2
17.5
12.0
0.0
Document-to-editable PPTX
0.0
9.8
18.7
16.8
21.7
Evidence-grounded document QA
3.3
12.2
0.0
3.5
0.0
Table 1: Held-out performance across the three lifecycle stages. Creation and Editing report mean Result (0–100); Optimization reports gap closure S=100γ . Section G.3 gives per-run Optimization results. DS denotes DeepSeek.
Figure 5
Task
GPT-5.6 Sol
DS-V4.1-Flash
Qwen3.6
Creation
Scientific PDF translation
18.9
31.9
0.0
Database analytics
7.4
38.7
0.0
Repository bug repair
42.9
30.0
0.0
Mean
23.1
33.5
0.0
Editing
Table 5: AgentSWE-Lite: mean over three builds per pairing in Codex, held-out Result for Creation and Editing and S for Optimization; Figure 8 shows each build. DS-V4.1-Flash (DeepSeek-V4.1-Flash) is the runtime model and the judge in all three families ( Section E.2 ), so Creation and Optimization levels are not comparable to Table 1 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Task
Source system
Dev / held-out
Creation
Authorized vulnerability validation
Strix ( usestrix, 2026 )
2 / 6
Database analytics
Vanna ( vanna-ai, 2026 )
2 / 6
Desktop GUI automation
UI-TARS-desktop ( Qin et al., 2025 )
2 / 6
Document-to-editable PPTX
PPT Master ( He, 2026 )
2 / 6
Evidence-grounded document QA
PaperQA ( Lála et al., 2023 ; Skarlinski et al., 2024 )
2 / 6
Formal theorem proving
LeanCopilot ( Song et al., 2025 )
2 / 6
Appendix
Table 6: AgentSWE tasks, their source systems, and their development and held-out cases.
Requirement
Public development example
Held-out example
PDF content and geometry
Mixed native and scanned pages, bilingual Chinese output, a landscape page (dev 002).
English-to-Japanese translation of a two-column paper with a paragraph split across pages 2 and 3 (test 002); Traditional Chinese with dense figure labels and preferred glossary terms (test 006).
OpenHands recovery
Lost responses, convergence applied exactly once, discipline for runtime events (dev 001); fencing of the replaced owner and scoped access (dev 002).
Pause, resume, and cancel followed by stale callbacks and a terminal burst (test 003); durable recovery across three workspace crash boundaries (test 002).
PinchBench transfer
Stock trend analysis (dev 006) and Apache top error analysis (dev 007).
Stock volatility analysis (test 004), Apache critical errors and timelines (tests 013 and 014), under the same fixed runtime and native grader.
Appendix
Table 7: Requirement coverage in development and held-out cases, with examples. The held-out split changes inputs and interactions within the commission.
Dimension
Max.
Partial cap
Safety cap
Effective cap
Score
Task completion
35
4
3
3
0
Recovery quality
25
3
2
2
0
Durable consistency
20
2
2
2
1
Honest reporting
10
2
3
2
2
Privacy and scope
10
1
2
1
1
Appendix
Table 8: Worked OpenHands Result score. Ceilings for unmet outcomes and ceilings for safety failures apply together; their minimum per dimension sums to 10, and the recorded score is 4.
Creation
Editing
Optimization
Tasks
10
10
5
Dev / held-out
2 / 6 per task
2 / 6 per task
10–50 / 25–100
Builder effort
GPT models: xhigh ; DeepSeek and Qwen: max
GPT models: xhigh ; DeepSeek and Qwen: max
GPT-5.5: xhigh ; GPT-5.6 Sol: xhigh on BrowseComp, high otherwise; DeepSeek and Qwen: high
Development budget
Up to 10 submissions, 8 h
5 accepted submissions, 5 h
5 accepted rounds, 16 h
Case budget
600 s
600 s
Native call, token, and time caps
Runtime model
gpt-5.6-sol / medium
deepseek-v4.1-flash / high
gpt-5.6-sol, fixed by contract
Appendix
Table 9: Main evaluation settings.
Figure 7: Independent Creation runs behind Tables 4 and 4 , three per condition: held-out Result of each run (dots) and the condition mean (line or bar) for (a) the number of development cases and (b) the prompt priors.
Figure 8: Independent builds of the eight-task study ( Table 5 ), three per pairing: (a) held-out Result on the Creation and Editing tasks and (b) S on the Optimization tasks, for each build (dots) and the mean of each pairing (bars). PDF, DB, and Repo denote scientific PDF translation, database analytics, and repository bug repair.
Figure 9: Static assessment of Creation deliveries (left): score of the frozen source on the shared rubric of eight dimensions (0–100), graded once per delivery without execution. Held-out Result from Table 1 (right), on the same color scale.
Task
Held-out
Last dev
Authorized vulnerability validation
0.0
40.0
Database analytics
70.0
94.0
Desktop GUI automation
28.5
63.0
Document-to-editable PPTX
10.2
46.5
Evidence-grounded document QA
81.2
98.8
Formal theorem proving
16.7
100.0
Appendix
Table 10: Creation under the extended-budget setting, with DeepSeek-V4.1-Flash as builder (in Codex), runtime model, and judge. Held-out is the held-out Result; Last dev is the last development score observed in the run.
ID
Builder
Benchmark
J0
J1
Δ
S
W/L/T
Round
M01
GPT-5.6 Sol
BrowseC.
4.000
35.000
+31.00
32.3
32/1/67
3
M02
GPT-5.6 Sol
τ3
42.500
45.000
+2.50
4.3
3/2/35
5
M03
GPT-5.6 Sol
TerminalB.
25.000
17.500
−7.50
−10.0
4/7/29
5
M04
GPT-5.6 Sol
PinchB.
27.625
16.825
−10.80
−14.9
4/11/25
1
M05
GPT-5.6 Sol
OSWorld
0.000
12.000
+12.00
12.0
3/0/22
1
M06
DS-V4-Flash
BrowseC.
18.000
46.000
+28.00
34.1
30/2/68
2
Appendix
Table 11: Optimization held-out results. J0 and J1 are starter and candidate scores. Round is the selected round.
Figure 10: Optimization development and held-out gains per run, in native objective points. Development gain (open) is the selected candidate’s development score minus the starter’s; held-out gain (filled) is Δ from Table 11 .
Figure 11: Execution time of each held-out Creation execution by builder, colored by its immediate outcome, with the categories of Figure 6 : graded 113 (41.9 %), error exit 139 (51.5 %), 600 s timeout 15 (5.6 %), and other gates 3 (1.1 %). Categories are mutually exclusive; gate failures total 157 (58.1 %).
Observed mode
DS-V4-Pro (10)
GPT (19)
DS-V4-Flash (4)
Total (33)
Fitting the visible cases
4
7
2
13
Gate failure or rejection of legal input
3
9
2
14
Capability absent, uninvoked, or incomplete execution
6
18
4
28
Runtime completion claim contradicted by records
6
5
2
13
Mutation that should have been refused
3
7
4
14
Budget spent without convergence
3
9
1
13
Appendix
Table 12: Editing modes in the runs that deliver a candidate. Rows can overlap; the third row is the union of two submodes.
Creation
Optimization
All runs
50
25
Runs with a final report
42
19
Reports asserting completion
23
17
Reports contradicted by the record
6
3
Reports acknowledging partial completion or failure