AgentSWE: Can Coding Agents Build the Agent You Actually Want?

Jiahao Wang; Hongjin Qian; Yuyang Hu; Jiajun Zhang; Zheng Liu

cs.SEOct 9, 2026

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

Authors: Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu

Organizations: Beijing Academy of Artificial Intelligence · Institute of Automation, Chinese Academy of Sciences · Renmin University of China

Abstract

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.

Figures & tables

Figure 1: AgentSWE at a glance. (a) Existing benchmarks grade the patch a coding agent writes, or the performance of an agent someone else built; neither asks whether a coding agent delivers the agent a requirement describes. (b) AgentSWE commissions an agent from a natural-language requirement and grades what the delivered agent does on held-out cases, over creation, editing and optimization. (c) Mean held-out Result (

S

for Optimization) of five builders in the Codex harness ( Table 1 ).

Figure 2: AgentSWE task families and benchmark construction. (a) Task objectives and construction processes. (b) Builder-visible inputs, evaluator-owned scoring contracts and held-out cases, supported by Harbor environments.

Figure 3: AgentSWE evaluation workflow: development with public feedback, delivery selection and freezing, and held-out evaluation using task-specific Result rubrics or native benchmark objectives.

Task	GPT-5.6 Sol	GPT-5.5	DS-V4-Pro	DS-V4-Flash	Qwen3.6
Creation
Authorized vulnerability validation	0.0	0.0	14.0	14.0	0.0
Database analytics	3.0	9.8	14.3	0.0	0.0
Desktop GUI automation	18.0	3.2	17.5	12.0	0.0
Document-to-editable PPTX	0.0	9.8	18.7	16.8	21.7
Evidence-grounded document QA	3.3	12.2	0.0	3.5	0.0

Table 1: Held-out performance across the three lifecycle stages. Creation and Editing report mean Result (0–100); Optimization reports gap closure

S=100\gamma

. Section G.3 gives per-run Optimization results. DS denotes DeepSeek.

Task	GPT-5.6 Sol	DS-V4.1-Flash	Qwen3.6
Creation
Scientific PDF translation	18.9	31.9	0.0
Database analytics	7.4	38.7	0.0
Repository bug repair	42.9	30.0	0.0
Mean	23.1	33.5	0.0
Editing

Table 5: AgentSWE-Lite: mean over three builds per pairing in Codex, held-out Result for Creation and Editing and

S

for Optimization; Figure 8 shows each build. DS-V4.1-Flash (DeepSeek-V4.1-Flash) is the runtime model and the judge in all three families ( Section E.2 ), so Creation and Optimization levels are not comparable to Table 1 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Task	Source system	Dev / held-out
Creation	Authorized vulnerability validation	Strix ( usestrix, 2026 )	2 / 6
	Database analytics	Vanna ( vanna-ai, 2026 )	2 / 6
	Desktop GUI automation	UI-TARS-desktop ( Qin et al., 2025 )	2 / 6
	Document-to-editable PPTX	PPT Master ( He, 2026 )	2 / 6
	Evidence-grounded document QA	PaperQA ( Lála et al., 2023 ; Skarlinski et al., 2024 )	2 / 6
	Formal theorem proving	LeanCopilot ( Song et al., 2025 )	2 / 6

Appendix

Table 6: AgentSWE tasks, their source systems, and their development and held-out cases.

Requirement	Public development example	Held-out example
PDF content and geometry	Mixed native and scanned pages, bilingual Chinese output, a landscape page (dev 002).	English-to-Japanese translation of a two-column paper with a paragraph split across pages 2 and 3 (test 002); Traditional Chinese with dense figure labels and preferred glossary terms (test 006).
OpenHands recovery	Lost responses, convergence applied exactly once, discipline for runtime events (dev 001); fencing of the replaced owner and scoped access (dev 002).	Pause, resume, and cancel followed by stale callbacks and a terminal burst (test 003); durable recovery across three workspace crash boundaries (test 002).
PinchBench transfer	Stock trend analysis (dev 006) and Apache top error analysis (dev 007).	Stock volatility analysis (test 004), Apache critical errors and timelines (tests 013 and 014), under the same fixed runtime and native grader.

Appendix

Table 7: Requirement coverage in development and held-out cases, with examples. The held-out split changes inputs and interactions within the commission.

Dimension	Max.	Partial cap	Safety cap	Effective cap	Score
Task completion	35	4	3	3	0
Recovery quality	25	3	2	2	0
Durable consistency	20	2	2	2	1
Honest reporting	10	2	3	2	2
Privacy and scope	10	1	2	1	1

Appendix

Table 8: Worked OpenHands Result score. Ceilings for unmet outcomes and ceilings for safety failures apply together; their minimum per dimension sums to 10, and the recorded score is 4.

	Creation	Editing	Optimization
Tasks	10	10	5
Dev / held-out	2 / 6 per task	2 / 6 per task	10–50 / 25–100
Builder effort	GPT models: xhigh ; DeepSeek and Qwen: max	GPT models: xhigh ; DeepSeek and Qwen: max	GPT-5.5: xhigh ; GPT-5.6 Sol: xhigh on BrowseComp, high otherwise; DeepSeek and Qwen: high
Development budget	Up to 10 submissions, 8 h	5 accepted submissions, 5 h	5 accepted rounds, 16 h
Case budget	600 s	600 s	Native call, token, and time caps
Runtime model	gpt-5.6-sol / medium	deepseek-v4.1-flash / high	gpt-5.6-sol, fixed by contract

Appendix

Table 9: Main evaluation settings.

Figure 7: Independent Creation runs behind Tables 4 and 4 , three per condition: held-out Result of each run (dots) and the condition mean (line or bar) for (a) the number of development cases and (b) the prompt priors.

Figure 8: Independent builds of the eight-task study ( Table 5 ), three per pairing: (a) held-out Result on the Creation and Editing tasks and (b)

S

on the Optimization tasks, for each build (dots) and the mean of each pairing (bars). PDF, DB, and Repo denote scientific PDF translation, database analytics, and repository bug repair.

Figure 9: Static assessment of Creation deliveries (left): score of the frozen source on the shared rubric of eight dimensions (0–100), graded once per delivery without execution. Held-out Result from Table 1 (right), on the same color scale.

Task	Held-out	Last dev
Authorized vulnerability validation	0.0	40.0
Database analytics	70.0	94.0
Desktop GUI automation	28.5	63.0
Document-to-editable PPTX	10.2	46.5
Evidence-grounded document QA	81.2	98.8
Formal theorem proving	16.7	100.0

Appendix

Table 10: Creation under the extended-budget setting, with DeepSeek-V4.1-Flash as builder (in Codex), runtime model, and judge. Held-out is the held-out Result; Last dev is the last development score observed in the run.

ID	Builder	Benchmark	$J_{0}$	$J_{1}$	$\Delta$	$S$	W/L/T	Round
M01	GPT-5.6 Sol	BrowseC.	4.000	35.000	$+31.00$	$32.3$	32/1/67	3
M02	GPT-5.6 Sol	$\tau^{3}$	42.500	45.000	$+2.50$	$4.3$	3/2/35	5
M03	GPT-5.6 Sol	TerminalB.	25.000	17.500	$-7.50$	$-10.0$	4/7/29	5
M04	GPT-5.6 Sol	PinchB.	27.625	16.825	$-10.80$	$-14.9$	4/11/25	1
M05	GPT-5.6 Sol	OSWorld	0.000	12.000	$+12.00$	$12.0$	3/0/22	1
M06	DS-V4-Flash	BrowseC.	18.000	46.000	$+28.00$	$34.1$	30/2/68	2

Appendix

Table 11: Optimization held-out results.

J_{0}

and

J_{1}

are starter and candidate scores. Round is the selected round.

Figure 10: Optimization development and held-out gains per run, in native objective points. Development gain (open) is the selected candidate’s development score minus the starter’s; held-out gain (filled) is

\Delta

from Table 11 .

Figure 11: Execution time of each held-out Creation execution by builder, colored by its immediate outcome, with the categories of Figure 6 : graded 113 (41.9 %), error exit 139 (51.5 %), 600 s timeout 15 (5.6 %), and other gates 3 (1.1 %). Categories are mutually exclusive; gate failures total 157 (58.1 %).

Observed mode	DS-V4-Pro (10)	GPT (19)	DS-V4-Flash (4)	Total (33)
Fitting the visible cases	4	7	2	13
Gate failure or rejection of legal input	3	9	2	14
Capability absent, uninvoked, or incomplete execution	6	18	4	28
Runtime completion claim contradicted by records	6	5	2	13
Mutation that should have been refused	3	7	4	14
Budget spent without convergence	3	9	1	13

Appendix

Table 12: Editing modes in the runs that deliver a candidate. Rows can overlap; the third row is the union of two submodes.

	Creation	Optimization
All runs	50	25
Runs with a final report	42	19
Reports asserting completion	23	17
Reports contradicted by the record	6	3
Reports acknowledging partial completion or failure	19	2
Contradictions among the latter reports	0	0

Appendix

Table 13: Audit of builders’ final reports.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

Jun 12, 2026

Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.

Brendan King, Jeffrey Flanigan

University of California, Santa Cruz

Coding Agents LLM Agent Evaluation AI Agent Benchmarks User Simulation Coding Agent Benchmarks

Ask PDF

Jun 29, 2026

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.

Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal +1

1Scale AI

Coding Agents Software Engineering Benchmarks AI Agent Evaluation User Simulation

Ask PDF

May 8, 2026

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution

We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.

Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12

1Scale AI

Coding Agents Software Engineering Benchmarks Automated Test Generation

Ask PDF

Abstract

Coding Agents

Explore similar work

Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution