CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

Prince Zizhuang Wang; Chenhao Liang; Zelong Xu; Aojie Yuan; Xiaolin Zhou; Haiyue Zhang; Yue Zhao; Xiyang Hu

cs.AISep 26, 2026

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

Authors: Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu,

Organizations: Carnegie Mellon University · University of Southern California · University of Wisconsin–Madison · Arizona State University ·

Abstract

Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored. Diagnosing a runtime interaction failure requires agents to connect visual observations with the responsible code, then use the application again to verify the repair. We introduce CUA-SWE, a benchmark, environment, and evaluation pipeline for software engineering with computer use. Beyond studying how GUI feedback supports diagnosis and repair, we ask whether agents can complete software engineering tasks when required specification or operational information is available only through the running application's visual interface. CUA-SWE spans four software engineering domains and requires agents to modify code and configuration, execute commands, interact with running software, and inspect visual feedback within the same task. Each task includes deterministic, task-specific tests that verify whether the resulting software satisfies the requirements and preserves specified behavior. Our evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes. We examine performance across domains and task information requirements, alongside the development behaviors associated with successful repairs. CUA-SWE provides a unified testbed for studying how agents use visual feedback and interaction to guide software engineering, with executable correctness criteria for the resulting software.

Figures & tables

Panel of Figure 1 — Figure 1: CUA-SWE benchmark overview and Frontier agents performance. Left: Agents inspect screenshots, interact with applications, and edit code; independent tests check their repairs. Right: Mean task success of Code-only and Hybrid CUA agents across four equally weighted domains. Red labels show the Hybrid CUA gain in percentage points.

Figure 2: The CUA-SWE development and evaluation pipeline.

Figure 3: A concrete benchmark task: repairing inventory updates in Captain Callisto. Agents combine coding tools with interaction and screenshot observations. In Captain Callisto, one pickup incorrectly changes the inventory from 0/2 to 2/2. The code excerpt shows the agent’s final edit, while “Correct behavior” shows a separate reference replay. Independent evaluation checks both the protected behavioral tests and the validity of tool use. The table lists expected outcomes for two replay scenarios.

Figure 4: Mobile example: repairing imported routes in Campus Pocket. The provider report identifies a closed lift and the travel costs used to construct a step-free route. The faulty route uses the closed H6 lift; the reference repair follows H3, H4, and H5 for a total of 74 m. The figure shows original-application and reference replays, with local route and persistence checks. Its agent loop illustrates the available tools.

Figure 5: Web example: preserving distinct masked-date input behaviors. Clearing the appointment date and leaving the field should remove the selection, while invalid nonempty input should preserve the previous date. The published agent frames and reference replay show the cleared state. The source excerpt is from the reference repair; the agent loop illustrates the coding and interaction interface.

	Web		Game		DevOps		Mobile
Model	Code	Hybrid CUA	Code	Hybrid CUA	Code	Hybrid CUA	Code	Hybrid CUA
GPT-6-Astra	27.8	66.7	17.2	37.9	0.0	80.0	0.0	55.0
GPT-5.6-Sol	19.4	58.3	10.3	10.3	0.0	55.0	0.0	45.0
Claude-Opus-5	22.2	55.6	13.8	20.7	0.0	60.0	—	—
Claude-Opus-4.8	16.7	27.8	3.4	20.7	0.0	55.0	0.0	40.0
Claude-Sonnet-5	8.3	0.0	6.9	3.4	0.0	40.0	0.0	35.0

Table 1: Single-attempt task success (%) across four domains. Code denotes code-only access; hybrid CUA adds screenshots and graphical actions. Teal marks column maxima; an em dash indicates no reported result. Full results: Appendix G , Table 8 .

Panel of Figure 6 — Figure 6: Success, time, and interaction effort. (a) Task success rates and mean episode times for nine models and systems, averaged equally across the four domains (circles: frontier models; open square: Codex). Appendix A.6 specifies resource coverage and aggregation. (b) Completed responses on the same Web tasks solved by both GPT-6-Astra and each peer (18 tasks per comparison); above the diagonal, GPT-6-Astra uses fewer responses.

Figure 7: SFT performance on Web domain. (a) Batch loss and its 50-update mean (log scale). (b) Native-verifier pass rate, one greedy attempt per task.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: From a seed example to a verified difficulty ladder. An LLM author turns a seed into a runnable task. Human review and executable checks establish task validity. Reviewed agent trials drive a separate expansion loop, with every new variant returning through both gates. The chart panels show the rendered application and its visible reference from a recorded observation. The ladder illustrates increasing layout coverage; trials assess the difficulty of proposed variants. Validated task versions support benchmark evaluation and post-training.

Panel of Figure 9 — Figure 9: The support-schedule and 2048 applications with reference repairs applied, illustrating the reference solutions.

Figure 10: A replay can show that more work is needed. Three frames from one Hextris run show the initial score of nine, an unsuccessful recheck after the first edit, and a later cascade reaching 27. The intervening work includes further inspection, temporary diagnostics, and reloads. Screenshots are cropped around the board; the task’s implementation hint is above the crop. The final source change is checked separately by the evaluator.

Domain	Task	Type	Specification basis
Web	MapLibre reset touch reacquisition	S	The instruction specifies pointer attachment and preservation of navigation.
Web	Fabric opposite-edge resize	S	Readable Help in index.html specifies the resize and selection contract.
Web	ECharts linked data-zoom toolbar	S	The instruction specifies corresponding chart ranges; source exposes range-mapping strategies.
Web	Fabric viewport rotation controls	S	The instruction specifies frame, handle, and zoom agreement under camera transforms.
Web	JupyterLab Shift-click cell selection	S	The instruction distinguishes cell-range selection from extension of text selection.
Web	Fabric through-edge resize with nested camera	S	Readable Help in index.html specifies the resize and selection contract.

Appendix

Table 2: Task information requirements. M denotes application-material dependent; S denotes source-specified (Appendix A.3 ). Each row gives the specification basis. Names recur across distinct scenarios; tasks follow the released manifest order within each domain.

Supervised action	Copies per task	Screenshots in context
Start browser	2	0
Apply gold patch	32	1
Validate repair	4	1
Wait and observe	2	1
Finish episode	4	2
Total	44	—

Appendix

Table 3: SFT prefix construction for each domain. Each row supervises the indicated action, conditioned on the preceding conversation. Screenshot counts refer to images available before that action.

Figure 11: RLVR extension using verifiable software repairs. For the same task, the policy samples several development trajectories in separate workspaces. Shell commands and computer actions return text and screenshots. At termination, permitted source changes are transferred to a clean scoring copy and checked by deterministic tests. Complete, successfully scored groups provide relative advantages for a policy update; observations condition the policy but receive no direct prediction loss. Application thumbnails show example Web and Game applications.

Domain	GPT-6- Astra	GPT-5.6- Sol	Claude- Fable-5
Web	66.7 / 75.0	58.3 / 58.3	47.2 / 61.1
Game	37.9 / 62.1	10.3 / 31.0	17.2 / 41.4
DevOps	80.0 / 90.0	55.0 / 60.0	55.0 / 70.0
Mobile	55.0 / 60.0	45.0 / 50.0	40.0 / 40.0

Appendix

Table 4: Three-attempt hybrid CUA success (%). Each cell reports pass@1 / pass@3. Pass@1 uses the first attempt; pass@3 uses all three.

Domain	Model	Mean of three	Pass@2	Pass@3	Diff. (pp)
Web	GPT-6-Astra	67.6	72.2	75.0	+7.4
	GPT-5.6-Sol	52.8	57.4	58.3	+5.6
	Claude-Fable-5	52.8	59.3	61.1	+8.3
Game	GPT-6-Astra	50.6	59.8	62.1	+11.5
	GPT-5.6-Sol	20.7	29.9	31.0	+10.3
	Claude-Fable-5	27.6	36.8	41.4	+13.8

Appendix

Table 5: Task success across attempt budgets (%). Mean of three is the average single-attempt success within this repeated cohort. Pass@2 averages all two-attempt subsets; pass@3 uses all three. The difference is pass@3 minus mean single-attempt success, in percentage points. Results cover Web and Game.

Domain	Model	0 successes	1 success	2 successes	3 successes
Web	GPT-6-Astra	25.0	8.3	5.6	61.1
	GPT-5.6-Sol	41.7	2.8	11.1	44.4
	Claude-Fable-5	38.9	5.6	13.9	41.7
Game	GPT-6-Astra	37.9	6.9	20.7	34.5
	GPT-5.6-Sol	69.0	3.4	24.1	3.4
	Claude-Fable-5	58.6	13.8	13.8	13.8

Appendix

Table 6: Success frequency across three attempts. Entries give the percentage of tasks with zero, one, two, or three successful attempts. Each row sums to 100% before rounding. Results cover Web and Game.

Domain	Compared with	Measure	Difference	Task 95% CI	Family 95% CI
Web	GPT-5.6-Sol	Mean of three	+14.8	[3.7, 25.9]	[-1.3, 33.3]
	GPT-5.6-Sol	Pass@3	+16.7	[2.8, 30.6]	[-6.7, 42.9]
	Claude-Fable-5	Mean of three	+14.8	[3.7, 25.9]	[0.0, 32.5]
	Claude-Fable-5	Pass@3	+13.9	[0.0, 27.8]	[-4.3, 33.3]
Game	GPT-5.6-Sol	Mean of three	+29.9	[17.2, 43.7]	[15.3, 47.2]
	GPT-5.6-Sol	Pass@3	+31.0	[13.8, 48.3]	[11.5, 50.0]

Appendix

Table 7: Paired task-success differences for GPT-6-Astra (pp). Intervals are 95% percentile intervals from 20,000 paired resamples. Task resampling operates within each domain; family resampling selects whole task families and retains task weighting. All model outcomes and three attempts for each task remain together. Results cover Web and Game.

	Web		Game		DevOps		Mobile
Model / system	Code	Hybrid CUA	Code	Hybrid CUA	Code	Hybrid CUA	Code	Hybrid CUA
GPT-6-Astra	27.8	66.7	17.2	37.9	0.0	80.0	0.0	55.0
GPT-5.6-Sol	19.4	58.3	10.3	10.3	0.0	55.0	0.0	45.0
GPT-5.6-Luna	11.1	5.6	0.0	6.9	0.0	15.0	0.0	35.0
GPT-5.6-Terra	11.1	13.9	3.4	6.9	0.0	25.0	0.0	35.0
Claude-Opus-5	22.2	55.6	13.8	20.7	0.0	60.0	—	—

Appendix

Table 8: Single-attempt task success (%) across four domains. Code denotes code-only access; hybrid CUA adds screenshots and graphical actions. Teal marks column maxima; an em dash indicates no reported result.

Figure 12: CUA task success on Web (top) and Game (bottom). Bars report success rates for nine frontier models and two agent systems over the full task inventories: 36 tasks on Web and 29 on Game.

Figure 13: CUA task success on DevOps (top) and Mobile (bottom). Bars report CUA task success rates for the evaluated models and agent systems. Colors follow Figure 12 .

Figure 14: Task success and median development time within each domain. Points pair task success (vertical axis) with median agent time (horizontal axis, logarithmic). Resource coverage and aggregation are specified in Appendix A.6 . Colored rings and direct labels highlight GPT-6-Astra, Claude-Opus-5, and Claude-Fable-5; the shared legend names the other entries. Circles denote frontier models; open and filled squares denote Codex and Claude Code, respectively, with nested squares for the nearly coincident Web estimates. Success axes span 0–50% for Game and 0–100% elsewhere.

Figure 15: CUA task success across task families. Cells report task success rates within each family, with purple intensity encoding the percentage. Web families group applications and tools; Game families retain source-game identity; DevOps and Mobile follow task-contract categories. Rows below the separator show agent systems. Rates use each family’s full task inventory. Em dashes denote unevaluated models. Question marks denote Grok-4.6’s two Mobile family estimates with unresolved task outcomes: 0–11.1% for Gantt schedules and 0–100% for Marble run (Appendix A.5 ).

Figure 16: Task-level Web outcomes. Rows cover all 36 Web tasks, grouped by task family. Each frontier model has adjacent code-only (C) and CUA (U) cells; each agent system has one CUA cell. Colors encode the recorded outcome according to the key, and gray em dashes indicate unavailable outcomes.

Figure 17: Task-level Game outcomes. Rows cover all 29 Game tasks, grouped by source game. Adjacent C and U cells show code-only and CUA outcomes for frontier models; the two system columns show CUA outcomes. The encoding follows Figure 16 .

Figure 18: Task-level DevOps outcomes. Rows cover all 20 DevOps tasks, grouped by operational contract. Adjacent C and U cells show code-only and CUA outcomes for frontier models; the two system columns show CUA outcomes. The encoding follows Figure 16 .

Figure 19: Task-level Mobile outcomes. Rows cover all 20 Mobile tasks, grouped by application contract. Adjacent C and U cells show code-only and CUA outcomes for frontier models; the two system columns show CUA outcomes. Gray em dashes denote unevaluated models. Grok-4.6’s two question marks identify tasks with one success between them and unresolved individual outcomes (Appendix A.5 ). Other colors follow Figure 16 .

Figure 20: Task-paired outcomes under the two access conditions. Each bar partitions a model’s matched tasks by the joint code-only and CUA outcome. The four segments sum to 100%. Values at right give the CUA minus code-only success difference in percentage points, computed as the share solved only with CUA minus the share solved only with code. Task membership is identical across conditions within each model and domain. Both conditions include source-level execution and command feedback; hybrid CUA additionally provides application screenshots and graphical actions. Information-requirement strata are reported in Table 10 .

Domain	Type	Tasks	Code-only	Hybrid CUA	$\Delta$ (pp)
Web	S	22	23.9	23.9	+0.0
Web	M	14	8.9	50.9	+42.0
Game	S	20	10.6	10.0	-0.6
Game	M	9	2.8	26.4	+23.6
DevOps	M	20	0.6	48.8	+48.1
Mobile	M	20	0.6	39.4	+38.8

Appendix

Table 9: Success by task information requirement, averaged over the eight frontier models evaluated under both access conditions in all four domains. Each domain–type group uses the same tasks for both conditions. M: application-material dependent; S: source-specified. The labels identify specification provenance; both groups require implementation and verification. Full model results appear in Appendix I .

Type	Model	$N$	Code-only	Hybrid CUA	$\Delta$
Web
S	GPT-6-Astra	22	27.3	54.5	+27.3
S	GPT-5.6-Sol	22	27.3	36.4	+9.1
S	GPT-5.6-Luna	22	13.6	4.5	-9.1
S	GPT-5.6-Terra	22	18.2	13.6	-4.5
S	Claude-Opus-5	22	27.3	31.8	+4.5

Appendix

Table 10: Task success by information requirement (%). Each row compares the same tasks and original single attempts under code-only and hybrid CUA access. M and S follow Appendix A.3 ;

\Delta

is hybrid minus code-only in percentage points. DevOps and Mobile contain no S tasks; their S results are not applicable. Only models evaluated under both access conditions are included.

Figure 21: Hybrid execution efficiency. Model responses per task, output tokens per response, and agent minutes per success for eight frontier models. Episode workloads are averaged within each domain and then weighted equally across the four domains. Agent minutes per success divides that mean cost by the full-inventory task-success probability. Appendix A.6 specifies workload coverage.

Figure 22: Interaction steps, response length, and development time. Hollow circles show code-only and filled circles show CUA. Markers give task medians; intervals span the 25th–75th percentiles. Steps count completed model responses. Response length is recorded output tokens divided by completed responses for each episode, summarized across tasks. Time covers the agent episode. Each metric uses the same eligible tasks across conditions within a model and domain, retaining the full range of task outcomes. Resource coverage is specified in Appendix A.6 .

Figure 23: Effort ratios on shared successful tasks. Each comparison uses the tasks successfully completed by both GPT-6-Astra and the corresponding frontier model with CUA. For each task, effort is divided by the peer’s effort; markers show median ratios and intervals span the 25th–75th percentiles. The line at one indicates equal effort. Axes are logarithmic. Steps count model responses; response length uses recorded output tokens per response; time covers the agent episode. Each peer has its own shared-success cohort, whose size appears in the top row. Pairs include executions with resource measurements for both models (Appendix A.6 ).

Figure 24: Response counts on shared successful Web tasks. Each connected pair gives GPT-6-Astra’s and its peer’s response counts on one task. The GPT-6-Astra–Grok-4.6 comparison covers 18 shared successes, with medians of 28.5 and 43.5 responses. The GPT-6-Astra–Claude-Opus-5 comparison also covers 18 shared successes, with medians of 21.5 and 27.5 responses. Each panel uses its corresponding pair’s shared-success task set.

Figure 25: Recorded input and cached-input tokens per CUA task. Markers show medians and intervals span the 25th–75th percentiles over each entry’s measured CUA cohort. The two counters retain their provider-specific definitions and are summarized separately. Input tokens use a logarithmic axis. Cached-input tokens use a symmetric logarithmic axis with a linear region from zero to 1,000 tokens; tick labels give token counts. Token summaries use episodes with complete recorded usage.

Figure 26: CUA interaction and output profiles. Markers show medians and intervals span the 25th–75th percentiles over each entry’s measured task cohort. Circles denote frontier models and diamonds denote agent systems. GUI actions and recorded screenshots use linear axes; output tokens use a logarithmic axis. Screenshot counts retain each source’s capture definition, and token summaries use complete recorded usage.

Figure 27: Native agent-system steps with CUA. Codex with GPT-5.6-Sol counts completed items; Claude Code with Claude-Opus-5 counts native assistant turns. The two step units appear in separately labeled panels. Pale points show individual tasks, diamonds give medians, and thick intervals span the 25th–75th percentiles. Each panel uses all available native step measurements for the system and domain.

Figure 28: Successive interaction-guided edits on Web. GPT-6-Astra repairs resizing, tests selection beyond the containing frame, and extends hit testing to selectable descendants. The scene panels are conceptual schematics of the reviewed trajectory; the edit description summarizes the source change. The resulting patch passes the task’s evaluator.

Figure 29: Coordinated restart restoration in Game. GPT-6-Astra combines lift-phase recovery, course-space camera restoration, and support-relative player attachment. The phase curve is a source-based schematic with an illustrative echo height, and the turn-18 text paraphrases the released first-valid trajectory. Deterministic state replay verifies the resulting repair. This episode belongs to the single-attempt evaluation in Table 8 .

Figure 30: Recovering gauge-update semantics in DevOps. The recorded observation and its update-history crop show behavior during development. Opening provisioning pulses reveal producer update modes; the source excerpts preserve signed update tokens and apply the inferred modes. The final browser check verifies GPT-6-Astra’s repair.

Figure 31: Translating a Mobile drawing into gear relationships. The two recorded drawing views guide GPT-6-Astra’s graph repair. The source excerpt shows four selected added edges from the patch. The completed repair passes all six task assertions.

	Code-only			Hybrid CUA
Model / system	Patch	Task	$P{=}1,U{=}0$	Patch	Task	$P{=}1,U{=}0$
Web ( $N=36$ )
GPT-6-Astra	27.8	27.8	0.0	66.7	66.7	0.0
GPT-5.6-Sol	19.4	19.4	0.0	58.3	58.3	0.0
GPT-5.6-Luna	11.1	11.1	0.0	13.9	5.6	8.3
GPT-5.6-Terra	11.1	11.1	0.0	13.9	13.9	0.0

Appendix

Table 11: Patch correctness and task success (%). Patch reports

P=1

; Task reports

PU=1

; the third column reports

P=1,U=0

. Each evaluated entry uses the same

N

tasks and single attempts as Table 8 . An em dash denotes an unevaluated condition.

Domain	Condition	$n$	O	Perc	D	I	V
Web	Code-only	18	18/0/0	0/18/0	2/14/2	5/0/13	18/0/0
Web	Hybrid	22	16/6/0	6/13/3	6/8/8	12/3/7	21/1/0
Game	Code-only	18	18/0/0	0/16/2	3/13/2	16/0/2	18/0/0
Game	Hybrid	22	15/7/0	6/12/4	7/9/6	21/0/1	22/0/0
DevOps	Code-only	18	18/0/0	1/16/1	14/0/4	16/0/2	18/0/0
DevOps	Hybrid	22	13/8/1	6/12/4	1/17/4	15/0/7	22/0/0

Appendix

Table 12: Failure-stage annotations by domain and access condition. Each cell gives supported/not observed/unknown counts among

n

sampled task failures. O: observation; Perc: interpretation; D: diagnosis; I: implementation; V: verification coverage. Stages may overlap.

Model / system	Condition	$n$	O	Perc	D	I	V
Web
GPT-6-Astra	Code-only	2	2/0	0/0	0/0	1/1	2/0
GPT-6-Astra	Hybrid	2	2/0	0/0	0/1	1/1	2/0
GPT-5.6-Sol	Code-only	2	2/0	0/0	0/0	0/2	2/0
GPT-5.6-Sol	Hybrid	2	2/0	0/0	1/0	0/2	2/0
GPT-5.6-Luna	Code-only	2	2/0	0/0	0/1	0/2	2/0

Appendix

Table 13: Failure-stage counts for each model and domain. Cells report supported/unknown counts; the remaining annotations are not observed. Counts describe the outcome-stratified review sample, with at most two failed attempts per model, domain, and condition. Stage definitions are shared with Table 12 .

Domain	Condition	$N/E$	Before edit Y/N/U	No runtime information	No GUI observation
Web	Code-only	27/27	0/27/0	21	27
Web	Hybrid	32/32	19/13/0	0	4
Game	Code-only	26/24	0/24/0	13	24
Game	Hybrid	33/31	14/16/1	0	2
DevOps	Code-only	19/18	1/17/0	3	18
DevOps	Hybrid	33/32	32/0/0	0	0

Appendix

Table 14: Observation and editing in the reviewed trajectories.

N/E

gives all/edited attempts. Before-edit observation reports yes/no/unknown among the

E

edited attempts; no-edit attempts are excluded. The last two columns count edited attempts with no relevant runtime information or no relevant GUI observation at any point.

			Task success (%)		Patch correctness (%)
Domain	GUI check	Y/N/U	Y	N	Y	N
Web	Post-edit	27/5/0	47.6	6.9	61.2	6.9
Web	Final-edit	24/7/1	45.8	5.7	61.3	5.7
Game	Post-edit	10/9/12	16.6	9.3	16.6	9.3
Game	Final-edit	7/11/13	16.6	12.8	16.6	12.8
DevOps	Post-edit	23/9/0	68.1	5.9	68.1	5.9

Appendix

Table 15: GUI re-verification and repair outcomes among edited hybrid attempts. Y/N/U are raw counts with an observed/absent/unknown GUI check. Task success

PU

and patch correctness

P

are inverse-inclusion-weighted percentages, shown separately for Y and N. Unknown check labels are retained in U and excluded from the Y–N comparison. All protocol statuses are included.

		Image deliveries		GUI actions
Model / system	$n$ (S/F)	S	F	S	F
Web
GPT-6-Astra	24/12	7.5 [6, 16.8]	19 [14, 26.5]	6 [4, 15.5]	17.5 [12.2, 25.2]
GPT-5.6-Sol	21/15	7 [5, 10]	13 [9.5, 18]	6 [4, 8]	12 [8.5, 17]
GPT-5.6-Luna	2/34	4 [3.5, 4.5]	2.5 [1, 3]	3.5 [3.2, 3.8]	3.5 [1, 7]
GPT-5.6-Terra	5/31	4 [2, 4]	2 [1, 4.5]	3 [2, 4]	3 [1, 5]

Appendix

Table 16: Observed image deliveries and GUI actions by repair outcome. Every entry is a median [first quartile, third quartile] over hybrid task successes (S) or failures (F);

n

gives S/F attempts. An em dash marks an empty success group. Counts are retained-log observations with interface-specific units, defined in the text.

Domain	Model	$N$	Task C/H	Patch C/H	Hybrid wins/losses	$P{=}1,U{=}0$
Web	GPT-5.6-Luna	36	4 / 2	4 / 5	0 / 2	3
Web	Claude-Sonnet-5	36	3 / 0	3 / 18	0 / 3	18
Game	Claude-Sonnet-5	29	2 / 1	2 / 3	1 / 2	2

Appendix

Table 17: Matched tasks for model–domain pairs with lower hybrid task success. C/H denotes code-only/hybrid. Wins and losses compare task success on the same task. The final column counts all hybrid passing patches with

U=0

, including tasks where code-only also fails.

Annotation	Agreement	(%)	$\kappa$
Editing before relevant observation	31/32	96.9	0.939
Relevant GUI behavior observed at any point	32/32	100.0	1.000
Final-edit GUI re-verification	28/32	87.5	0.771
Post-edit GUI re-verification	28/32	87.5	0.775
Relevant observation before first edit	29/32	90.6	0.819
Relevant runtime information observed	32/32	100.0	1.000

Appendix

Table 18: Reviewer agreement before adjudication. The same 32 trajectories receive both annotations. Unknown is an explicit category in nominal Cohen's

\kappa

Computer-Use Agent Benchmarks

Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.

See all 120 papers

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
Jun 29, 2026Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal +1Coding Agents Software Engineering Benchmarks
1Scale AI
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Jun 8, 2026Wanli Li, Bowen Zhou, Yunyao Yu +4Agent Benchmarks Computer-Use Agents
Microsoft Research Asia · Tsinghua University
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI

Jun 29, 2026J/K move · Enter open · S save

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He

1Scale AI

Coding Agents Software Engineering Benchmarks AI Agent Evaluation User Simulation

Open paper AskPDF arXiv

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.

Abstract

Computer-Use Agent Benchmarks

Explore similar work

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark