cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Pranjal Aggarwal; Lawrence Keunho Jang; Sean Welleck; Daniel Fried; Ruslan Salakhutdinov; Jing Yu Koh

cs.LGSep 30, 2026

cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Authors: Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh

Organizations: Carnegie Mellon University

Abstract

Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.

Figures & tables

Figure 1: cua-speedrun overview. cua-speedrun is a standardized platform for benchmarking the performance, speed, and cost of computer-use agents. It combines representative task selection with standardized infrastructure on Modal, enabling consistent comparisons across models, agent harnesses, and reasoning settings. Our findings reveal that more reasoning can reduce both task time and model cost for CUAs, while faster environment I/O can make agents slower.

Figure 2: Different agents define the speed and cost frontiers. Mean score vs. mean task time and model cost on OSWorld (top, 54 configs) and OSWorld2 (bottom, 19 configs), excluding MiniMax M3. Dashed lines connect the observed Pareto-frontier points, which are circled. Colors and marker shapes identify models, and darker shades indicate increased reasoning effort. The plots compare complete agent configs. Costs use recorded charges or token usage at the applicable API prices. Astra (low) uses five-seed means for each I/O setting. Full configs and plots in Appendix E .

Figure 3: Less reasoning can make an agent slower. Gemini 3 Flash Preview at low, medium, and high effort on OSWorld. The panels show mean task time, generated tokens including reasoning, and model calls across all 50 tasks. Scores are 33.6%, 57.6%, and 59.6%, respectively. Medium and high effort generate more tokens than low effort but require fewer calls and complete tasks faster.

Figure 4: The useful reasoning setting changes across benchmarks. Gemini 3.8 Flash at low, medium, and high effort; labels also report mean cost per task. Each curve keeps the model, agent harness, and execution settings fixed. Additional reasoning brings no gain on OSWorld, whereas moving from low to medium substantially improves OSWorld2 performance.

Figure 5: Faster I/O can be offset by more waiting and agent interaction. GPT-6 Astra (low), with normal and fast I/O. Panel (a) separates agent time, agent-requested waits, and remaining environment processing, averaged over five seeds per setting; error bars show the standard deviation of mean task time across seeds. Panel (b) shows two consecutive screenshots after opening Save As: the dialog is absent in the first and visible in the second, requested 5.8 s later with no intervening action.

Figure 6: Choosing an agent by performance, time, and cost. Joint Pareto frontier on OSWorld; the OSWorld2 frontier is in Appendix E , Fig. 13 . Circled points are the configurations on the Pareto frontier. Time and cost axes are logarithmic.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Infrastructure	Click	Escape	Ctrl+U	Type 100 characters
Gym-Anything	2758.70	2865.00	2858.29	3521.72
OSWorld	2515.41	2707.74	2708.41	2756.47
Fast I/O (ours)	27.58	2.43	2.91	14.98

Appendix

Table 1: Fast I/O reduces action-to-observation latency to milliseconds. Median latency in milliseconds over 20 calls per action. For each system, the timer starts immediately before env.step(action) and stops when it returns the observation. All default action waits are included.

Gym-Anything
Component (ms)	Click	Escape	Ctrl+U	Type 100 characters
Fixed waits	2100.2	2100.2	2100.2	2100.2
Action request	153.5	153.7	154.0	805.5
Screenshot capture	258.9	354.0	354.3	354.1
Screenshot transfer	238.1	212.4	220.6	211.3
Cleanup and other	53.1	53.1	53.1	53.1

Appendix

Table 2: Fixed waits account for most baseline latency. Mean component times from 10 profiled calls per action for Gym-Anything and 20 for OSWorld. Components sum to the total before rounding. The final row for each system is the profiled total minus its two-second post-action wait.

Component (ms)	Click	Escape	Ctrl+U	Type 100 characters
Input execution	27.30	1.85	2.55	18.39
Frame capture	1.03	0.77	0.77	0.80
Foreground image preparation	0.27	0.02	0.02	0.18
Other environment processing	0.93	0.94	1.00	0.90
Transport	4.67	4.99	4.76	4.62
Screenshot-file delivery	0.43	0.45	0.43	0.44

Appendix

Table 3: Fast I/O completes the four tested actions in under 50 ms. Mean times over 20 commands per action, from the agent issuing a command to receiving the screenshot file. This includes command overhead, communication between sandboxes, and screenshot-file delivery in addition to the env.step call. Components sum to the total before rounding.

Run	Score (%)	Time (s)	Agent (s)	Steps	Env. (s)	Waits (s)	Env. $-$ waits (s)
1	91.62	103.96	98.07	8.68	5.891	5.553	0.338
2	91.62	93.98	89.02	8.04	4.955	4.628	0.327
3	91.62	96.48	91.67	7.96	4.812	4.488	0.324
4	87.62	100.22	95.28	8.20	4.934	4.604	0.330
5	89.62	100.46	95.68	8.12	4.785	4.424	0.361
Mean	90.42	99.02	93.94	8.20	5.075	4.739	0.336

Appendix

Table 4: Fast I/O remains consistent across five evaluations. Each row averages all 50 OSWorld tasks for GPT-6 Astra at low reasoning effort. SD is the sample standard deviation across run-level means; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent.

ID	Configuration	Batches	Actions	Responses	Tokens
1	GPT-6 Astra / xhigh	7.14	19.90	–	2,060
2	Gemini 3.8 Flash / low	16.60	20.02	18.52	1,564
3	Claude Opus 5 / high	19.92	26.92	20.62	3,735
4	Gemini 3.8 Flash / medium	27.16	34.70	29.10	5,365
5	GPT-6 Astra / medium	6.30	19.26	–	1,366
6	GPT-6 Astra / high	6.52	19.76	–	1,593

Appendix

Table 5: Interaction and token counts on OSWorld. Means per task, including unsuccessful tasks. IDs match the OSWorld configuration tables in Appendix E . Tokens include reasoning and other output. Astra (low) uses five-seed means for each I/O setting.

ID	Configuration	Batches	Actions	Responses	Tokens
1	GPT-6 Astra / xhigh	52.81	194.10	–	18,332
2	GPT-6 Astra / high	51.29	149.02	–	14,146
3	GPT-6 Astra / low	42.58	151.40	–	10,190
4	GPT-6 Astra / medium	45.46	151.08	–	11,382
5	Gemini 3.8 Flash / high	216.54	331.88	218.48	72,149
6	Gemini 3.8 Flash / medium	167.79	252.65	169.65	49,532

Appendix

Table 6: Interaction and token counts on OSWorld2. Means per task, including unsuccessful tasks. IDs match the OSWorld2 configuration tables in Appendix E . Tokens include reasoning and other output.

Figure 7: Performance–interaction and performance–token frontiers on OSWorld. Mean score against mean action batches, individual actions, and generated tokens per task. Tokens include reasoning and other output. Panels (a)–(b) show all 56 configurations; panel (c) shows the 54 with token counts. Dashed lines connect the circled frontier points. IDs match the configuration tables in Appendix E ; darker green indicates greater reasoning effort. Astra (low) uses five-seed means for each I/O setting.

Figure 8: Performance–interaction and performance–token frontiers on OSWorld2. Mean score against mean action batches, individual actions, and generated tokens per task. Panels (a)–(b) show all 21 configurations; panel (c) shows the 19 with token counts. IDs match the OSWorld2 configuration table in Appendix E . Other plotting conventions follow Figure 7 .

ID	Configuration	Score (%)	Time (s)	Cost ($)	Tokens
1	GPT-6 Astra / xhigh	91.6	126.8	0.708	2,060
2	Gemini 3.8 Flash / low	91.6	127.1	0.108	1,564
3	Claude Opus 5 / high	91.6	136.0	0.661	3,735
4	Gemini 3.8 Flash / medium	91.6	250.1	0.221	5,365
5	GPT-6 Astra / medium	89.6	90.2	0.600	1,366
6	GPT-6 Astra / high	89.6	100.8	0.636	1,593

Appendix

Table 7: OSWorld configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.

ID	Configuration	Score (%)	Time (s)	Cost ($)	Tokens
27	GPT-5.6 Luna / low / Codex	75.6	144.8	0.025	3,191
28	MiniMax M3 / thinking_off	75.6	253.8	–	–
29	GPT-5.6 Luna / high / API	81.6	160.5	0.029	3,358
30	Kimi K3 / low / batched	73.6	149.5	0.174	1,865
31	Muse Spark 1.3 / high	73.6	506.9	0.148	10,538
32	Kimi K3 / low / single	71.6	176.6	0.280	3,171

Appendix

Table 8: OSWorld configurations, continued. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.

ID	Configuration	Score (%)	Time (s)	Cost ($)	Tokens
1	GPT-6 Astra / xhigh	76.9	1237.3	8.175	18,332
2	GPT-6 Astra / high	75.0	914.0	7.391	14,146
3	GPT-6 Astra / low	68.2	904.2	6.336	10,190
4	GPT-6 Astra / medium	67.2	828.1	6.519	11,382
5	Gemini 3.8 Flash / high	60.2	3058.8	5.019	72,149
6	Gemini 3.8 Flash / medium	59.9	2051.8	2.915	49,532

Appendix

Table 9: OSWorld2 configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.

Figure 9: Full OSWorld performance–time and performance–cost comparisons. All 56 configurations appear in the time comparison; the cost comparison uses configurations with reported or estimated costs. IDs identify the model, reasoning effort, and harness in Tables 7 – 8 . Dashed lines connect nondominated configurations for each pair of axes. Scores, times, and costs are means across the same 50 tasks, including unsuccessful tasks.

Figure 10: Full OSWorld comparison at a common token rate. Both plots use the same 54 configurations with token counts. The right plot replaces measured mean task time with mean recorded environment time plus generated tokens divided by 100 tokens/s. Tokens include reasoning and other output; scores and trajectories are unchanged. Point IDs refer to the OSWorld configuration tables.

Figure 11: Full OSWorld median-time view and fast-region detail. Mean verifier score versus median task time for all 56 configurations. For Astra (low), the median task times are averaged across five seeds. The right plot enlarges the region containing the fastest configurations; point IDs match the full OSWorld tables. Green dotted lines show the published human references of 72.36% and 111.94 s from the original 369-task OSWorld benchmark ( Xie et al., 2024 ) .

Figure 12: Full OSWorld2 performance–time and performance–cost comparisons. All 21 configurations are evaluated on the same 52 tasks. Point IDs refer to the OSWorld2 configuration table. Marker shapes follow Figure 2 ; darker green indicates greater reasoning effort.

Figure 13: Joint performance–time–cost frontier on OSWorld2. Circled points are the evaluated frontier configurations. Other configurations appear in gray. Time and cost axes are logarithmic. Marker shapes follow Figure 2 ; darker green indicates greater reasoning effort.

Figure 14: Performance–time and performance–cost frontiers on MyPCBench and CUA-World. GPT-6 Astra through Codex at four reasoning settings. Each point averages the same 38 MyPCBench tasks (top) or 26 CUA-World tasks (bottom). Darker green indicates greater reasoning effort. Dashed lines connect the observed Pareto-frontier points, which are circled.

Effort	Score (%)	Time (s)	Cost ($)	Score (%)	Time (s)	Cost ($)
	MyPCBench (38 tasks)			CUA-World (26 tasks)
Low	89.9	514	4.57	90.3	807	9.29
Medium	88.3	495	4.33	86.7	810	8.42
High	88.6	568	4.81	94.2	1,108	11.11
Xhigh	93.6	619	4.81	93.0	1,515	15.06

Appendix

Table 10: Reasoning effort on MyPCBench and CUA-World. GPT-6 Astra through Codex, reporting mean partial-credit score, time, and model cost per task. Bold indicates the highest score, lowest time, or lowest cost within each benchmark.

Run	Score (%)	Time (s)	Agent (s)	Steps	Env. (s)	Waits (s)	Env. $-$ waits (s)
1	87.62	86.11	72.90	5.90	13.209	1.328	11.881
2	91.62	89.56	75.31	5.82	14.248	1.082	13.166
3	91.62	90.13	76.52	5.88	13.606	1.040	12.566
4	91.62	89.81	75.78	5.78	14.033	1.152	12.881
5	91.62	92.10	78.19	5.94	13.914	1.012	12.902
Mean	90.82	89.54	75.74	5.86	13.802	1.123	12.679

Appendix

Table 11: Variation across five seeds for GPT-6 Astra at low reasoning effort. Each row averages all 50 OSWorld tasks. SD is the sample standard deviation of these means across the five runs; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent.

I/O	$k$	Pass@ $k$ (%)	Best-of- $k$ partial score (%)
Standard I/O	1	87.20	90.82
	2	88.00	91.62
	3	88.00	91.62
	4	88.00	91.62
	5	88.00	91.62
Fast I/O	1	86.80	90.42

Appendix

Table 12: Task-level repeatability across five seeds of Astra at low effort. Values average uniformly over all size-

k

subsets of the five runs, then over the 50 tasks. Pass@

k

measures whether any of the

k

attempts succeeds; best-of-

k

uses the highest partial score.

Benchmark	Full set	Selected subset
OSWorld ( Xie et al., 2024 )	295	50
OSWorld2 ( Yuan et al., 2026 )	63	52
CUA-World ( Aggarwal et al., 2026 )	143	26
MyPCBench ( Jang et al., 2026a )	184	38

Appendix

Table 13: Task counts for the representative subsets selected for cua-speedrun .

Figure 15: Score error and rank correlation on held-out models. OSWorld uses nine leave-one-agent-out folds; OSWorld2 uses seven. The randomized baseline averages 300 difficulty-stratified selections per task count. Dotted vertical lines mark 50 and 52 tasks; horizontal lines mark rank correlation 0.95. Each task count is optimized independently.

Benchmark	$K$	Partial $\rho$	Exact $\rho$	Partial MAE	Exact MAE
OSWorld	49	0.967	0.967	4.36	4.57
	50	0.983	0.975	2.16	2.17
	51	1.000	0.992	4.21	3.95
OSWorld2	51	0.964	0.955	1.70	2.28
	52	0.964	0.955	1.33	2.03
	53	0.964	0.982	0.74	1.04

Appendix

Table 14: Held-out-model validation around the selected task count. MAE is in percentage points. Each model is excluded from task selection in its fold. Both correlations exceed 0.95 at the selected task count and its two neighbors.

Figure 16: Speed–performance frontiers from full-set scores and held-out subset estimates. Dark points and connecting lines mark nondominated configurations. Model IDs are: 1, Sonnet 5; 2, Gemini 3.6 Flash; 3, Gemini 3 Flash Preview; 4, GLM-5V Turbo; 5, Luna; 6, Kimi K3; 7, Muse Spark; 8, MiniMax M3; 9, Qwen3.5-9B. Each point in (b) uses a 50-task subset selected with that model held out.

Method	Partial MAE	Exact MAE	Mean MAE
Uniform random	5.88	6.01	5.95
Difficulty-stratified random	5.05	5.11	5.08
IRT-inspired selection	7.94	8.21	8.08
Energy-based selection	4.50	4.34	4.42

Appendix

Table 15: Task-selection methods on OSWorld with 32 tasks. Errors are averaged across nine leave-one-agent-out folds and reported in percentage points. Mean MAE averages partial-score and exact-completion MAE. Random baselines average 300 selections per fold.

Application	Task UUID
chrome	06fe7178-4491-4589-810f-2e2bc9502122
chrome	2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3
chrome	3720f614-37fd-4d04-8a6b-76f54f8c222d
chrome	44ee5668-ecd5-4366-a6ce-c1c9b8d4e938
chrome	af630914-714e-4a24-a7bb-f9af687d3b91
gimp	045bf3ff-9077-4b86-b483-a1040a949cff

Appendix

Table 16: Selected OSWorld task identities.

Environment	Task name
ardour_env	broadcast_podcast_stem_delivery
dhis2_env	rmncah_scorecard_dashboard
docker_desktop_env	diagnose_broken_microservices_stack
gpredict_env	poes_downlink_schedule_setup
gvsig_desktop_env	vulnerability_map_remote_communities
jstock_env	quarterly_portfolio_rebalance

Appendix

Table 17: Selected CUA-World task identities.

Reviewer or outcome	Included	Excluded
Reviewer 1	297	72
Reviewer 2	302	67
Reviewer 3	295	74
Unanimous decision	295	67
Disagreement	7

Appendix

Table 18: Independent inclusion decisions for 369 OSWorld tasks. The seven disagreements are excluded by the unanimous-inclusion rule.

Requested input	Received input	Cause
Type <	>	Incorrect X11 key/modifier mapping
Type accented text	Accented characters omitted	Characters absent from the key map
Type a literal shell variable	Variable expanded into a path	Text interpreted by the command shell
Keypad Enter or Menu	No corresponding key event	Missing key-name translation
Toggle Caps Lock, then type	Lowercase text	Caps Lock press silently omitted

Appendix

Table 19: Input errors detected by CUA-AutoDebug when executing actions through SSH and PyAutoGUI. The tests compare requested input with the keyboard and mouse events received.

Harness	Model response	Executed behavior
Gemini	Scroll down by five wheel clicks	Scroll up by 600 ticks
Gemini	Press F5 or Page Down	Type the key name as text
Qwen3.5	Middle-click the target	No action
Qwen3.5	Ctrl-click the target	Release Ctrl before clicking
Qwen3.5	Move, then scroll or drag	Execute only the first tool call

Appendix

Table 20: Harness errors can change a correct model action. Examples detected by comparing model responses with the actions executed and the input events received.

Action family	Operations
Mouse clicks	Left, right, middle, double, and triple click at a coordinate
Pointer motion	Move, drag along coordinates, hold or release a mouse button
Scrolling	Signed vertical scroll at the current pointer position
Keyboard	Type text, press a key or chord, hold and release modifiers
Waiting	Wait for an agent-specified duration
Observation	Request a screenshot without changing the desktop

Appendix

Table 21: Desktop actions supported by the infrastructure. A batch contains one or more actions executed in order. The model-facing tool schema can differ between harnesses.

Model	Effort	Interaction rule	Score	Time	Batches	Actions	Tokens
Astra	xhigh	Batched	91.62	126.82	7.14	19.90	2,060
Astra	xhigh	One action + image	91.62	182.73	16.50	16.50	2,559
Kimi K3	low	Single tool	71.62	176.61	8.12	55.10	3,171
Kimi K3	low	Batched tools	73.62	149.50	5.74	30.00	1,865
Kimi K3	high	Single tool	81.62	305.07	8.92	113.60	7,567
Kimi K3	high	Batched tools	79.62	358.13	8.76	60.20	7,125

Appendix

Table 22: Batching comparisons within the same model and reasoning effort on OSWorld. Score is a percentage; time is in seconds. All counts and times are means over the same 50 tasks. Tokens include reasoning and other output.

Figure 17: Harness performance depends on reasoning effort. GPT-5.6 Luna at low, medium, and high reasoning effort, using the direct-API agent or Codex on OSWorld. Each curve connects reasoning settings within one harness. All six evaluations use the same 50-task set.

Model	Tasks	Effort	Time	Batches	Actions	Tokens	Tok/s
Gemini 3.8 Flash	5	low	1291.0	116.40	171.60	29,596	47.57
		medium	1363.9	112.60	158.40	34,446	43.10
		high	3024.5	190.60	246.60	73,578	42.18
GPT-6 Astra	13	low	791.8	47.08	146.69	7,878	15.18
		medium	752.6	47.15	138.85	8,376	17.07
		high	817.3	53.15	171.23	11,001	21.31

Appendix

Table 23: Reasoning effort on tasks completed fully at every compared effort on OSWorld2. Every reported score is 100%. Time is in seconds per task; token rate divides total generated tokens by total agent time. Each model uses its own common task set.

Figure 18: Validation gains do not generalize to the held-out model. Across 34 candidate methods, the coding agent receives only validation errors. We measure test error separately on the held-out model. Each method selects 32 tasks. Error averages the mean absolute errors in partial score and exact completion, in percentage points.

Explore similar work

Jun 8, 2026cs.AI

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.

Wanli Li, Bowen Zhou, Yunyao Yu +4

Microsoft Research Asia · Tsinghua University

Computer-Use Agents Orchestration Traces

Ask PDF

May 7, 2026cs.SE

Computer Use at the Edge of the Statistical Precipice

Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically address. We show that a 1MB replay script that blindly executes a recorded action sequence without ever observing the screen outperforms frontier models on prominent static benchmarks, and prove that its expected success rate is exactly equal to the source agent's pass@k in deterministic environments. We trace this and other failures to two root causes: non-principled environment design (static, unsandboxed, or unreliably verified environments) and non-principled evaluation methodology (naive aggregation and misuse of pass@k for stateful UI interactions). To address the first, we propose PRISM, five design principles for CUA environments (privileged verification, realistic environments, integrity-checked configurations, sandboxed execution, and multifactorial variability) and instantiate them in DigiWorld, a benchmark of 15 realistic sandboxed mobile applications able to evaluate agents in over 3.2 million verified unique configurations. To address the second, we develop an aggregation framework pairing Wilson score intervals with hierarchical bootstrap, producing confidence intervals that correctly account for the nested structure of CUA benchmarks, as we empirically demonstrate. All together, we show that principled environment design and rigorous evaluation methodology are not optional refinements but prerequisites for meaningful CUA research.

Pierluca D'Oro, Sneha Silwal, William Wong +6

Meta Superintelligence Labs

Computer-Use Agents Agentic Evaluations Edge

Ask PDF

Jul 30, 2026cs.AI

How Benchmarks Mis-Score Computer-Use Agents

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Zihan Dong, Zhiyuan Ma, Zekun Wang +5

Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3

Computer-Use Agents Agentic Evaluations Independent-Pool Single-Draw Oracle

Ask PDF