ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

Tianchen Guan; Xinlei Lin; Royce Cheng-Yue; Xiangjun Wang; Shuyan Zhou

cs.AIAug 18, 2026

ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

Authors: Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou

Organizations: Duke University · Amazon AGI SF Lab

Abstract

Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.

Figures & tables

Figure 1: ComponentBench evaluates computer-use agents on 97 canonical UI component types organized into 14 families. Left : Tasks span diverse interaction types implemented across Ant Design, MUI, and Mantine. Right : The ontology covers the breadth of modern web UI interaction.

Figure 2: A concrete ComponentBench task ( data_table_filterable-mantine-T10 ). (a) The agent sees a rendered page containing three visually similar mini-tables. (b) The instruction, programmatic verifier, and structured metadata. The agent must disambiguate the correct table instance (Invoices), apply two filters, and commit via the instance-local Apply button.

Model	Browser-Use	AX-tree	SoM	Pixel
Gemini 3 Flash	95.2	89.6	87.1	85.4
GPT-5.4	90.4	81.5	77.0	83.8
Gemini 3.1 Flash-Lite	87.4	77.7	73.5	63.3
GPT-5 mini	87.0	83.1	78.5	48.9
GPT-5.4 mini	85.8	79.1	74.7	77.1
Qwen3-VL-235B ‡	78.8	77.0	54.4	50.5

Table 1: ComponentBench-Full pass rates (%) by model and observation/action space. All runs use a 20-step budget. Browser-Use is a tool-rich reference regime with DOM access; AX-tree, SoM, and Pixel share the same harness and differ only in observation and action space. Task-resampling bootstrap 95% CIs (

B{=}10{,}000

) have half-width

\leq

1.9% per cell. † Native pixel-only harness with coordinate prediction. ‡ Browser-Use served via Amazon Bedrock (precision unspecified); other Qwen cells use local vLLM FP8 (Appendix M ).

	Browser-Use			AX-tree			SoM			Pixel
Model	Pass	$\leq H$	$\leq$ 2 $H$	Pass	$\leq H$	$\leq$ 2 $H$	Pass	$\leq H$	$\leq$ 2 $H$	Pass	$\leq H$	$\leq$ 2 $H$
Gemini 3 Flash	95.2	56.1	85.3	89.6	74.7	82.5	87.1	73.2	81.4	85.4	65.0	77.8
GPT-5.4	90.4	40.6	75.3	81.5	56.5	72.4	77.0	49.6	68.2	83.8	48.1	69.5
Gemini 3.1 FL	87.4	50.2	77.3	77.7	68.6	73.9	73.5	64.8	70.9	63.3	47.3	56.2
GPT-5 mini	87.0	51.7	78.5	83.1	66.6	76.5	78.5	63.2	73.4	48.9	19.0	28.2
GPT-5.4 mini	85.8	51.0	77.9	79.1	69.0	73.8	74.7	64.0	70.6	77.1	60.2	68.9

Table 2: Step efficiency (% of 2,910 tasks).

\leq H

\leq

H

: solved within 1/2

\times

human steps, where

H=\max(\text{human steps},1)

; hover-only references count as

H{=}1

. Browser-Use “steps” are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. † Native pixel-only harness. Full

\leq

H

H

breakdown in Appendix C .

Family	BU	AX	SoM	Pix	Avg
Drag/Drop & Workspace	29.7	72.1	31.8	57.4	47.7
Continuous Precision	74.0	53.8	53.9	56.2	59.4
Advanced Editors	83.6	67.8	49.7	46.4	61.9
Date & Time	83.4	76.2	65.8	59.4	71.2
Disclosure & Progressive	80.1	76.0	64.1	68.3	72.1
Discrete Choice	97.2	85.7	83.3	79.0	86.3

Table 3: Pass rate (%) by family and mode, averaged across all six models (excluding UI-TARS). Top: five hardest families; bottom: three easiest. Full breakdown in Figure 3 (Appendix).

Failure category	Deterministic	Layer-2
	(5 models, $n$ =8,864)	(1 model, $n$ =2,752)
Continuous calibration error	20.2%	—
Transient state loss	19.9%	15.0%
Missing commit or confirmation	11.6%	20.0%
Target acquisition / wrong instance	11.2%	31.6%
Repetition / no-progress loop	11.2%	5.6%

Table 4: Failure taxonomy over all failed BrowserGym-mode traces, under two complementary labelings: a deterministic trace-feature pass over five models, and Layer-2 LLM diagnostic labels for Gemini 3.1 Flash-Lite mapped onto the same categories. The two labelings surface overlapping high-frequency mechanism families, though their percentages are not directly comparable (different model sets and schemas). Continuous-calibration and drag-execution failures are not represented as separate categories in the original Layer-2 schema, hence absent from that column.

Model	Mode	Pass	$\leq H$	$\leq 2H$	$\leq 3H$
Gemini 3 Flash	Browser-Use	84.5	51.5	71.5	78.5
Gemini 3 Flash	Pixel	60.9	30.5	51.0	56.0
GPT-5.4 mini	Browser-Use	57.8	36.8	51.2	55.3
GPT-5.4 mini	Pixel	37.7	22.0	32.1	34.1
Opus 4.6	Pixel	65.4	34.1	53.8	59.4

Table 5: ComponentBench-Core results (% of 912 tasks). Same metrics (and Browser-Use turn-level caveat) as Table 2 , on the hard-only Core suite. Pass rates drop 10–39% from Full , confirming that Core concentrates diagnostic mass on unresolved interaction families. Opus 4.6 is evaluated only on Core Pixel.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Levels (count)	Coverage
Canonical type	97 types $\times$ 30 tasks	100%
Family	14 families (120–300 tasks each)	100%
Library	antd 1,000 / mui 910 / mantine 970 / external 30	4
Task template	24 templates; most-used match_reference (351),	24/24
	least-used file_manage (10)
Difficulty bucket	easy 972 / mid 985 / hard 953	3/3

Appendix

Table 6: Realized diversity of the 2,910 ComponentBench-Full tasks over libraries, templates, difficulty, and the eight controlled scene factors. The 30 external tasks are the markdown-editor tasks, which use the third-party @uiw/react-md-editor because none of the three primary libraries ships a core markdown editor. One additional ad-hoc template variant ( replace_code ) is used by a single OTP-input task. The realized difficulty buckets deviate slightly from the prompted 970/970/970 split because two later-added types (breadcrumb, pagination) were generated by a run that did not enforce the per-type balance quota.

Model	Mode	Pass	$\leq H$	$\leq 2H$	$\leq 3H$	$\leq 5H$
Gemini 3 Flash	Browser-Use	95.2	56.1	85.3	89.1	91.9
	AX-tree	89.6	74.7	82.5	84.7	87.4
	SoM	87.1	73.2	81.4	83.6	85.2
	Pixel	85.4	65.0	77.8	80.9	83.6
GPT-5.4	Browser-Use	90.4	40.6	75.3	82.9	86.8
	AX-tree	81.5	56.5	72.4	76.3	79.4

Appendix

Table 7: Full step efficiency on ComponentBench-Full (% of 2,910 tasks). Pass : solved within 20 steps.

\leq H

\leq 2H

\leq 3H

\leq 5H

: solved within 1/2/3/5

\times

the human step count. Bold : best per model. Browser-Use steps are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. † Native pixel-only interface.

Model	Mode	Human	Agent	Ratio	Med. Agent
Gemini 3 Flash	AX-tree	4.5s	21.5s	4.7 $\times$	13.5s
	SoM	4.6s	22.0s	4.8 $\times$	14.4s
	Pixel	4.3s	27.6s	6.4 $\times$	17.1s
	Browser-Use	4.6s	32.6s	7.1 $\times$	21.2s
Gemini 3.1 Flash-Lite	AX-tree	4.1s	14.5s	3.6 $\times$	11.2s
	SoM	3.8s	14.4s	3.8 $\times$	11.6s

Appendix

Table 8: Full time efficiency on successful ComponentBench-Full tasks. Human and Agent columns report mean wall-clock duration over the same task set: the tasks each model–mode configuration solved (hence the Human column varies by row). Ratio is agent-to-human mean time.

Model	SoM	Pixel	$\Delta$ (SoM $-$ Pixel)
GPT-5 mini	78.5	48.9	$+$ 29.6
Gemini 3.1 Flash-Lite	73.5	63.3	$+$ 10.2
Qwen3-VL-235B	54.4	50.5	$+$ 3.9
Gemini 3 Flash	87.1	85.4	$+$ 1.7
GPT-5.4 mini	74.7	77.1	$-$ 2.4
GPT-5.4	77.0	83.8	$-$ 6.8

Appendix

Table 9: SoM–Pixel delta on ComponentBench-Full (%). The SoM advantage is model-dependent, ranging from

+

29.6% to

-

6.8%.

Model	Browser-Use	Mean(AX,SoM,Pix)	$\Delta$
Qwen3-VL-235B	78.8	60.6	$+$ 18.2
GPT-5 mini	87.0	70.2	$+$ 16.8
Gemini 3.1 Flash-Lite	87.4	71.5	$+$ 15.9
GPT-5.4	90.4	80.8	$+$ 9.6
GPT-5.4 mini	85.8	77.0	$+$ 8.8
Gemini 3 Flash	95.2	87.4	$+$ 7.8

Appendix

Table 10: Browser-Use advantage over mean non-Browser-Use pass rate (%). The advantage is largest for weaker models.

Figure 3: Pass rate (%) by component family, model, and observation mode on ComponentBench-Full. Families sorted by difficulty (hardest at top). The heatmap reveals where mode ordering inverts: Drag/Drop favors Pixel over Browser-Use; Advanced Editors strongly favor Browser-Use.

Figure 4: Human-agent difficulty inversion. Each point is one canonical type. The lower-left quadrant contains components trivial for humans (

\leq

2 steps) but hard for agents (

<

60% pass).

Figure 5: Clutter and spacing disproportionately hurt Pixel mode, widening the AX-tree–Pixel gap from 10% to 24% (clutter) and 12% to 21% (spacing).

Figure 6: GPT-5.4 mini SoM vs. Pixel task outcomes by component family. For each family, tasks are classified as both-pass, SoM-only, Pixel-only, or both-fail. Drag/Drop and Continuous Precision families show the strongest Pixel-over-SoM advantage.

Figure 7: Browser-Use (CUA) advantage by component family, averaged across the six models evaluated in all four regimes. The advantage ranges from

+

29% (Advanced Editors) to

-

24% (Drag/Drop), confirming that DOM-level tool access is not uniformly beneficial.

Figure 8: Pass rate by task template and observation mode. Templates requiring spatial control (set_range, drag_operation) or complex editing (editor_operation) are substantially harder than simple activation or disclosure tasks.

Figure 9: Efficiency frontier: pass rate vs. mean time per successful task. Each point is one model-mode combination among the six models evaluated in the shared and Browser-Use regimes; the native UI-TARS configuration is omitted. The human reference point (100% pass, 4.8s) is shown for comparison.

Figure 10: Marginal tasks solved per additional step (Gemini 3 Flash). Most diagnostic value is concentrated in the first 5 steps; steps 6–20 contribute diminishing returns.

Difficulty axis	Overall	AX-tree	Pixel	Browser-Use
Precision requirement	$+$ 0.44	$+$ 0.30	$+$ 0.41	$+$ 0.33
Target acquisition	$+$ 0.31	$+$ 0.17	$+$ 0.36	$+$ 0.19
Density / choice interf.	$+$ 0.23	$+$ 0.10	$+$ 0.31	$+$ 0.12
Feedback dynamics	$+$ 0.23	$+$ 0.12	$+$ 0.29	$+$ 0.13
Depth / layering	$+$ 0.20	$+$ 0.11	$+$ 0.29	$+$ 0.05
Semantic observability	$+$ 0.13	$+$ 0.08	$+$ 0.17	$+$ 0.05

Appendix

Table 11: Pearson correlation between intended difficulty axes and agent failure rate. Precision requirement is the most predictive axis overall; all axes predict Pixel failure more strongly than Browser-Use failure.

Tier	AX-tree	SoM	Pixel	Browser-Use
L0 (easy)	87.7%	85.9%	83.0%	92.8%
L1 (medium)	82.0%	75.1%	69.8%	87.6%
L2 (hard)	77.4%	66.4%	57.6%	84.3%
L3 (hard+)	71.4%	59.4%	49.5%	80.4%

Appendix

Table 12: Pass rate by intended difficulty tier and observation mode, averaged across all six models (excluding UI-TARS). The AX-tree–Pixel gap widens monotonically from 4.6% (L0) to 21.9% (L3).

Component type	Agent %	Human steps	Family
resizable_columns	24.4	1.7	Drag/Drop
window_splitter	38.3	1.3	Disclosure
slider_range	39.9	1.9	Continuous
rich_text_editor	40.7	4.9	Adv. Editors
meter	46.5	1.6	Continuous
datetime_picker_range	48.9	10.5	Date/Time

Appendix

Table 13: The 15 hardest canonical types by mean agent pass rate (averaged across all models and modes, excluding UI-TARS). Human steps column shows the mean normalized human reference steps. Components with

\leq

2 human steps but

<

60% agent pass rate represent the human-agent difficulty inversion.

Metric		Value
Mean normalized steps (Original / A1 / A2)		2.92 / 2.90 / 3.14
Median normalized steps (all annotators)		2
Pairwise step-count Pearson correlation		0.79–0.94
Exact step-count agreement		71%–84%
Mean absolute step-count difference		0.32–0.64

Appendix

Table 14: Agreement between the original reference annotator and two additional annotators (A1, A2) on the 278-task validation subset (action tasks only). Human-to-human variation is small at the aggregate level.

Model	Mode	Mean pass (%)	Max dev	Agree%	Jaccard	95% CI
Gemini 3 Flash	Browser-Use	95.1	1.1	97.5	0.974	[92.6, 97.3]
Gemini 3 Flash	Pixel	87.4	0.7	90.6	0.898	[84.0, 90.8]
Gemini 3.1 Flash-Lite	Browser-Use	87.1	0.7	94.2	0.936	[83.3, 90.6]
Gemini 3.1 Flash-Lite	Pixel	77.0	0.0	86.3	0.837	[72.5, 81.3]
GPT-5 mini	Browser-Use	86.7	1.4	92.1	0.913	[82.9, 90.3]
GPT-5 mini	Pixel	52.5	0.0	82.7	0.718	[47.1, 57.9]

Appendix

Table 15: Repeated-run stability on the 278-task subset (2 runs per cell, same harness commit and endpoint). Max dev: run-to-run pass-rate deviation (%). Agree%: task-level exact pass/fail agreement. Jaccard: overlap of passed-task sets. CI: bootstrap 95% interval on the mean pass rate.

Computer-Use Agent Benchmarks

Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.

See all 120 papers

Jun 8, 2026

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.

Wanli Li, Bowen Zhou, Yunyao Yu +4

Microsoft Research Asia · Tsinghua University

Agent Benchmarks Computer-Use Agents AI Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks Computer-Use Agent Benchmarks+2

Ask PDF

Jul 30, 2026

How Benchmarks Mis-Score Computer-Use Agents

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Zihan Dong, Zhiyuan Ma, Zekun Wang +5

Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3

Agent Failure Analysis Computer-Use Agent Benchmarks Benchmark Auditing Computer-Use Agents Benchmark Validity Automated Evaluation+1

Ask PDF

May 12, 2026

Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git

Miaosen Zhang, Xiaohan Zhao, Zhihong Tan +14

1Southeast University · 2Mohamed bin Zayed University of Artificial Intelligence · 3Wuhan University +2

Computer-Use Agent Benchmarks Computer-Use Agents Synthetic Data Generation AI Agent Evaluation

Ask PDF

Abstract

Computer-Use Agent Benchmarks

Explore similar work

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

How Benchmarks Mis-Score Computer-Use Agents

Covering Human Action Space for Computer Use: Data Synthesis and Benchmark