The Art of Building Verifiers for Computer Use Agents

Corby Rosset; Pratyusha Sharma; Andrew Zhao; Miguel Gonzalez-Fernandez; Ahmed Awadallah

cs.CRApr 5, 2026

The Art of Building Verifiers for Computer Use Agents

Authors: Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, Ahmed Awadallah

Organizations: Microsoft Research · Browserbase

Abstract

Verifying the success of computer use agent (CUA) trajectories is a critical challenge: without reliable verification, neither evaluation nor training signal can be trusted. In this paper, we present lessons learned from building a best-in-class verifier for web tasks we call the Universal Verifier. We design the Universal Verifier around four key principles: 1) constructing rubrics with meaningful, non-overlapping criteria to reduce noise; 2) separating process and outcome rewards that yield complementary signals, capturing cases where an agent follows the right steps but gets blocked or succeeds through an unexpected path; 3) distinguishing between controllable and uncontrollable failures scored via a cascading-error-free strategy for finer-grained failure understanding; and 4) a divide-and-conquer context management scheme that attends to all screenshots in a trajectory, improving reliability on longer task horizons. We validate these findings on CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels, showing that our Universal Verifier agrees with humans as often as humans agree with each other. We report a reduction in false positive rates to near zero compared to baselines like WebVoyager ( $\geq$ 45%) and WebJudge ( $\geq$ 22%). We emphasize that these gains stem from the cumulative effect of the design choices above. We also find that an auto-research agent achieves 70% of expert quality in 5% of the time, but fails to discover all strategies required to replicate the Universal Verifier. We open-source our Universal Verifier system along with CUAVerifierBench; available at https://github.com/microsoft/fara.

Figures & tables

Figure 1: We compare whether an auto-research system can design a CUA trajectory verifier as well as the expert human-designed Universal Verifier, as measured in agreement with human labels. The human expert iterated over 32 experiments across three weeks; the auto-research agent completed the same in roughly one day. Qualitatively, auto-research edits tended to be conservative and incremental, missing the design intuition behind the human’s highest-impact structural decisions (tagged)

Verifier	LLM	Rubric	Screenshots	Action hist.	Final ans.
WebJudge (OM2W)	o4-mini	✗ Not used	✓ Top- $k$ most relevant	✓ Full	✗
			(scored 1–5, kept if
			$\geq$ threshold; capped at 5)
WebVoyager GPT eval	gpt-4o	✗ Not used	✓ All screenshots	✗	✓
			(last $N$ if over limit;
			default $N{=}30$ )

Table 1: Comparison of different computer use trajectory verifiers’ characteristics

	Internal Dataset ( $n{=}140$ )					Browserbase OM2W ( $n{=}106$ )
	WebVoy.		WebJudge		UV	WebVoy.		WebJudge		UV
	GPT-4o	GPT-5.2	o4-mini	GPT-5.2	GPT-5.2	GPT-4o	GPT-5.2	o4-mini	GPT-5.2	GPT-5.2
Agreement with outcome human labels
Accuracy ( $\uparrow$ )	$0.67$	$0.70$	$0.72$	$0.64$	${0.81}$	$0.48$	$0.74$	$0.64$	$0.74$	${0.88}$
F1 ( $\uparrow$ )	$0.73$	$0.69$	$0.74$	$0.58$	${0.81}$	$0.35$	$0.50$	$0.44$	$0.46$	${0.65}$
Cohen’s $\kappa$ ( $\uparrow$ )	$0.31$	$0.43$	$0.44$	$0.33$	$0.64$	$0.13$	$0.36$	$0.26$	$0.31$	$0.58$

Table 2: Agreement between three verifiers and humans in CUAVerifierBench. Upgrading external verifier to GPT-5.2 results in only modest improvement, confirming the UV’s advantage is architectural.

	WebVoyager		OM2W		WebTailBench
	Fara-7B	GPT-5	Fara-7B	GPT-5	Fara-7B	GPT-5
$N$ (tasks scored)	594	593	298	276	599	597
Unterminated (%)	4.2	3.4	5.0	7.2	17.0	7.7
Success rate (%)
Native verifier	74.6	90.6	32.2	62.0	39.6	62.5
UV Process	49.0	79.4	25.8	64.9	39.6	63.5

Table 3: Agreement between native benchmark verifiers and the Universal Verifier (UV) across three benchmarks and two agent models. The UV is treated as the reference label.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: An snapshot of our internal visualization tool for viewing verification results for a trajectory addressing the task “find the best men’s face wash according to GQ or Men’s Health, then buy it on Amazon”

Figure 3: A snapshot for the same example of how an individual criterion was scored, in this case, the model lost a point because it transcribed ”Cardon” incorrectly as ”Caron” in its action history based on multi-modal evidence. These kinds of meticulous analysis helps us detect hallucinations that otherwise would slip through.

Old Rubric	Pts	Improved Rubric	Pts
Task A: On Eventbrite.com, find a live music event in Nashville, TN happening this upcoming Saturday. Then on Spotify.com, find songs by any of the performing artists from that event.
Event details Name, date & time, venue & location	2/3	Event details Name, date & time, venue & location	2/3
Ticket information Includes ticket price or free indicator	0/1	Performing artists list All performers named	1/1
Event link Direct URL to Eventbrite page	0/1	Spotify artist search Searches at least one artist	1/1
Performing artists list All performers named in event description	1/1	Song selection 3–5 song titles per artist searched	1/1
Spotify artist search Searches for at least one artist on Spotify	1/1

Appendix

Table 4: Three examples of rubric failure modes of positive trajectories comparing old rubric generation (left) against the improved Universal Verifier’s (right). Red text indicates flawed rubric criteria that were e.g. never requested by the task.

Figure 4: An example of how the model made a computation error for the task “List all the members of the bands Nsync and BackStreet Boys. Find the net worth of the one with the longest last name.” – it thought “Timberlake” was the longest when in fact “Kirkpatrick” is. This mistake was identified, but notably, the error did NOT cascade to the last criterion “Find and report the net worth of the identified longest-last-name member”

Criterion	Pts
Task: How much does it cost to select a window seat on a direct AirAsia flight from Singapore to Langkawi from November 24 to November 27? If there are no available flights for those dates, please indicate that in your answer.
Access AirAsia booking flow and run the specified flight search Navigate to AirAsia, search for SIN $\rightarrow$ LGK on Nov 24 and LGK $\rightarrow$ SIN on Nov 27.	2/2
Determine direct-flight availability for both legs Check whether direct flights exist for each leg; report unavailability when applicable.	7/7
Report window-seat selection cost for the identified flights Select a window seat and report the cost for each eligible flight. Conditional: Only applies if $\geq$ 1 eligible direct AirAsia flight exists for Nov 24 (SIN $\rightarrow$ LGK) and Nov 27 (LGK $\rightarrow$ SIN). Condition met: Yes.	1/4
Total: 10/13

Appendix

Table 5: Example of a conditional rubric criterion. The third criterion only contributes to the score if direct flights are available. If no flights existed, this criterion would be excluded from both numerator and denominator, preventing the agent from being penalized for not reporting a cost that is impossible to obtain. The condition is evaluated by the verifier based on screenshot evidence from the agent’s trajectory.

Figure 5: An example of a hallucination caught by the Universal Verifier where the model claimed in its final answer that a model exhibited “+6.2% CIDEr score” when in fact it had “+2.8% in CIDEr” – and the agent did see the abstract of the model on ArXiv. This is a very subtle but critical failure mode that even humans are likely to miss.

Figure 6: An example relevance matrix where 13 screenshots were scored against five criteria in the rubric for the task “find the best men’s face wash according to GQ or Men’s Health, then buy it on Amazon”

Figure 7: An example of an unsolicited side effect that was not anticipated when the rubric was generated. The task is “Compare shipping options and delivery times for the TK Evolution APU coolant sensor between Amazon and AutoZone—make sure to check the actual product pages for the most up-to-date shipping costs and delivery estimates.” , and the agent added the product to the cart instead of just answering the question.

Scenario	Process Score	Outcome Label
Agent solved task correctly, no blockers, no side effects	Success	Success
Environment blocker (CAPTCHA, login wall, site down, out of stock); agent reported clearly and did not attempt alternative	Success (best effort)	Failure (goal not achieved)
Agent overcame blocker via alternative source, delivered correct result	Success	Success
Controllable mistake (wrong product, wrong date, missed option)	Failure (deduct per criterion)	Failure (if mistake affects goal)
Correct approach but wrong final answer (computational or reasoning error)	Failure (moderate deduction)	Failure (wrong answer)
Unsolicited side effects (extraneous cart items, unauthorized substitutions)	Failure	Failure

Appendix

Table 6: How the multimodal rubric verifier handles representative scenarios. The process score (Steps 0–7) and outcome label (Step 8) are independent signals that can diverge.

Category	Verdict	Example
Contradiction : screenshots show $X$ , agent claims $\neg X$	Failure	Screenshot shows a booking calendar exists; agent says “no booking system available”
Fabrication : agent claims $X$ with zero evidentiary basis	Failure	Agent states a price that appears nowhere in any screenshot
Omission : agent did not view everything needed; screenshots lack evidence of $X$ , but $X$ is commonly known to exist	Failure	Task: “highest ranked NHL team in Western Conference.” Agent only checked Central Division, never viewed Pacific Division
Supported inference from absence : screenshots show no evidence of $X$ across all pages, AND $X$ is not commonly known to exist	Success	No booking UI visible anywhere $\rightarrow$ agent reports “no online booking available”
Visual confirmation without explicit statement : agent omits justification but screenshots visually confirm the correct result	Success	Agent found female cardiologists but did not state “female”—photos in screenshots confirm they are female-presenting

Appendix

Table 7: Visual evidence taxonomy for evaluating agent claims against screenshot evidence. Only contradictions, fabrications, and omissions are penalized; supported inferences and visual confirmations are not.

Step	LLM Calls	Parallelism
1a: Initial Rubric Generation	1	—
1b: Dependency Checking	1	—
1c: Action-History-Only Scoring	1	—
2: Screenshot-Criteria Relevance Scoring	$M$	Fully parallel
3: Group Top-k Screenshots by Criteria	0	—
4a: Evidence Analysis (batched)	$S\leq K\times N$	Fully parallel

Appendix

Table 8: LLM calls per pipeline step. Steps marked with

\dagger

are run

N_{\text{vote}}

times when majority voting is enabled.

Benchmark	Model	Selection	Hallucination	Exec. & Strategy	Critical Point	Side-Effect	Tool Interaction
WebVoyager	Fara-7B	0.442	0.821	0.740	0.000	0.002	0.019
	GPT-5	0.206	0.424	0.382	0.000	0.002	0.040
OM2W	Fara-7B	0.724	0.905	1.456	0.007	0.007	0.046
	GPT-5	0.331	0.404	0.879	0.000	0.007	0.026
WebTailBench	Fara-7B	0.785	1.078	0.988	0.000	0.010	0.036
	GPT-5	0.485	0.495	1.054	0.000	0.020	0.047

Appendix

Table 9: Failure points normalized by number of trajectories per error category by benchmark and model.

Error Type	Description
1. Selection
1.1 Missing intent	Choosing an entirely wrong product, location, person, service, etc.
1.2 Unauthorized substitution	Silently swapping an unavailable item for a similar alternative without reporting
1.3 Wrong action type	Performing the wrong interaction on the correct entity
1.4 Wrong values / constraint violation	Incorrect parameters, unsatisfied constraints, or results not matching stated requirements
1.5 Other	Selection error not covered above

Appendix

Table 10: Error taxonomy for computer-use agent failures.

Figure 8: Fara-7B + GPT -5 side-by-side on WebVoyager. The histogram counts are normalized by number of trajectories

Figure 9: Fara-7B + GPT -5 side-by-side on Online-Mind2Web. The histogram counts are normalized by number of trajectories

Figure 10: Fara-7B + GPT -5 side-by-side on WebTailBench. The histogram counts are normalized by number of trajectories

Rubric Creation	Scoring	FNR ( $\downarrow$ )	FPR ( $\downarrow$ )	Acc ( $\uparrow$ )	F1 ( $\uparrow$ )	Cohen’s $\kappa$ ( $\uparrow$ )
GPT-4o	GPT-4o	0.16 / 0.12	0.41 / 0.36	0.78 / 0.78	0.85 / 0.82	0.42 / 0.53
o4-mini	o4-mini	0.28 / 0.25	0.24 / 0.15	0.73 / 0.79	0.80 / 0.81	0.40 / 0.59
o3	o3	0.26 / 0.20	0.21 / 0.068	0.76 / 0.86	0.82 / 0.87	0.45 / 0.72
GPT-5	GPT-5	0.17 / 0.21	0.12 / 0.051	0.84 / 0.86	0.89 / 0.87	0.63 / 0.72
GPT-5.1	GPT-5.1	0.15 / 0.15	0.29 / 0.17	0.81 / 0.84	0.87 / 0.86	0.52 / 0.68
GPT-5.2	GPT-5.2	0.23 / 0.28	0.03 / 0.00	0.82 / 0.84	0.87 / 0.84	0.61 / 0.68

Appendix

Table 11: Agreement with human labels when each model both generates its own rubric and scores it. Metrics are reported as process

\dagger

/ outcome. GPT-5.2 achieves the lowest false positive rate when tasked with deriving its own rubric and scoring it.

Rubric Creation	Scoring	FNR ( $\downarrow$ )	FPR ( $\downarrow$ )	Acc ( $\uparrow$ )	F1 ( $\uparrow$ )	Cohen’s $\kappa$ ( $\uparrow$ )
GPT-5.2	GPT-4o	0.20 / 0.14	0.32 / 0.34	0.77 / 0.78	0.84 / 0.82	0.43 / 0.54
GPT-5.2	o4-mini	0.23 / 0.25	0.21 / 0.068	0.78 / 0.83	0.84 / 0.84	0.49 / 0.66
GPT-5.2	o3	0.26 / 0.20	0.09 / 0.068	0.78 / 0.86	0.83 / 0.87	0.52 / 0.72
GPT-5.2	GPT-5	0.22 / 0.24	0.059 / 0.034	0.82 / 0.85	0.87 / 0.86	0.60 / 0.70
GPT-5.2	GPT-5.1	0.19 / 0.14	0.12 / 0.12	0.83 / 0.87	0.88 / 0.89	0.60 / 0.74
GPT-5.2	GPT-5.2	0.23 / 0.28	0.03 / 0.00	0.82 / 0.84	0.87 / 0.84	0.61 / 0.68

Appendix

Table 12: Agreement with human labels when rubrics are fixed (generated by GPT-5.2) and only the scoring model varies. Metrics are reported as process

\dagger

/ outcome. GPT-5.2 achieves the lowest FPR while GPT-5 is also competitive.

	UV-Blind	UV-Informed
Agreement with outcome human labels
Accuracy ( $\uparrow$ )	0.79	0.91
F1 ( $\uparrow$ )	0.50	0.69
Cohen’s $\kappa$ ( $\uparrow$ )	0.39	0.63
FNR ( $\downarrow$ )	0.62	0.35
FPR ( $\downarrow$ )	0.05	0.04

Appendix

Table 13: Universal Verifier’s agreement with human labels on the Browserbase-OM2W dataset (

n{=}106

) trajectories of Fara-7B on Online-Mind2Web, each labeled by two independent annotators). In the UV-Blind stage, annotators judged outcome and process success without seeing the UV’s output; in the UV-Informed stage, annotators were shown the UV’s verdict and asked whether they agreed. Outcome human labels are aggregated as majority vote; process human labels are the median rubric score binarized at

\geq 0.8

Figure 11: UV rubric score vs. human process score on the Browserbase OM2W dataset (215 annotator–task pairs). Each dot is colored by the annotator’s final UV-informed process verdict: green = process success, red = process failure. Dashed lines mark the 0.8 binarization threshold. The Pearson correlation is

r=0.61

and Spearman

\rho=0.58

	UV-Blind		UV-Informed
	% Agree	$\kappa$	% Agree	$\kappa$
Outcome (binary)	79.6	0.57	82.5	0.53
Process (binary)	72.8	0.45	72.8	0.40
Process (score $\geq$ 0.8)	68.9	0.36	—	—

Appendix

Table 14: Inter-annotator agreement on 103 Browserbase-OM2W tasks with two raters. Process labels show consistently lower agreement than outcome labels, reflecting the greater subjectivity of process evaluation. The continuous process scores have Pearson

r=0.62

and MAE

=0.21

	Internal Dataset ( $n{=}140$ )			Browserbase OM2W ( $n{=}106$ )
	WebVoy.	WebJudge	UV	WebVoy.	WebJudge	UV
	(GPT-4o)	(o4-mini)	(GPT-5.2)	(GPT-4o)	(o4-mini)	(GPT-5.2)
Agreement with outcome human labels
Accuracy ( $\uparrow$ )	$0.67\pm 0.01$	$0.72\pm 0.01$	$0.81\pm 0.02$	$0.48\pm 0.01$	$0.64\pm 0.02$	${0.88\pm 0.00}$
F1 ( $\uparrow$ )	$0.73\pm 0.01$	$0.74\pm 0.00$	$0.81\pm 0.02$	$0.35\pm 0.00$	$0.44\pm 0.02$	${0.65\pm 0.03}$
Cohen’s $\kappa$ ( $\uparrow$ )	$0.31\pm 0.01$	$0.44\pm 0.01$	$0.64\pm 0.03$	$0.13\pm 0.01$	$0.26\pm 0.03$	$0.58\pm 0.04$

Appendix

Table 15: Agreement between three verifiers and human labels on two datasets. All values are mean

\pm

std over 3 independent runs. Internal Dataset are internally annotated trajectories while Browserbase OM2W are Fara-7B trajectories from Online-Mind2Web with UV-informed human labels aggregated across two annotators per task by majority vote (outcome) or median rubric score binarized at

\geq 0.8

(process); see § 5 . For the UV, outcome uses the binary outcome signal and process uses the rubric score binarized at

\geq 0.8

. WebVoyager and WebJudge each produce a single binary prediction compared against both label types.

	Success	Fail	Total	Success Rate
Over-budget (truncated)	44	663	707	6.2%
Terminated (agent stopped)	312	283	595	52.4%

Appendix

Table 16: Partitions AgentRewardBench’s 1302 human annotated trajectories based on its relation to the step budget and human-annotated success

Figure 12: An example false positive from AgentRewardBench. The task is ” Navigate to the item on this page whose image is a desktop screenshot ”. Although this is a screenshot, the spring mattress screenshot is a mobile screenshot, not a desktop screenshot

Figure 13: Process Cohen’s

\kappa

agreement with human labels across successive verifier design iterations. Compare with the outcome

\kappa

in Figure 1 . Process agreement is consistently lower than outcome agreement for all three settings, reflecting the greater subjectivity of process evaluation.

Figure 14: Outcome false positive rate (FPR) and false negative rate (FNR) across successive design iterations. See Figure 1 for the corresponding Cohen’s

\kappa

Figure 15: Process false positive rate (FPR) and false negative rate (FNR) across successive design iterations. See Figure 13 for the corresponding Cohen’s

\kappa

Run	Purpose	Decision
0	Baseline	BASELINE
1	Outcome verification fixes	ROLLED BACK (process FPR 8.82%)
2	Semantic precision + entity non-existence + nitpick calibration	COMMITTED
3	Variant/tier + binding examples + CP rule	ROLLED BACK (process FPR 5.88%)
4	Similar to run 3, different approach	ROLLED BACK (process FPR 5.88%)
5	Binding example matching	ROLLED BACK (outcome FPs=2)

Appendix

Table 17: Summary of auto-research agent iterations continuing from the human expert’s best verifier. Each run represents a single prompt modification cycle. Green rows were committed (improved

\kappa

without increasing FPR), red rows were rolled back, and yellow indicates a stochastic baseline check.

Run	Change Type	What the Agent Did	Why It Helped
2	Nitpick calibration (prompt)	Added explicit test: “Would a reasonable user say this output is useful?” Enumerated always-nitpick scenarios (approximate walk times, price tier symbols, common knowledge inferences).	Fixed 10+ false negatives where minor issues were treated as critical failures.
2	Semantic precision in rubric generation (prompt)	Added rule: criteria must test the exact concept the task asks about, not a related one. E.g., “how many people work remotely” $\neq$ “how many remote job postings.”	Fixed false positives from rubrics testing the wrong quantity.
6	Rubric score context (code)	Computed normalized rubric score and appended calibration guidance to the outcome prompt. If rubric $\geq 95\%$ , verifier must identify a specific critical issue to override.	Most impactful single change: provided quantitative signal instead of adding more text to an already-long prompt.
10	Critical point output rule (prompt)	When screenshots confirm the agent reached a transaction boundary (checkout, passenger info page) with correct selections, a brief output message is a nitpick, not grounds for failure.	Fixed persistent false negatives on booking/flight tasks where the agent correctly stopped but didn’t restate details.
11	Forced rule checking (prompt)	Expanded the mandatory example_match_check JSON field to require the LLM to also check named rules (Entity Non-Existence, Multi-Item Cart, Critical Point Output, etc.) before making its verdict.	Mitigated the “rules exist but aren’t applied” problem in ${\sim}1{,}800$ -line prompts.

Appendix

Table 18: Representative prompt and code changes made by the auto-research agent across its iterations. Changes span prompt engineering (calibration rules, forced structured checking) and code modifications (injecting rubric scores as quantitative context).

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 145 papers

How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
OpenComputer: Verifiable Software Worlds for Computer-Use Agents
May 19, 2026Jinbiao Wei, Qianran Ma, Yilun Zhao +4Computer-Use Agent Benchmarks Computer-Use Agents
Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill
VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards
Oct 19, 2026Zhizhou Wang, Yaxi Lu, Zhong Zhang +4AI Agent Reliability RL for GUI Agents
Southwest Jiaotong University · Tsinghua University · University of Electronic Science and Technology of China +3

Jul 30, 2026J/K move · Enter open · S save

How Benchmarks Mis-Score Computer-Use Agents

Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li+4

Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center+3

Agent Failure Analysis Computer-Use Agent Benchmarks Benchmark Auditing Computer-Use Agents Benchmark Validity Automated Evaluation+1

Open paper AskPDF arXiv

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Abstract

LLM Agent Verification

Explore similar work

How Benchmarks Mis-Score Computer-Use Agents

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards