Verifying the success of computer use agent (CUA) trajectories is a critical challenge: without reliable verification, neither evaluation nor training signal can be trusted. In this paper, we present lessons learned from building a best-in-class verifier for web tasks we call the Universal Verifier. We design the Universal Verifier around four key principles: 1) constructing rubrics with meaningful, non-overlapping criteria to reduce noise; 2) separating process and outcome rewards that yield complementary signals, capturing cases where an agent follows the right steps but gets blocked or succeeds through an unexpected path; 3) distinguishing between controllable and uncontrollable failures scored via a cascading-error-free strategy for finer-grained failure understanding; and 4) a divide-and-conquer context management scheme that attends to all screenshots in a trajectory, improving reliability on longer task horizons. We validate these findings on CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels, showing that our Universal Verifier agrees with humans as often as humans agree with each other. We report a reduction in false positive rates to near zero compared to baselines like WebVoyager (≥ 45%) and WebJudge (≥ 22%). We emphasize that these gains stem from the cumulative effect of the design choices above. We also find that an auto-research agent achieves 70% of expert quality in 5% of the time, but fails to discover all strategies required to replicate the Universal Verifier. We open-source our Universal Verifier system along with CUAVerifierBench; available at https://github.com/microsoft/fara.
Figures & tables
Figure 1: We compare whether an auto-research system can design a CUA trajectory verifier as well as the expert human-designed Universal Verifier, as measured in agreement with human labels. The human expert iterated over 32 experiments across three weeks; the auto-research agent completed the same in roughly one day. Qualitatively, auto-research edits tended to be conservative and incremental, missing the design intuition behind the human’s highest-impact structural decisions (tagged)
Verifier
LLM
Rubric
Screenshots
Action hist.
Final ans.
WebJudge (OM2W)
o4-mini
✗ Not used
✓ Top- k most relevant
✓ Full
✗
(scored 1–5, kept if
≥ threshold; capped at 5)
WebVoyager GPT eval
gpt-4o
✗ Not used
✓ All screenshots
✗
✓
(last N if over limit;
default N=30 )
Table 1: Comparison of different computer use trajectory verifiers’ characteristics
Internal Dataset ( n=140 )
Browserbase OM2W ( n=106 )
WebVoy.
WebJudge
UV
WebVoy.
WebJudge
UV
GPT-4o
GPT-5.2
o4-mini
GPT-5.2
GPT-5.2
GPT-4o
GPT-5.2
o4-mini
GPT-5.2
GPT-5.2
Agreement with outcome human labels
Accuracy ( ↑ )
0.67
0.70
0.72
0.64
0.81
0.48
0.74
0.64
0.74
0.88
F1 ( ↑ )
0.73
0.69
0.74
0.58
0.81
0.35
0.50
0.44
0.46
0.65
Cohen’s κ ( ↑ )
0.31
0.43
0.44
0.33
0.64
0.13
0.36
0.26
0.31
0.58
Table 2: Agreement between three verifiers and humans in CUAVerifierBench. Upgrading external verifier to GPT-5.2 results in only modest improvement, confirming the UV’s advantage is architectural.
WebVoyager
OM2W
WebTailBench
Fara-7B
GPT-5
Fara-7B
GPT-5
Fara-7B
GPT-5
N (tasks scored)
594
593
298
276
599
597
Unterminated (%)
4.2
3.4
5.0
7.2
17.0
7.7
Success rate (%)
Native verifier
74.6
90.6
32.2
62.0
39.6
62.5
UV Process
49.0
79.4
25.8
64.9
39.6
63.5
Table 3: Agreement between native benchmark verifiers and the Universal Verifier (UV) across three benchmarks and two agent models. The UV is treated as the reference label.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: An snapshot of our internal visualization tool for viewing verification results for a trajectory addressing the task “find the best men’s face wash according to GQ or Men’s Health, then buy it on Amazon”
Figure 3: A snapshot for the same example of how an individual criterion was scored, in this case, the model lost a point because it transcribed ”Cardon” incorrectly as ”Caron” in its action history based on multi-modal evidence. These kinds of meticulous analysis helps us detect hallucinations that otherwise would slip through.
Old Rubric
Pts
Improved Rubric
Pts
Task A: On Eventbrite.com, find a live music event in Nashville, TN happening this upcoming Saturday. Then on Spotify.com, find songs by any of the performing artists from that event.
Event details Name, date & time, venue & location
2/3
Event details Name, date & time, venue & location
2/3
Ticket information Includes ticket price or free indicator
0/1
Performing artists list All performers named
1/1
Event link Direct URL to Eventbrite page
0/1
Spotify artist search Searches at least one artist
1/1
Performing artists list All performers named in event description
1/1
Song selection 3–5 song titles per artist searched
1/1
Spotify artist search Searches for at least one artist on Spotify
1/1
Appendix
Table 4: Three examples of rubric failure modes of positive trajectories comparing old rubric generation (left) against the improved Universal Verifier’s (right). Red text indicates flawed rubric criteria that were e.g. never requested by the task.
Figure 4: An example of how the model made a computation error for the task “List all the members of the bands Nsync and BackStreet Boys. Find the net worth of the one with the longest last name.” – it thought “Timberlake” was the longest when in fact “Kirkpatrick” is. This mistake was identified, but notably, the error did NOT cascade to the last criterion “Find and report the net worth of the identified longest-last-name member”
Criterion
Pts
Task: How much does it cost to select a window seat on a direct AirAsia flight from Singapore to Langkawi from November 24 to November 27? If there are no available flights for those dates, please indicate that in your answer.
Access AirAsia booking flow and run the specified flight search Navigate to AirAsia, search for SIN → LGK on Nov 24 and LGK → SIN on Nov 27.
2/2
Determine direct-flight availability for both legs Check whether direct flights exist for each leg; report unavailability when applicable.
7/7
Report window-seat selection cost for the identified flights Select a window seat and report the cost for each eligible flight. Conditional: Only applies if ≥ 1 eligible direct AirAsia flight exists for Nov 24 (SIN → LGK) and Nov 27 (LGK → SIN). Condition met: Yes.
1/4
Total: 10/13
Appendix
Table 5: Example of a conditional rubric criterion. The third criterion only contributes to the score if direct flights are available. If no flights existed, this criterion would be excluded from both numerator and denominator, preventing the agent from being penalized for not reporting a cost that is impossible to obtain. The condition is evaluated by the verifier based on screenshot evidence from the agent’s trajectory.
Figure 5: An example of a hallucination caught by the Universal Verifier where the model claimed in its final answer that a model exhibited “+6.2% CIDEr score” when in fact it had “+2.8% in CIDEr” – and the agent did see the abstract of the model on ArXiv. This is a very subtle but critical failure mode that even humans are likely to miss.
Figure 6: An example relevance matrix where 13 screenshots were scored against five criteria in the rubric for the task “find the best men’s face wash according to GQ or Men’s Health, then buy it on Amazon”
Figure 7: An example of an unsolicited side effect that was not anticipated when the rubric was generated. The task is “Compare shipping options and delivery times for the TK Evolution APU coolant sensor between Amazon and AutoZone—make sure to check the actual product pages for the most up-to-date shipping costs and delivery estimates.” , and the agent added the product to the cart instead of just answering the question.
Scenario
Process Score
Outcome Label
Agent solved task correctly, no blockers, no side effects
Success
Success
Environment blocker (CAPTCHA, login wall, site down, out of stock); agent reported clearly and did not attempt alternative
Success (best effort)
Failure (goal not achieved)
Agent overcame blocker via alternative source, delivered correct result
Correct approach but wrong final answer (computational or reasoning error)
Failure (moderate deduction)
Failure (wrong answer)
Unsolicited side effects (extraneous cart items, unauthorized substitutions)
Failure
Failure
Appendix
Table 6: How the multimodal rubric verifier handles representative scenarios. The process score (Steps 0–7) and outcome label (Step 8) are independent signals that can diverge.
Category
Verdict
Example
Contradiction : screenshots show X , agent claims ¬X
Failure
Screenshot shows a booking calendar exists; agent says “no booking system available”
Fabrication : agent claims X with zero evidentiary basis
Failure
Agent states a price that appears nowhere in any screenshot
Omission : agent did not view everything needed; screenshots lack evidence of X , but X is commonly known to exist
Failure
Task: “highest ranked NHL team in Western Conference.” Agent only checked Central Division, never viewed Pacific Division
Supported inference from absence : screenshots show no evidence of X across all pages, AND X is not commonly known to exist
Visual confirmation without explicit statement : agent omits justification but screenshots visually confirm the correct result
Success
Agent found female cardiologists but did not state “female”—photos in screenshots confirm they are female-presenting
Appendix
Table 7: Visual evidence taxonomy for evaluating agent claims against screenshot evidence. Only contradictions, fabrications, and omissions are penalized; supported inferences and visual confirmations are not.
Step
LLM Calls
Parallelism
1a: Initial Rubric Generation
1
—
1b: Dependency Checking
1
—
1c: Action-History-Only Scoring
1
—
2: Screenshot-Criteria Relevance Scoring
M
Fully parallel
3: Group Top-k Screenshots by Criteria
0
—
4a: Evidence Analysis (batched)
S≤K×N
Fully parallel
Appendix
Table 8: LLM calls per pipeline step. Steps marked with † are run Nvote times when majority voting is enabled.
Benchmark
Model
Selection
Hallucination
Exec. & Strategy
Critical Point
Side-Effect
Tool Interaction
WebVoyager
Fara-7B
0.442
0.821
0.740
0.000
0.002
0.019
GPT-5
0.206
0.424
0.382
0.000
0.002
0.040
OM2W
Fara-7B
0.724
0.905
1.456
0.007
0.007
0.046
GPT-5
0.331
0.404
0.879
0.000
0.007
0.026
WebTailBench
Fara-7B
0.785
1.078
0.988
0.000
0.010
0.036
GPT-5
0.485
0.495
1.054
0.000
0.020
0.047
Appendix
Table 9: Failure points normalized by number of trajectories per error category by benchmark and model.
Error Type
Description
1. Selection
1.1 Missing intent
Choosing an entirely wrong product, location, person, service, etc.
1.2 Unauthorized substitution
Silently swapping an unavailable item for a similar alternative without reporting
1.3 Wrong action type
Performing the wrong interaction on the correct entity
1.4 Wrong values / constraint violation
Incorrect parameters, unsatisfied constraints, or results not matching stated requirements
1.5 Other
Selection error not covered above
Appendix
Table 10: Error taxonomy for computer-use agent failures.
Figure 8: Fara-7B + GPT -5 side-by-side on WebVoyager. The histogram counts are normalized by number of trajectories
Figure 9: Fara-7B + GPT -5 side-by-side on Online-Mind2Web. The histogram counts are normalized by number of trajectories
Figure 10: Fara-7B + GPT -5 side-by-side on WebTailBench. The histogram counts are normalized by number of trajectories
Rubric Creation
Scoring
FNR ( ↓ )
FPR ( ↓ )
Acc ( ↑ )
F1 ( ↑ )
Cohen’s κ ( ↑ )
GPT-4o
GPT-4o
0.16 / 0.12
0.41 / 0.36
0.78 / 0.78
0.85 / 0.82
0.42 / 0.53
o4-mini
o4-mini
0.28 / 0.25
0.24 / 0.15
0.73 / 0.79
0.80 / 0.81
0.40 / 0.59
o3
o3
0.26 / 0.20
0.21 / 0.068
0.76 / 0.86
0.82 / 0.87
0.45 / 0.72
GPT-5
GPT-5
0.17 / 0.21
0.12 / 0.051
0.84 / 0.86
0.89 / 0.87
0.63 / 0.72
GPT-5.1
GPT-5.1
0.15 / 0.15
0.29 / 0.17
0.81 / 0.84
0.87 / 0.86
0.52 / 0.68
GPT-5.2
GPT-5.2
0.23 / 0.28
0.03 / 0.00
0.82 / 0.84
0.87 / 0.84
0.61 / 0.68
Appendix
Table 11: Agreement with human labels when each model both generates its own rubric and scores it. Metrics are reported as process † / outcome. GPT-5.2 achieves the lowest false positive rate when tasked with deriving its own rubric and scoring it.
Rubric Creation
Scoring
FNR ( ↓ )
FPR ( ↓ )
Acc ( ↑ )
F1 ( ↑ )
Cohen’s κ ( ↑ )
GPT-5.2
GPT-4o
0.20 / 0.14
0.32 / 0.34
0.77 / 0.78
0.84 / 0.82
0.43 / 0.54
GPT-5.2
o4-mini
0.23 / 0.25
0.21 / 0.068
0.78 / 0.83
0.84 / 0.84
0.49 / 0.66
GPT-5.2
o3
0.26 / 0.20
0.09 / 0.068
0.78 / 0.86
0.83 / 0.87
0.52 / 0.72
GPT-5.2
GPT-5
0.22 / 0.24
0.059 / 0.034
0.82 / 0.85
0.87 / 0.86
0.60 / 0.70
GPT-5.2
GPT-5.1
0.19 / 0.14
0.12 / 0.12
0.83 / 0.87
0.88 / 0.89
0.60 / 0.74
GPT-5.2
GPT-5.2
0.23 / 0.28
0.03 / 0.00
0.82 / 0.84
0.87 / 0.84
0.61 / 0.68
Appendix
Table 12: Agreement with human labels when rubrics are fixed (generated by GPT-5.2) and only the scoring model varies. Metrics are reported as process † / outcome. GPT-5.2 achieves the lowest FPR while GPT-5 is also competitive.
UV-Blind
UV-Informed
Agreement with outcome human labels
Accuracy ( ↑ )
0.79
0.91
F1 ( ↑ )
0.50
0.69
Cohen’s κ ( ↑ )
0.39
0.63
FNR ( ↓ )
0.62
0.35
FPR ( ↓ )
0.05
0.04
Appendix
Table 13: Universal Verifier’s agreement with human labels on the Browserbase-OM2W dataset ( n=106 ) trajectories of Fara-7B on Online-Mind2Web, each labeled by two independent annotators). In the UV-Blind stage, annotators judged outcome and process success without seeing the UV’s output; in the UV-Informed stage, annotators were shown the UV’s verdict and asked whether they agreed. Outcome human labels are aggregated as majority vote; process human labels are the median rubric score binarized at ≥0.8 .
Figure 11: UV rubric score vs. human process score on the Browserbase OM2W dataset (215 annotator–task pairs). Each dot is colored by the annotator’s final UV-informed process verdict: green = process success, red = process failure. Dashed lines mark the 0.8 binarization threshold. The Pearson correlation is r=0.61 and Spearman ρ=0.58 .
UV-Blind
UV-Informed
% Agree
κ
% Agree
κ
Outcome (binary)
79.6
0.57
82.5
0.53
Process (binary)
72.8
0.45
72.8
0.40
Process (score ≥ 0.8)
68.9
0.36
—
—
Appendix
Table 14: Inter-annotator agreement on 103 Browserbase-OM2W tasks with two raters. Process labels show consistently lower agreement than outcome labels, reflecting the greater subjectivity of process evaluation. The continuous process scores have Pearson r=0.62 and MAE =0.21 .
Internal Dataset ( n=140 )
Browserbase OM2W ( n=106 )
WebVoy.
WebJudge
UV
WebVoy.
WebJudge
UV
(GPT-4o)
(o4-mini)
(GPT-5.2)
(GPT-4o)
(o4-mini)
(GPT-5.2)
Agreement with outcome human labels
Accuracy ( ↑ )
0.67±0.01
0.72±0.01
0.81±0.02
0.48±0.01
0.64±0.02
0.88±0.00
F1 ( ↑ )
0.73±0.01
0.74±0.00
0.81±0.02
0.35±0.00
0.44±0.02
0.65±0.03
Cohen’s κ ( ↑ )
0.31±0.01
0.44±0.01
0.64±0.03
0.13±0.01
0.26±0.03
0.58±0.04
Appendix
Table 15: Agreement between three verifiers and human labels on two datasets. All values are mean ± std over 3 independent runs. Internal Dataset are internally annotated trajectories while Browserbase OM2W are Fara-7B trajectories from Online-Mind2Web with UV-informed human labels aggregated across two annotators per task by majority vote (outcome) or median rubric score binarized at ≥0.8 (process); see § 5 . For the UV, outcome uses the binary outcome signal and process uses the rubric score binarized at ≥0.8 . WebVoyager and WebJudge each produce a single binary prediction compared against both label types.
Success
Fail
Total
Success Rate
Over-budget (truncated)
44
663
707
6.2%
Terminated (agent stopped)
312
283
595
52.4%
Appendix
Table 16: Partitions AgentRewardBench’s 1302 human annotated trajectories based on its relation to the step budget and human-annotated success
Figure 12: An example false positive from AgentRewardBench. The task is ” Navigate to the item on this page whose image is a desktop screenshot ”. Although this is a screenshot, the spring mattress screenshot is a mobile screenshot, not a desktop screenshot
Figure 13: Process Cohen’s κ agreement with human labels across successive verifier design iterations. Compare with the outcome κ in Figure 1 . Process agreement is consistently lower than outcome agreement for all three settings, reflecting the greater subjectivity of process evaluation.
Figure 14: Outcome false positive rate (FPR) and false negative rate (FNR) across successive design iterations. See Figure 1 for the corresponding Cohen’s κ .
Figure 15: Process false positive rate (FPR) and false negative rate (FNR) across successive design iterations. See Figure 13 for the corresponding Cohen’s κ .
Table 17: Summary of auto-research agent iterations continuing from the human expert’s best verifier. Each run represents a single prompt modification cycle. Green rows were committed (improved κ without increasing FPR), red rows were rolled back, and yellow indicates a stochastic baseline check.
Run
Change Type
What the Agent Did
Why It Helped
2
Nitpick calibration (prompt)
Added explicit test: “Would a reasonable user say this output is useful?” Enumerated always-nitpick scenarios (approximate walk times, price tier symbols, common knowledge inferences).
Fixed 10+ false negatives where minor issues were treated as critical failures.
2
Semantic precision in rubric generation (prompt)
Added rule: criteria must test the exact concept the task asks about, not a related one. E.g., “how many people work remotely” = “how many remote job postings.”
Fixed false positives from rubrics testing the wrong quantity.
6
Rubric score context (code)
Computed normalized rubric score and appended calibration guidance to the outcome prompt. If rubric ≥95% , verifier must identify a specific critical issue to override.
Most impactful single change: provided quantitative signal instead of adding more text to an already-long prompt.
10
Critical point output rule (prompt)
When screenshots confirm the agent reached a transaction boundary (checkout, passenger info page) with correct selections, a brief output message is a nitpick, not grounds for failure.
Fixed persistent false negatives on booking/flight tasks where the agent correctly stopped but didn’t restate details.
11
Forced rule checking (prompt)
Expanded the mandatory example_match_check JSON field to require the LLM to also check named rules (Entity Non-Existence, Multi-Item Cart, Critical Point Output, etc.) before making its verdict.
Mitigated the “rules exist but aren’t applied” problem in ∼1,800 -line prompts.
Appendix
Table 18: Representative prompt and code changes made by the auto-research agent across its iterations. Changes span prompt engineering (calibration rules, forced structured checking) and code modifications (injecting rubric scores as quantitative context).
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.
Zihan Dong, Zhiyuan Ma, Zekun Wang +5
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.
Jinbiao Wei, Qianran Ma, Yilun Zhao +4
Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill
Reinforcement learning for computer-use agents increasingly relies on proactive reward agents that interact with the post-execution GUI environment to verify what recorded screenshots cannot reveal. The reliability of such verification hinges on two coupled decisions: what to check and when to stop. Existing methods leave these decisions to prompted foundation models. However, neither scaling up the model nor providing few-shot examples reliably improves verification: prompted Judges ranging from 4B to GPT-5.4 all plateau at 68%-70% macro-average agreement. This is because the value of a check depends on the environment rather than the task alone, and becomes apparent only through execution. Learning from execution is itself constrained, as the Actor's terminal state cannot be faithfully replicated during training, allowing only one verification attempt per state. We introduce VeriCUA, a proactive reward agent that learns its verification strategy from these single attempts. A trainable Judge decides what to check and when to stop, while a frozen Probe executes the checks in the live environment. Online reinforcement learning optimizes the Judge for both verdict correctness and efficient Probe use, with returns normalized across terminal states. Trained only on OSWorld and evaluated without further training on MobileWorld and WebArena-Verified, VeriCUA achieves the highest reward agreement across all three benchmarks and lies on the agreement-cost Pareto frontier among proactive verifiers. It surpasses few-shot GPT-5.4 by 5.36 points with 47.6%-59.8% fewer Probe tokens, and even a 1B VeriCUA exceeds every prompted Judge. As the reward for Actor RL, it raises task success by 3.21 points overall and 5.40 points on held-out multi-application tasks. These results identify the verification strategy, not verifier scale, as the key to reliable GUI rewards.
Zhizhou Wang, Yaxi Lu, Zhong Zhang +4
Southwest Jiaotong University · Tsinghua University · University of Electronic Science and Technology of China +3