Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep
Figures & tables
Figure 1: Terminal outcome alone cannot distinguish qualitatively different failures. In the task of finding Finley Clark’s message and reacting with a :fire: emoji, Fara fails to locate the correct target, while GUI-Owl reaches the correct thread but applies the wrong reaction.
Figure 2: Overview of WebStep . Each visual website is paired with a semantic MDP dual that converts raw GUI actions into interpretable semantic transitions. For example, clicking and typing merge to transitions such as ViewRepo , StarRepo , and CreateIssue , making explicit which item was visited and which semantic action was taken. These traces enable automatic process-level analyses such as exploration, skill invocation, and step efficiency.
Deterministic
Tasks
Process-eval
Hard negative
Online-Mind2Web ( Xue et al., 2025 )
✗
300
✗
✗
WebVoyager ( He et al., 2024 )
✗
643
✗
✗
WebWalkerQA ( Wu et al., 2025 )
✗
680
✗
✗
AssistantBench ( Yoran et al., 2024 )
✗
214
✗
✗
WebArena ( Zhou et al., 2024 )
✓
812
✗
✗
VisualWebArena ( Koh et al., 2024 )
✓
910
✗
✗
Table 1: Comparison with existing web agent benchmarks. Deterministic : self-hosted deterministic websites. Process eval : Key node uses manual milestones; MDP enables automatic stage decomposition and skill attribution. Hard negative : controlled distractors.
Agent
Success Rate (%) ↑
Information ↑
Steps
Terminal
Exploration
Execution
Coverage (%)
GUI
Semantic
\nicefracGUISemantic↓
Fara-7B
35.7
47.3
75.0
76.4
18.9
8.3
2.3
GUI-Owl-1.5-8B
34.4
43.3
78.6
79.5
28.3
10.4
2.7
UI-TARS-1.5-7B
37.1
49.2
73.9
80.3
35.0
14.0
2.5
Qwen3.5-122B
58.2
66.3
86.7
89.1
22.1
9.8
2.3
OpenAI CUA
82.7
89.4
92.4
96.8
19.7
10.0
2.0
Table 2: Aggregate evaluation results. We report terminal success together with process-level metrics from semantic MDP traces: exploration, execution, and information coverage. These metrics reveal behavioral differences not visible from terminal success alone.
Figure 3: Skill-level strengths and weaknesses. Agents exhibit distinct within-website skill profiles. e.g., on Housing, OpenAI CUA is stronger on navigation but weaker on filter , showing that an agent’s weakness can be concentrated in a specific interaction skill. Point labels (e.g., +16.2% ) denote the gap relative to the compared model.
Figure 4: Temporal skill invocation patterns.
Figure 5: Trajectory bifurcations at shared semantic states. Bifurcation analysis localizes where failure is introduced. (a) Wrong branch : the decision that first sends the failing trajectory off course. (b) Delayed commit : the extra behavior after missing the right time to finish. (c) Premature commit : the missing behavior skipped before finishing too early.
Figure 6: Exploration success by task complexity. (a) Performance by hard negative counts separates OpenAI CUA from others. (b) Oracle trajectory length shows downward trend. (c) Tasks requiring page detail evidence show different model gaps from list-level tasks.
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
Agent
Model
Backend
Input
Action Space
Max Imgs
Temp.
Max Tokens
Fara
microsoft/Fara-7B
vLLM
Screenshot
Function call (coordinate)
3
0.0
800
UI-TARS
ByteDance-Seed/UI-TARS-1.5-7B
vLLM
Screenshot
Thought + Action (coordinate)
5
0.0
2048
GUI-Owl
mPLUG/GUI-Owl-1.5-8B
vLLM
Screenshot
Tool call (coordinate)
4
0.0
2048
Qwen3.5
Qwen/Qwen3.5-122B-FP8
vLLM
Screenshot
Thought + Action (coordinate)
5
0.7
2048
OpenAI CUA
GPT-5.4
OpenAI API
Screenshot
Built-in computer tool (coordinate)
–
–
–
Claude CUA
claude-sonnet-4-6
Anthropic API
Screenshot + DOM
Tool call (ref + coordinate)
3
–
–
Appendix
Table 3: Agent configurations. Input : what the agent receives each turn. Action space : how the agent specifies actions. Max imgs : number of recent screenshots retained in context. Claude CUA follows Anthropic’s DOM-augmented browser harness (ref-based targeting with coordinate fallback), whereas all other agents operate purely from pixels.
Figure 7: WebStep task distribution. The inner ring shows 10 websites (180 tasks each). The outer ring shows task templates with instance counts. Templates range from 4 to 30 instances per template, reflecting the diversity of interaction patterns within each site.
Figure 8: Screenshots of all 10 WebStep websites. Top row : Mail, Calendar, Shopping, Accommodation, Food Delivery. Bottom row : Housing, Coding Q&A, Code Repository, Job Network, and Team Chat. Each site is a standalone HTML/CSS/JS application with full MDP instrumentation.
Table 5: Visibility partition for each site. ∣Vcard∣ : attributes visible on list surfaces. ∣Vdetail∣ : attributes visible only on detail surfaces. Surfaces are categorized as Card (list views), Detail (entity detail pages), and Commit (surfaces where task-completing actions are available).
Category
Description
Navigate
Actions that move between surfaces or manage the current view without modifying application state. Includes pagination, folder/tab switching, back navigation, closing detail views, and temporal navigation (e.g., next/previous week).
Search
Actions that issue or clear search queries to change the visible entity set. Includes keyword search, people search, repository search, and clearing search context.
Filter
Actions that narrow or reorder the visible entities within the current result set. Includes applying categorical or range filters, sorting, and clearing filters.
Inspect
Actions that reveal additional information about an entity without modifying application state. Includes opening detail views, expanding hidden content, viewing profiles, and selecting entities for closer examination.
Commit
The task-completing action: the final step of the task’s oracle trajectory, performed on the target entity—e.g., placing the order, sending the reply, starring the target thread, or, for read-only tasks, opening the target item. State-modifying actions that are not the task’s commit (e.g., adding an item to the cart before placing the order, or filling a form before sending) form the task’s pre-commit sequence ; in trajectory analyses these steps are merged into the step that follows them rather than counted as a separate category.
Appendix
Table 6: Skill category definitions. The four exploration categories (Navigate, Search, Filter, Inspect) are global action-type sets; Commit is task-relative, defined per task as the final step of its oracle trajectory.
Table 7: Skill taxonomy per site (Part 1 of 2). Skills are organized into five categories: Navigate (surface traversal), Search (query formulation), Filter (attribute filtering and sorting), Inspect (entity viewing), and Commit (task-completing actions).
Table 8: Skill taxonomy per site (Part 2 of 2). See Table 6 for category definitions.
Figure 9: Overview of the website pipeline.
Agent
Mail
Cal.
Shop.
Acco.
Food
Hous.
Q&A
Code
Jobs
Chat
Claude CUA
91.7
79.4
82.8
87.2
86.1
79.4
77.8
91.1
84.4
93.3
OpenAI CUA
89.4
73.3
78.9
88.3
81.1
66.7
79.4
88.3
91.1
90.0
Qwen3.5-122B
60.6
52.2
56.1
53.9
73.9
25.6
63.9
65.6
66.7
63.9
Fara
50.0
16.1
37.8
31.1
63.9
13.3
37.2
41.1
40.0
26.1
GUI-Owl
58.3
10.6
12.8
11.7
67.2
20.0
50.6
43.3
35.6
33.9
UI-TARS
71.1
18.9
16.7
8.9
65.6
27.2
46.1
36.7
48.3
31.1
Appendix
Table 9: Terminal Success Rate (%) by site. Best per site in bold.
Agent
Mail
Cal.
Shop.
Acco.
Food
Hous.
Q&A
Code
Jobs
Chat
Claude CUA
91.7
79.4
98.9
93.3
97.8
82.8
85.6
91.1
93.3
93.3
OpenAI CUA
90.0
77.8
95.6
91.7
97.2
70.0
89.4
88.3
92.8
92.2
Qwen3.5-122B
62.8
58.9
71.1
70.0
91.7
27.8
78.9
65.0
67.8
66.7
Fara
53.3
22.2
47.8
52.2
82.2
13.9
52.2
46.1
43.9
55.6
GUI-Owl
58.9
13.3
13.3
47.8
82.2
22.2
60.0
50.6
34.4
47.2
UI-TARS
76.1
26.7
20.0
51.1
82.8
28.9
55.6
41.7
53.3
53.3
Appendix
Table 10: Exploration Success Rate (%) by site. Best per site in bold.
Agent
Mail
Cal.
Shop.
Acco.
Food
Hous.
Q&A
Code
Jobs
Chat
Claude CUA
100.0
100.0
83.7
93.5
88.1
96.0
89.6
100.0
90.5
100.0
OpenAI CUA
99.4
94.3
82.6
96.4
83.4
95.2
88.8
100.0
97.6
97.6
Qwen3.5-122B
94.7
88.7
78.9
77.0
80.6
80.0
79.6
100.0
96.7
95.8
Fara
93.8
72.5
79.1
59.6
77.0
92.0
70.2
89.2
91.1
47.0
GUI-Owl
99.1
79.2
95.8
24.4
81.8
77.5
84.3
85.7
100.0
71.8
UI-TARS
93.4
70.8
83.3
17.4
77.9
82.7
82.0
88.0
87.5
58.3
Appendix
Table 11: Execution SR ∣ Exploration Success (%) by site. Best per site in bold.
Agent
Mail
Cal.
Shop.
Acco.
Food
Hous.
Q&A
Code
Jobs
Chat
Claude CUA
91.3
96.7
100.0
95.5
97.8
100.0
93.3
90.9
98.4
99.1
OpenAI CUA
88.7
97.2
99.6
97.3
98.9
98.9
98.9
90.9
98.4
98.7
Qwen3.5-122B
83.3
94.4
97.2
83.7
93.4
94.4
88.6
68.8
92.7
94.1
Fara
85.0
80.6
93.0
76.6
89.2
62.9
65.3
53.9
76.1
81.7
GUI-Owl
93.9
76.4
74.0
78.5
80.7
96.2
80.6
55.8
72.0
86.8
UI-TARS
84.9
83.1
79.6
81.7
82.5
82.2
81.9
52.5
87.4
86.9
Appendix
Table 12: Informational Coverage at Commit (%) by site. Best per site in bold.
Agent
Site
Search
Filter
Inspect
Navigate
Commit
Claude CUA
Mail
94
–
88
70
79
Cal.
–
–
100
92
81
Shop.
100
21
100
89
99
Acco.
97
99
100
71
97
Food
70
85
100
63
76
Hous.
99
100
91
69
98
Appendix
Table 13: Skill invocation rates (%) by site and agent (Part 1 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
Agent
Site
Search
Filter
Inspect
Navigate
Commit
Fara
Mail
97
–
89
41
39
Cal.
–
–
88
63
21
Shop.
99
71
88
64
35
Acco.
77
83
88
53
6
Food
63
81
96
40
23
Hous.
69
81
71
43
21
Appendix
Table 14: Skill invocation rates (%) by site and agent (Part 2 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
Agent
Search
Filter
Inspect
Navigate
Commit
Claude CUA
92.9
87.8
97.1
80.9
88.3
OpenAI CUA
94.5
78.9
97.2
79.7
83.8
Qwen3.5-122B
90.9
87.2
95.5
76.3
58.5
Fara
80.4
70.9
82.8
60.3
29.4
GUI-Owl
83.3
94.0
78.6
58.1
39.7
UI-TARS
90.5
77.3
82.9
65.2
27.7
Appendix
Table 15: Aggregate skill invocation rates (%). Each cell shows the fraction of episodes requiring that skill where the agent invoked it. Commit invocation requires the trajectory to end with the task’s commit action. Best per column in bold.
Agent
HN=0
HN=1
HN=2
HN=3
Claude CUA
90.6
98.7
90.3
89.8
OpenAI CUA
91.7
93.6
87.9
83.5
Qwen3.5-122B
76.4
78.2
65.2
51.4
Fara
60.5
64.1
41.9
32.8
GUI-Owl
59.4
43.6
34.5
32.3
UI-TARS
60.6
53.8
45.6
37.3
Appendix
Table 16: Exploration SR (%) by hard negative count (the number of card-level confusables planted by the generator; a designed lower bound). Best per column in bold. HN = 0 tasks have no identification ambiguity but vary on other difficulty axes, so monotone trends should be read over HN ≥ 1.
Agent
1
2
3
4
5
6
7
8
9
10
Claude CUA
96.3
96.4
89.9
93.9
87.4
84.9
86.7
89.7
95.4
85.9
OpenAI CUA
94.4
96.7
92.9
94.7
80.7
82.4
80.3
87.2
93.1
73.2
Qwen3.5-122B
72.2
73.8
83.2
75.0
67.4
60.4
51.4
53.3
68.2
45.1
Fara
42.6
45.4
61.3
57.6
48.1
44.7
40.5
36.9
42.8
40.8
GUI-Owl
50.0
43.7
57.1
65.2
41.5
43.4
27.6
31.3
38.7
35.2
UI-TARS
50.0
52.3
54.2
62.9
58.5
52.2
40.8
35.4
43.4
38.0
Appendix
Table 17: Exploration SR (%) by oracle trajectory length. Best per column in bold.
Agent
Card
Filter
Detail
Claude CUA
95.1
98.2
88.3
OpenAI CUA
93.2
96.7
85.8
Qwen3.5-122B
70.4
90.1
59.4
Fara
58.6
72.8
38.9
GUI-Owl
60.5
65.0
35.2
UI-TARS
61.7
61.4
44.2
Appendix
Table 18: Exploration SR (%) by information access level. Best per column in bold. Detail tasks require opening detail pages (non-empty information gap); Filter tasks have no detail gap but their oracle trajectory uses search/filter/sort actions (classified with the shared action taxonomy); Card tasks are solvable from list-level cards alone.
Figure 10: Artifact viewer interface. Left : task list with success/failure indicators. Center : per-turn screenshot browser with navigation controls. Right : episode metadata, per-turn action details (action type, coordinates, resolved MDP target), surface, and coverage. Bottom : semantic MDP trajectory showing the sequence of semantic actions with surface annotations.
Field
Value
Template
find_email_extract
Site
Mail
Instruction pattern
”{sender} has sent you several similar emails. Find the one that mentions ’{keyword}’ in its body and {action} it.”
Information gap
body, cc, attachments (detail-only fields)
Constraint count
3
Commit action
Star / Archive / Label
Appendix
Table 19: Task template: find_email_extract (Mail). Requires searching, inspecting multiple threads to find one matching body-level content, and starring it.
Field
Value
Template
find_product_extract
Site
Shopping
Instruction pattern
”Search for {query} in {department}. Find the one with {detail_attr}: ’{value}’ and add it to your cart.”
Table 20: Task template: find_product_extract (Shopping). Requires searching, navigating to product detail pages, and adding the correct product to cart based on a detail-only attribute.
Field
Value
Template
search_filter_book
Site
Accommodation
Instruction pattern
”Book the {superlative} {property_type} in {location} that has {amenity} for {guests} guests, {dates}.”
Information gap
amenities, cancellation_policy, host details
Constraint count
4–6
Commit action
ConfirmBooking
Appendix
Table 21: Task template: search_filter_book (Accommodation). A multi-step booking task requiring search, filter application, sort, listing inspection, and checkout completion.
Step
Oracle
Fara
UI-TARS
0
SearchEmails("Priya Patel")
OpenThread(THR-006)
SearchEmails("ProjectAlpha006")
1
OpenThread(THR-019)
SearchEmails("ProjectAlpha006")
Star(THR-006)
2
CloseThread()
OpenThread(THR-006)
SwitchFolder(STARRED)
3
OpenThread(THR-050)
Star(THR-006)
4
CloseThread()
5
OpenThread(THR-006)
Appendix
Table 22: Semantic trajectory comparison on mail_0001 . All trajectories are shown as semantic MDP actions extracted from the environment trace. The oracle systematically inspects hard negatives before committing; both agents find target item without sufficient information.
Step
Oracle
Fara
UI-TARS
0
Search("fiction")
Search("Books")
Search("Books")
1
OpenProduct(PRD-039)
ApplyFilter(dept=Books)
ApplyFilter(dept=Books)
2
AddToCart()
OpenProduct(PRD-009)
ClearFilters()
3
GoBack()
ApplyFilter(dept=Books)
4
OpenProduct(PRD-027)
NewSearch("Leather")
5
GoBack()
NewSearch("Leather")
Appendix
Table 23: Semantic trajectory comparison on shopping_0010 . The oracle executes a minimal 3-action sequence. Fara searches but inspects wrong products before adding one to cart (failure-wrong product). UI-TARS repeatedly reformulates queries without reaching the target.
Figure 21: Case 1: Model scale and browsing quality. OpenAI-CUA vs. UI-TARS on the same Mail task (”David Kim has sent you several similar emails. Find the one that mentions ProjectAlpha073 in its body and star it”). Both models succeed (Terminal Pass), yet their trajectories diverge sharply. OpenAI-CUA exhibits a more browsing-like trajectory–searching by sender name ”David Kim,” opening and expanding messages across three threads before finally starring the correct one (14 MDP actions). UI-TARS takes a shortcut, directly searching for ”ProjectAlpha073” to locate the target thread and star it in just 3 actions. The trajectory tables below each screenshot make this gap concrete: model scale produces qualitatively different exploration behavior, and our process metrics make this divergence visible where SR cannot.
Figure 22: Case 2: Premature commitment to a hard negative. UI-TARS on a Shopping task (”Search for women’s dresses in Clothing. Find the one with Warranty: 2 Years and add it to your cart.”). The agent clicks on the Instant Pot Deluxe Cocktail Dress from the search results (Step 1), then immediately adds it to cart without scrolling down to verify the warranty specification (Step 2), and declares the task done (Step 3). The agent’s chain-of-thought confidently claims the dress ”comes with a two-year warranty,” yet this detail was never verified on the product page. This is a premature commitment : the agent selected the first plausible candidate without confirming the critical constraint, exactly the type of hard-negative failure our benchmark’s controlled difficulty design is intended to expose.
Figure 23: Case 3: Thorough exploration before commitment (positive example). UI-TARS on a Shopping task (”Search for camping in Sports & Outdoors. Find the one with Weight: 0.3 lbs and add it to your cart.”). The agent systematically checks all three search results: it opens the first product (Instant Pot sleeping bag), confirms Weight = 0.3 lbs in the Technical Details (Step 2), then returns to the search results and checks the second product (Sony Classic Lantern, 2.5 lbs, Step 4) and the third (Samsung lantern, 3.8 lbs, Step 6) before returning to add the correct item to cart (Step 8). Coverage = 100% at commit. Task: PASS. A positive contrast to Case 2: the agent’s chain-of-thought explicitly tracks constraints (” I still need to keep looking for other suitable products ”), verifying its choice was uniquely correct before committing.
Figure 24: Case 4: Exploration success, execution failure. Two examples from Fara on Coding tasks. Left: Fara navigates to the correct StackOverflow answer by bug_hunter and attempts to upvote it, but the click coordinates (150, 157) land on the code block instead of the upvote button (Verification: Exploration Success ✓ , Execution Failure × ). Right: Fara navigates to the correct GitHub repository ”ultra-pipeline” and attempts to star it, but the click coordinates (1275, 250) miss the Star button (Verification: Exploration Success ✓ , Execution Failure × ). This execution bottleneck (missing small UI targets due to imprecise coordinate grounding) accounts for much of Fara’s gap between its Exploration rate and task SR.
Figure 25: Case 5: Exploration failure, execution success. Fara on a Code Repo task (”Search for ’api’ and filter by TypeScript language. Sort by forks and star the one with the most forks.”). At Step 1, the agent intends to select ”TypeScript” from the Language filter dropdown, but instead clicks a ”TypeScript” label displayed in a repository’s language listing: mistaking it for the correct filter option. The filter is not applied, yet the agent proceeds to navigate back (Step 2), sort by forks (Steps 3–4), and star the top repository (Step 5), declaring success at Step 6. The execution mechanics are correct; the exploration target is wrong. Task: FAIL. The mirror image of Case 4: our four-way taxonomy classifies this as an exploration failure with correct execution : the agent knew how to commit but not where to look.
Figure 26: Case 6: Redundant process despite success. Fara on a Shopping task (”Add the Apple Deluxe Mobile and the Samsung Essential Pot Set to your cart. Then remove the Apple Deluxe Mobile from the cart.”). The oracle trajectory completes the task in 9 MDP actions; Fara executes 13, with 6 extra steps highlighted in orange in the agent trajectory table. After successfully adding the first product, Fara issues garbled concatenated queries (”Apple Deluxe MobilSamsung Essential Pot Set…”) that return no results (Steps 8–9), navigates to the home page (Step 10), and searches by category (”Home &…”) before finally locating the second product. The task ultimately succeeds (Terminal Pass), but the extra steps reveal inefficient error recovery. Our process-recording system captures this waste: process-level evaluation distinguishes efficient success from success-by-persistence.
Figure 27: Case 7: Failure by safeguard, not by competence. Fara on an Accommodation task (Book the cheapest condo in Seattle that has a Pool for 3 guests, April 14–17.”). The agent successfully explores all listings, identifies the correct property (Vibrant Loft in Capitol Hill, $485/night), and navigates to the final ”Confirm and pay” page, achieving Exploration: Pass, Coverage: 100%, and Hard Neg: 0. However, Fara is trained to stop when completing the task would require entering personal and payment information, so it issues Done at step 16 without clicking ”Confirm and pay.” The task is marked FAIL under standard SR, yet our Safe Pass metric recognizes that the agent performed every step within its allowed scope correctly. This case motivates the Safe Pass metric: it distinguishes agents that fail due to a built-in safety guardrail from those that fail due to genuine incompetence.
Developing autonomous agents for web-based tasks is a core challenge in AI. While Large Language Model (LLM) agents can interpret complex user requests, they often operate as black boxes, making it difficult to diagnose why they fail or how they plan. This paper addresses this gap by formally treating web tasks as sequential decision-making processes. We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms: Step-by-Step agents to Breadth-First Search (BFS), Tree Search agents to Best-First Tree Search, and Full-Plan-in-Advance agents to Depth-First Search (DFS). This framework allows for a principled diagnosis of system failures like context drift and incoherent task decomposition. To evaluate these behaviors, we propose five novel evaluation metrics that assess trajectory quality beyond simple success rates. We support this analysis with a new dataset of 794 human-labeled trajectories from the WebArena benchmark. Finally, we validate our evaluation framework by comparing a baseline Step-by-Step agent against a novel Full-Plan-in-Advance implementation. Our results reveal that while the Step-by-Step agent aligns more closely with human gold trajectories (38% overall success), the Full-Plan-in-Advance agent excels in technical measures such as element accuracy (89%), demonstrating the necessity of our proposed metrics for selecting appropriate agent architectures based on specific application constraints.
Orit Shahnovsky, Rotem Dror
Faculty of Computer and Information Science, University of Haifa, Haifa, Israel
Web agents are an important application of large language models, yet their evaluation often depends on rule based or language model evaluators that inspect only the final outcome. Human verification of task completion and detailed analysis of failed trajectories remain limited. We audit all 165 WebArena Lite tasks under six evaluation conditions built from GPT 5.5 and an untrained Qwen3.5 9B model. The audit retains the original score, corrects false negatives from the automatic evaluator, identifies the first consequential error, and examines progress across the trajectory. We also study a Memory and Analysis Support Mechanism (MASM), which maintains explicit execution state, and Guide Text, which provides task relevant procedural guidance. Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator. With a 25 step budget, Guide Text raises corrected success with MASM from 34.55% to 38.18%. On the untrained Qwen3.5 9B model, MASM raises the evaluator score from 13.90% to 18.80%. Review of 102 failed GPT 5.5 trajectories reveals frequent scrolling loops, unfinished exploration, premature answers, invalid actions, and incomplete form workflows. Step level evidence further shows that substantial early progress can coexist with a final failure. These results show why final scores alone provide an incomplete account of web agent behavior and motivate human grounded, trajectory aware verification.
Reliable evaluation of AI agents operating in complex, real-world environments requires methodologies that are robust, transparent, and contextually aligned with the tasks agents are intended to perform. This study identifies persistent shortcomings in existing AI agent evaluation practices that are particularly acute in web agent evaluation, as exemplified by our audit of WebVoyager, including task-framing ambiguity and operational variability that hinder meaningful and reproducible performance comparisons. To address these challenges, we introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology through clear guidelines for task instantiation, failure handling, annotation, and reporting. Emergence WebVoyager achieves an inter-annotator agreement of 95.9%, indicating improved clarity and reliability in both task formulation and evaluation. Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types, with an overall success rate of 68.6%, substantially lower than the 87% previously reported by OpenAI, demonstrating the utility of our approach for more rigorous and comparable web agent evaluation.