cs.AIApr 8, 2026
SaveWhere Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
Organizations: Yonsei University · Microsoft Research
Abstract
Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep
Figures & tables
Figure 1: Terminal outcome alone cannot distinguish qualitatively different failures. In the task of finding Finley Clark’s message and reacting with a :fire: emoji, Fara fails to locate the correct target, while GUI-Owl reaches the correct thread but applies the wrong reaction.
Figure 2: Overview of WebStep . Each visual website is paired with a semantic MDP dual that converts raw GUI actions into interpretable semantic transitions. For example, clicking and typing merge to transitions such as ViewRepo , StarRepo , and CreateIssue , making explicit which item was visited and which semantic action was taken. These traces enable automatic process-level analyses such as exploration, skill invocation, and step efficiency.
| Deterministic | Tasks | Process-eval | Hard negative | |
|---|---|---|---|---|
| Online-Mind2Web ( Xue et al., 2025 ) | ✗ | 300 | ✗ | ✗ |
| WebVoyager ( He et al., 2024 ) | ✗ | 643 | ✗ | ✗ |
| WebWalkerQA ( Wu et al., 2025 ) | ✗ | 680 | ✗ | ✗ |
| AssistantBench ( Yoran et al., 2024 ) | ✗ | 214 | ✗ | ✗ |
| WebArena ( Zhou et al., 2024 ) | ✓ | 812 | ✗ | ✗ |
| VisualWebArena ( Koh et al., 2024 ) | ✓ | 910 | ✗ | ✗ |
Table 1: Comparison with existing web agent benchmarks. Deterministic : self-hosted deterministic websites. Process eval : Key node uses manual milestones; MDP enables automatic stage decomposition and skill attribution. Hard negative : controlled distractors.
| Agent | Success Rate (%) | Information | Steps | ||||
|---|---|---|---|---|---|---|---|
| Terminal | Exploration | Execution | Coverage (%) | GUI | Semantic | ||
| Fara-7B | 35.7 | 47.3 | 75.0 | 76.4 | 18.9 | 8.3 | 2.3 |
| GUI-Owl-1.5-8B | 34.4 | 43.3 | 78.6 | 79.5 | 28.3 | 10.4 | 2.7 |
| UI-TARS-1.5-7B | 37.1 | 49.2 | 73.9 | 80.3 | 35.0 | 14.0 | 2.5 |
| Qwen3.5-122B | 58.2 | 66.3 | 86.7 | 89.1 | 22.1 | 9.8 | 2.3 |
| OpenAI CUA | 82.7 | 89.4 | 92.4 | 96.8 | 19.7 | 10.0 | 2.0 |
Table 2: Aggregate evaluation results. We report terminal success together with process-level metrics from semantic MDP traces: exploration, execution, and information coverage. These metrics reveal behavioral differences not visible from terminal success alone.
Figure 3: Skill-level strengths and weaknesses. Agents exhibit distinct within-website skill profiles. e.g., on Housing, OpenAI CUA is stronger on navigation but weaker on filter , showing that an agent’s weakness can be concentrated in a specific interaction skill. Point labels (e.g., ) denote the gap relative to the compared model.
Figure 4: Temporal skill invocation patterns.
Figure 5: Trajectory bifurcations at shared semantic states. Bifurcation analysis localizes where failure is introduced. (a) Wrong branch : the decision that first sends the failing trajectory off course. (b) Delayed commit : the extra behavior after missing the right time to finish. (c) Premature commit : the missing behavior skipped before finishing too early.
Figure 6: Exploration success by task complexity. (a) Performance by hard negative counts separates OpenAI CUA from others. (b) Oracle trajectory length shows downward trend. (c) Tasks requiring page detail evidence show different model gaps from list-level tasks.
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent | Model | Backend | Input | Action Space | Max Imgs | Temp. | Max Tokens |
|---|---|---|---|---|---|---|---|
| Fara | microsoft/Fara-7B | vLLM | Screenshot | Function call (coordinate) | 3 | 0.0 | 800 |
| UI-TARS | ByteDance-Seed/UI-TARS-1.5-7B | vLLM | Screenshot | Thought + Action (coordinate) | 5 | 0.0 | 2048 |
| GUI-Owl | mPLUG/GUI-Owl-1.5-8B | vLLM | Screenshot | Tool call (coordinate) | 4 | 0.0 | 2048 |
| Qwen3.5 | Qwen/Qwen3.5-122B-FP8 | vLLM | Screenshot | Thought + Action (coordinate) | 5 | 0.7 | 2048 |
| OpenAI CUA | GPT-5.4 | OpenAI API | Screenshot | Built-in computer tool (coordinate) | – | – | – |
| Claude CUA | claude-sonnet-4-6 | Anthropic API | Screenshot + DOM | Tool call (ref + coordinate) | 3 | – | – |
Table 3: Agent configurations. Input : what the agent receives each turn. Action space : how the agent specifies actions. Max imgs : number of recent screenshots retained in context. Claude CUA follows Anthropic’s DOM-augmented browser harness (ref-based targeting with coordinate fallback), whereas all other agents operate purely from pixels.
Figure 7: WebStep task distribution. The inner ring shows 10 websites (180 tasks each). The outer ring shows task templates with instance counts. Templates range from 4 to 30 instances per template, reflecting the diversity of interaction patterns within each site.
Figure 8: Screenshots of all 10 WebStep websites. Top row : Mail, Calendar, Shopping, Accommodation, Food Delivery. Bottom row : Housing, Coding Q&A, Code Repository, Job Network, and Team Chat. Each site is a standalone HTML/CSS/JS application with full MDP instrumentation.
| Site | Domain | Surfaces | Actions | Templates | Tasks | Card / Detail Fields | Decision Object |
|---|---|---|---|---|---|---|---|
| Productivity | 3 | 36 | 6 | 180 | 10 / 9 | Thread | |
| Calendar | Productivity | 6 | 14 | 6 | 180 | 8 / 6 | Event |
| Team Chat | Productivity | 3 | 24 | 12 | 180 | 6 / 5 | Message |
| Shopping | E-Commerce | 6 | 23 | 6 | 180 | 12 / 12 | Product |
| Food Delivery | E-Commerce | 6 | 24 | 15 | 180 | 8 / 7 | Restaurant / Item |
| Accommodation | Discovery | 4 | 16 | 12 | 180 | 9 / 8 | Listing |
Table 4: Per-site MDP statistics. Surfaces : distinct UI views (pages, modals). Actions : typed semantic action count. Templates : task template count. Tasks : total task instances. Card fields : attributes visible on list views. Detail fields : attributes requiring detail-page navigation.
| Site | Card Surfaces | Detail Surfaces | Commit Surfaces | ||
|---|---|---|---|---|---|
| ThreadList | ThreadView | ComposeModal, ThreadList, ThreadView | 10 | 9 | |
| Calendar | WeekView, DayView, MonthView | EventDetail | EventEditor | 8 | 6 |
| Team Chat | ChannelView, SearchResults | ChannelView (thread) | ChannelView, DirectMessageView | 6 | 5 |
| Shopping | SearchResults | ProductDetail | Checkout, OrderConfirmation | 12 | 12 |
| Food Delivery | RestaurantList | RestaurantDetail, MenuItemDetail | Checkout, OrderStatus | 8 | 7 |
| Accommodation | SearchResults | ListingDetail | Checkout, Reservations | 9 | 8 |
Table 5: Visibility partition for each site. : attributes visible on list surfaces. : attributes visible only on detail surfaces. Surfaces are categorized as Card (list views), Detail (entity detail pages), and Commit (surfaces where task-completing actions are available).
| Category | Description |
|---|---|
| Navigate | Actions that move between surfaces or manage the current view without modifying application state. Includes pagination, folder/tab switching, back navigation, closing detail views, and temporal navigation (e.g., next/previous week). |
| Search | Actions that issue or clear search queries to change the visible entity set. Includes keyword search, people search, repository search, and clearing search context. |
| Filter | Actions that narrow or reorder the visible entities within the current result set. Includes applying categorical or range filters, sorting, and clearing filters. |
| Inspect | Actions that reveal additional information about an entity without modifying application state. Includes opening detail views, expanding hidden content, viewing profiles, and selecting entities for closer examination. |
| Commit | The task-completing action: the final step of the task’s oracle trajectory, performed on the target entity—e.g., placing the order, sending the reply, starring the target thread, or, for read-only tasks, opening the target item. State-modifying actions that are not the task’s commit (e.g., adding an item to the cart before placing the order, or filling a form before sending) form the task’s pre-commit sequence ; in trajectory analyses these steps are merged into the step that follows them rather than counted as a separate category. |
Table 6: Skill category definitions. The four exploration categories (Navigate, Search, Filter, Inspect) are global action-type sets; Commit is task-relative, defined per task as the final step of its oracle trajectory.
| Site | Category | Skills |
|---|---|---|
| Navigate | folder_navigation, pagination, backtrack_navigation | |
| Search | search_refinement | |
| Filter | filter_application | |
| Inspect | thread_inspection, message_expansion | |
| Commit | compose_setup, compose_field_entry, email_management, bulk_management, send_commit | |
| Calendar | Navigate | temporal_navigation, view_switching, backtrack_navigation |
Table 7: Skill taxonomy per site (Part 1 of 2). Skills are organized into five categories: Navigate (surface traversal), Search (query formulation), Filter (attribute filtering and sorting), Inspect (entity viewing), and Commit (task-completing actions).
| Site | Category | Skills |
|---|---|---|
| Housing | Navigate | pagination |
| Search | query_formulation | |
| Filter | price_filtering, beds_filtering, baths_filtering, sqft_filtering, type_filtering, attribute_filtering, sort_usage | |
| Inspect | property_inspection, agent_inspection, comparison | |
| Commit | save_management, tour_scheduling, agent_contact, saved_homes_review | |
| Coding Q&A | Navigate | back_navigation, pagination |
Table 8: Skill taxonomy per site (Part 2 of 2). See Table 6 for category definitions.
Figure 9: Overview of the website pipeline.
| Agent | Cal. | Shop. | Acco. | Food | Hous. | Q&A | Code | Jobs | Chat | |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude CUA | 91.7 | 79.4 | 82.8 | 87.2 | 86.1 | 79.4 | 77.8 | 91.1 | 84.4 | 93.3 |
| OpenAI CUA | 89.4 | 73.3 | 78.9 | 88.3 | 81.1 | 66.7 | 79.4 | 88.3 | 91.1 | 90.0 |
| Qwen3.5-122B | 60.6 | 52.2 | 56.1 | 53.9 | 73.9 | 25.6 | 63.9 | 65.6 | 66.7 | 63.9 |
| Fara | 50.0 | 16.1 | 37.8 | 31.1 | 63.9 | 13.3 | 37.2 | 41.1 | 40.0 | 26.1 |
| GUI-Owl | 58.3 | 10.6 | 12.8 | 11.7 | 67.2 | 20.0 | 50.6 | 43.3 | 35.6 | 33.9 |
| UI-TARS | 71.1 | 18.9 | 16.7 | 8.9 | 65.6 | 27.2 | 46.1 | 36.7 | 48.3 | 31.1 |
Table 9: Terminal Success Rate (%) by site. Best per site in bold.
| Agent | Cal. | Shop. | Acco. | Food | Hous. | Q&A | Code | Jobs | Chat | |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude CUA | 91.7 | 79.4 | 98.9 | 93.3 | 97.8 | 82.8 | 85.6 | 91.1 | 93.3 | 93.3 |
| OpenAI CUA | 90.0 | 77.8 | 95.6 | 91.7 | 97.2 | 70.0 | 89.4 | 88.3 | 92.8 | 92.2 |
| Qwen3.5-122B | 62.8 | 58.9 | 71.1 | 70.0 | 91.7 | 27.8 | 78.9 | 65.0 | 67.8 | 66.7 |
| Fara | 53.3 | 22.2 | 47.8 | 52.2 | 82.2 | 13.9 | 52.2 | 46.1 | 43.9 | 55.6 |
| GUI-Owl | 58.9 | 13.3 | 13.3 | 47.8 | 82.2 | 22.2 | 60.0 | 50.6 | 34.4 | 47.2 |
| UI-TARS | 76.1 | 26.7 | 20.0 | 51.1 | 82.8 | 28.9 | 55.6 | 41.7 | 53.3 | 53.3 |
Table 10: Exploration Success Rate (%) by site. Best per site in bold.
| Agent | Cal. | Shop. | Acco. | Food | Hous. | Q&A | Code | Jobs | Chat | |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude CUA | 100.0 | 100.0 | 83.7 | 93.5 | 88.1 | 96.0 | 89.6 | 100.0 | 90.5 | 100.0 |
| OpenAI CUA | 99.4 | 94.3 | 82.6 | 96.4 | 83.4 | 95.2 | 88.8 | 100.0 | 97.6 | 97.6 |
| Qwen3.5-122B | 94.7 | 88.7 | 78.9 | 77.0 | 80.6 | 80.0 | 79.6 | 100.0 | 96.7 | 95.8 |
| Fara | 93.8 | 72.5 | 79.1 | 59.6 | 77.0 | 92.0 | 70.2 | 89.2 | 91.1 | 47.0 |
| GUI-Owl | 99.1 | 79.2 | 95.8 | 24.4 | 81.8 | 77.5 | 84.3 | 85.7 | 100.0 | 71.8 |
| UI-TARS | 93.4 | 70.8 | 83.3 | 17.4 | 77.9 | 82.7 | 82.0 | 88.0 | 87.5 | 58.3 |
Table 11: Execution SR Exploration Success (%) by site. Best per site in bold.
| Agent | Cal. | Shop. | Acco. | Food | Hous. | Q&A | Code | Jobs | Chat | |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude CUA | 91.3 | 96.7 | 100.0 | 95.5 | 97.8 | 100.0 | 93.3 | 90.9 | 98.4 | 99.1 |
| OpenAI CUA | 88.7 | 97.2 | 99.6 | 97.3 | 98.9 | 98.9 | 98.9 | 90.9 | 98.4 | 98.7 |
| Qwen3.5-122B | 83.3 | 94.4 | 97.2 | 83.7 | 93.4 | 94.4 | 88.6 | 68.8 | 92.7 | 94.1 |
| Fara | 85.0 | 80.6 | 93.0 | 76.6 | 89.2 | 62.9 | 65.3 | 53.9 | 76.1 | 81.7 |
| GUI-Owl | 93.9 | 76.4 | 74.0 | 78.5 | 80.7 | 96.2 | 80.6 | 55.8 | 72.0 | 86.8 |
| UI-TARS | 84.9 | 83.1 | 79.6 | 81.7 | 82.5 | 82.2 | 81.9 | 52.5 | 87.4 | 86.9 |
Table 12: Informational Coverage at Commit (%) by site. Best per site in bold.
| Agent | Site | Search | Filter | Inspect | Navigate | Commit |
|---|---|---|---|---|---|---|
| Claude CUA | 94 | – | 88 | 70 | 79 | |
| Cal. | – | – | 100 | 92 | 81 | |
| Shop. | 100 | 21 | 100 | 89 | 99 | |
| Acco. | 97 | 99 | 100 | 71 | 97 | |
| Food | 70 | 85 | 100 | 63 | 76 | |
| Hous. | 99 | 100 | 91 | 69 | 98 |
Table 13: Skill invocation rates (%) by site and agent (Part 1 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
| Agent | Site | Search | Filter | Inspect | Navigate | Commit |
|---|---|---|---|---|---|---|
| Fara | 97 | – | 89 | 41 | 39 | |
| Cal. | – | – | 88 | 63 | 21 | |
| Shop. | 99 | 71 | 88 | 64 | 35 | |
| Acco. | 77 | 83 | 88 | 53 | 6 | |
| Food | 63 | 81 | 96 | 40 | 23 | |
| Hous. | 69 | 81 | 71 | 43 | 21 |
Table 14: Skill invocation rates (%) by site and agent (Part 2 of 2). ”–” = fewer than 20 required episodes. Commit is task-relative: the final task-completing action of each task’s oracle trajectory. Best value per site–skill cell across all six agents in bold.
| Agent | Search | Filter | Inspect | Navigate | Commit |
|---|---|---|---|---|---|
| Claude CUA | 92.9 | 87.8 | 97.1 | 80.9 | 88.3 |
| OpenAI CUA | 94.5 | 78.9 | 97.2 | 79.7 | 83.8 |
| Qwen3.5-122B | 90.9 | 87.2 | 95.5 | 76.3 | 58.5 |
| Fara | 80.4 | 70.9 | 82.8 | 60.3 | 29.4 |
| GUI-Owl | 83.3 | 94.0 | 78.6 | 58.1 | 39.7 |
| UI-TARS | 90.5 | 77.3 | 82.9 | 65.2 | 27.7 |
Table 15: Aggregate skill invocation rates (%). Each cell shows the fraction of episodes requiring that skill where the agent invoked it. Commit invocation requires the trajectory to end with the task’s commit action. Best per column in bold.
| Agent | HN=0 | HN=1 | HN=2 | HN=3 |
|---|---|---|---|---|
| Claude CUA | 90.6 | 98.7 | 90.3 | 89.8 |
| OpenAI CUA | 91.7 | 93.6 | 87.9 | 83.5 |
| Qwen3.5-122B | 76.4 | 78.2 | 65.2 | 51.4 |
| Fara | 60.5 | 64.1 | 41.9 | 32.8 |
| GUI-Owl | 59.4 | 43.6 | 34.5 | 32.3 |
| UI-TARS | 60.6 | 53.8 | 45.6 | 37.3 |
Table 16: Exploration SR (%) by hard negative count (the number of card-level confusables planted by the generator; a designed lower bound). Best per column in bold. HN 0 tasks have no identification ambiguity but vary on other difficulty axes, so monotone trends should be read over HN 1.
| Agent | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude CUA | 96.3 | 96.4 | 89.9 | 93.9 | 87.4 | 84.9 | 86.7 | 89.7 | 95.4 | 85.9 |
| OpenAI CUA | 94.4 | 96.7 | 92.9 | 94.7 | 80.7 | 82.4 | 80.3 | 87.2 | 93.1 | 73.2 |
| Qwen3.5-122B | 72.2 | 73.8 | 83.2 | 75.0 | 67.4 | 60.4 | 51.4 | 53.3 | 68.2 | 45.1 |
| Fara | 42.6 | 45.4 | 61.3 | 57.6 | 48.1 | 44.7 | 40.5 | 36.9 | 42.8 | 40.8 |
| GUI-Owl | 50.0 | 43.7 | 57.1 | 65.2 | 41.5 | 43.4 | 27.6 | 31.3 | 38.7 | 35.2 |
| UI-TARS | 50.0 | 52.3 | 54.2 | 62.9 | 58.5 | 52.2 | 40.8 | 35.4 | 43.4 | 38.0 |
Table 17: Exploration SR (%) by oracle trajectory length. Best per column in bold.
| Agent | Card | Filter | Detail |
|---|---|---|---|
| Claude CUA | 95.1 | 98.2 | 88.3 |
| OpenAI CUA | 93.2 | 96.7 | 85.8 |
| Qwen3.5-122B | 70.4 | 90.1 | 59.4 |
| Fara | 58.6 | 72.8 | 38.9 |
| GUI-Owl | 60.5 | 65.0 | 35.2 |
| UI-TARS | 61.7 | 61.4 | 44.2 |
Table 18: Exploration SR (%) by information access level. Best per column in bold. Detail tasks require opening detail pages (non-empty information gap); Filter tasks have no detail gap but their oracle trajectory uses search/filter/sort actions (classified with the shared action taxonomy); Card tasks are solvable from list-level cards alone.
Figure 10: Artifact viewer interface. Left : task list with success/failure indicators. Center : per-turn screenshot browser with navigation controls. Right : episode metadata, per-turn action details (action type, coordinates, resolved MDP target), surface, and coverage. Bottom : semantic MDP trajectory showing the sequence of semantic actions with surface annotations.
| Field | Value |
|---|---|
| Template | find_email_extract |
| Site | |
| Instruction pattern | ”{sender} has sent you several similar emails. Find the one that mentions ’{keyword}’ in its body and {action} it.” |
| Information gap | body, cc, attachments (detail-only fields) |
| Constraint count | 3 |
| Commit action | Star / Archive / Label |
Table 19: Task template: find_email_extract (Mail). Requires searching, inspecting multiple threads to find one matching body-level content, and starring it.
| Field | Value |
|---|---|
| Template | find_product_extract |
| Site | Shopping |
| Instruction pattern | ”Search for {query} in {department}. Find the one with {detail_attr}: ’{value}’ and add it to your cart.” |
| Information gap | specifications, bullet_points, seller, shipping_cost |
| Constraint count | 1–3 |
| Commit action | AddToCart |
Table 20: Task template: find_product_extract (Shopping). Requires searching, navigating to product detail pages, and adding the correct product to cart based on a detail-only attribute.
| Field | Value |
|---|---|
| Template | search_filter_book |
| Site | Accommodation |
| Instruction pattern | ”Book the {superlative} {property_type} in {location} that has {amenity} for {guests} guests, {dates}.” |
| Information gap | amenities, cancellation_policy, host details |
| Constraint count | 4–6 |
| Commit action | ConfirmBooking |
Table 21: Task template: search_filter_book (Accommodation). A multi-step booking task requiring search, filter application, sort, listing inspection, and checkout completion.
| Step | Oracle | Fara | UI-TARS |
|---|---|---|---|
| 0 | SearchEmails("Priya Patel") | OpenThread(THR-006) | SearchEmails("ProjectAlpha006") |
| 1 | OpenThread(THR-019) | SearchEmails("ProjectAlpha006") | Star(THR-006) |
| 2 | CloseThread() | OpenThread(THR-006) | SwitchFolder(STARRED) |
| 3 | OpenThread(THR-050) | Star(THR-006) | |
| 4 | CloseThread() | ||
| 5 | OpenThread(THR-006) |
Table 22: Semantic trajectory comparison on mail_0001 . All trajectories are shown as semantic MDP actions extracted from the environment trace. The oracle systematically inspects hard negatives before committing; both agents find target item without sufficient information.
| Step | Oracle | Fara | UI-TARS |
|---|---|---|---|
| 0 | Search("fiction") | Search("Books") | Search("Books") |
| 1 | OpenProduct(PRD-039) | ApplyFilter(dept=Books) | ApplyFilter(dept=Books) |
| 2 | AddToCart() | OpenProduct(PRD-009) | ClearFilters() |
| 3 | GoBack() | ApplyFilter(dept=Books) | |
| 4 | OpenProduct(PRD-027) | NewSearch("Leather") | |
| 5 | GoBack() | NewSearch("Leather") |
Table 23: Semantic trajectory comparison on shopping_0010 . The oracle executes a minimal 3-action sequence. Fara searches but inspects wrong products before adding one to cart (failure-wrong product). UI-TARS repeatedly reformulates queries without reaching the target.
Figure 11: Mail site: 3 surfaces, 36 typed actions (8 inter-surface, 28 intra-surface).
Figure 12: Calendar site: 6 surfaces, 14 typed actions (9 inter-surface, 5 intra-surface).
Figure 13: Shopping site: 6 surfaces, 23 typed actions (8 inter-surface, 15 intra-surface).
Figure 14: Accommodation site: 4 surfaces, 16 typed actions (7 inter-surface, 9 intra-surface).
Figure 15: Food Delivery site: 6 surfaces, 24 typed actions (13 inter-surface, 11 intra-surface).
Figure 16: Housing site: 4 surfaces, 16 typed actions (8 inter-surface, 8 intra-surface).
Figure 17: Coding Q&A site: 4 surfaces, 18 typed actions (12 inter-surface, 6 intra-surface).
Figure 18: Code Repo site: 6 surfaces, 32 typed actions (11 inter-surface, 21 intra-surface).
Figure 19: Job Network site: 6 surfaces, 26 typed actions (18 inter-surface, 8 intra-surface).
Figure 20: Team Chat site: 3 surfaces, 24 typed actions (8 inter-surface, 16 intra-surface).
Figure 21: Case 1: Model scale and browsing quality. OpenAI-CUA vs. UI-TARS on the same Mail task (”David Kim has sent you several similar emails. Find the one that mentions ProjectAlpha073 in its body and star it”). Both models succeed (Terminal Pass), yet their trajectories diverge sharply. OpenAI-CUA exhibits a more browsing-like trajectory–searching by sender name ”David Kim,” opening and expanding messages across three threads before finally starring the correct one (14 MDP actions). UI-TARS takes a shortcut, directly searching for ”ProjectAlpha073” to locate the target thread and star it in just 3 actions. The trajectory tables below each screenshot make this gap concrete: model scale produces qualitatively different exploration behavior, and our process metrics make this divergence visible where SR cannot.
Figure 22: Case 2: Premature commitment to a hard negative. UI-TARS on a Shopping task (”Search for women’s dresses in Clothing. Find the one with Warranty: 2 Years and add it to your cart.”). The agent clicks on the Instant Pot Deluxe Cocktail Dress from the search results (Step 1), then immediately adds it to cart without scrolling down to verify the warranty specification (Step 2), and declares the task done (Step 3). The agent’s chain-of-thought confidently claims the dress ”comes with a two-year warranty,” yet this detail was never verified on the product page. This is a premature commitment : the agent selected the first plausible candidate without confirming the critical constraint, exactly the type of hard-negative failure our benchmark’s controlled difficulty design is intended to expose.
Figure 23: Case 3: Thorough exploration before commitment (positive example). UI-TARS on a Shopping task (”Search for camping in Sports & Outdoors. Find the one with Weight: 0.3 lbs and add it to your cart.”). The agent systematically checks all three search results: it opens the first product (Instant Pot sleeping bag), confirms Weight = 0.3 lbs in the Technical Details (Step 2), then returns to the search results and checks the second product (Sony Classic Lantern, 2.5 lbs, Step 4) and the third (Samsung lantern, 3.8 lbs, Step 6) before returning to add the correct item to cart (Step 8). Coverage = 100% at commit. Task: PASS. A positive contrast to Case 2: the agent’s chain-of-thought explicitly tracks constraints (” I still need to keep looking for other suitable products ”), verifying its choice was uniquely correct before committing.
Figure 24: Case 4: Exploration success, execution failure. Two examples from Fara on Coding tasks. Left: Fara navigates to the correct StackOverflow answer by bug_hunter and attempts to upvote it, but the click coordinates (150, 157) land on the code block instead of the upvote button (Verification: Exploration Success , Execution Failure ). Right: Fara navigates to the correct GitHub repository ”ultra-pipeline” and attempts to star it, but the click coordinates (1275, 250) miss the Star button (Verification: Exploration Success , Execution Failure ). This execution bottleneck (missing small UI targets due to imprecise coordinate grounding) accounts for much of Fara’s gap between its Exploration rate and task SR.
Figure 25: Case 5: Exploration failure, execution success. Fara on a Code Repo task (”Search for ’api’ and filter by TypeScript language. Sort by forks and star the one with the most forks.”). At Step 1, the agent intends to select ”TypeScript” from the Language filter dropdown, but instead clicks a ”TypeScript” label displayed in a repository’s language listing: mistaking it for the correct filter option. The filter is not applied, yet the agent proceeds to navigate back (Step 2), sort by forks (Steps 3–4), and star the top repository (Step 5), declaring success at Step 6. The execution mechanics are correct; the exploration target is wrong. Task: FAIL. The mirror image of Case 4: our four-way taxonomy classifies this as an exploration failure with correct execution : the agent knew how to commit but not where to look.
Figure 26: Case 6: Redundant process despite success. Fara on a Shopping task (”Add the Apple Deluxe Mobile and the Samsung Essential Pot Set to your cart. Then remove the Apple Deluxe Mobile from the cart.”). The oracle trajectory completes the task in 9 MDP actions; Fara executes 13, with 6 extra steps highlighted in orange in the agent trajectory table. After successfully adding the first product, Fara issues garbled concatenated queries (”Apple Deluxe MobilSamsung Essential Pot Set…”) that return no results (Steps 8–9), navigates to the home page (Step 10), and searches by category (”Home &…”) before finally locating the second product. The task ultimately succeeds (Terminal Pass), but the extra steps reveal inefficient error recovery. Our process-recording system captures this waste: process-level evaluation distinguishes efficient success from success-by-persistence.
Figure 27: Case 7: Failure by safeguard, not by competence. Fara on an Accommodation task (Book the cheapest condo in Seattle that has a Pool for 3 guests, April 14–17.”). The agent successfully explores all listings, identifies the correct property (Vibrant Loft in Capitol Hill, $485/night), and navigates to the final ”Confirm and pay” page, achieving Exploration: Pass, Coverage: 100%, and Hard Neg: 0. However, Fara is trained to stop when completing the task would require entering personal and payment information, so it issues Done at step 16 without clicking ”Confirm and pay.” The task is marked FAIL under standard SR, yet our Safe Pass metric recognizes that the agent performed every step within its allowed scope correctly. This case motivates the Safe Pass metric: it distinguishes agents that fail due to a built-in safety guardrail from those that fail due to genuine incompetence.