Web agents based on large language models (LLMs) rely on observations of web pages -- commonly represented as HTML -- as the basis for identifying available actions and planning subsequent steps. Prior work has treated the verbosity of HTML as an obstacle to performance and adopted observation reduction as a standard practice. We revisit this trend and demonstrate that the optimal observation representation depends on model capability and thinking token budget: (1) compact observations (accessibility trees) are preferable for lower-capability models, while detailed observations (HTML) are advantageous for higher-capability models; moreover, increasing thinking tokens further amplifies the benefit of HTML. (2) Our error analysis suggests that higher-capability models exploit layout information in HTML for better action grounding, while lower-capability models suffer from increased hallucination under longer inputs. We also find that incorporating observation history improves performance across most models and settings, and a diff-based representation offers a token-efficient alternative. Based on these findings, we suggest practical guidelines: adaptively select observation representations based on model capability and thinking token budget, and incorporate observation history using diff-based representations.
Figures & tables
Figure 1: Relationship between observation representation and task success rate (WorkArena L1). The x-axis shows the average number of input tokens per step; the y-axis shows the task success rate. “h” denotes HTML and “a” denotes the accessibility tree (a11y). Arrows indicate the change from HTML to a11y within the same model. Left: For lower-capability (open-source) models, reducing the observation (h → a) improves the success rate. Right: For higher-capability (proprietary) models, the success rate decreases instead.
a11y
html
a11y+scrs
#input tokens
6720
56653
7446
Higher-Capability Models (Proprietary)
Claude Sonnet 4.6
52.4
67.0 ( +14.6 )
53.6 ( +1.2 )
gpt-5.1-2025-11-13 (high)
55.8
73.3 ( +17.5 )
59.7 ( +3.9 )
gpt-5.1-2025-11-13 (low)
49.1
57.9 ( +8.8 )
55.8 ( +6.7 )
gemini-2.5-flash (budget=16384)
45.5
56.7 ( +11.2 )
48.5 ( +3.0 )
Table 1: Task success rate (%) by observation representation and model, evaluated on Workarena L1. Numbers in parentheses indicate the difference from a11y. N/A indicates that the model does not support image input. Values in parentheses after model names indicate reasoning_effort (high/low) or thinking_budget . Input token counts are provided as reference values for gpt-5.1 (high) , including image inputs.
xml
indent
a11y
html
a11y
html
Higher-Capability Models (Proprietary)
gpt-5.1-2025-11-13 (high)
60.0
73.3 ( +13.3 )
55.8
76.7 ( +20.9 )
gemini-2.5-flash (budget=16384)
51.5
56.7 ( +5.2 )
45.5
48.2 ( +2.7 )
Lower-Capability Models (Open-Source)
gpt-oss-120b (high)
50.0
38.8 ( -11.2 )
46.7
26.7 ( -20.0 )
Table 2: Task success rate (%) by observation format (Workarena L1). Comparing xml format vs indent format for both a11y and html representations. For each format group, the a11y column shows the baseline success rate, and the html column shows the difference from a11y in parentheses.
Form
Filter
Sort
Dashboard
Navi
Knowledge
Catalog
#tasks
50
60
60
40
20
10
90
Higher-Capability Models (Proprietary)
Claude Sonnet 4.6
+1.2
+11.8
0.0
+1.5
0.0
0.0
0.0
gpt-5.1-2025-11-13 (high)
-1.8
+8.8
+6.7
+3.0
0.0
+0.3
+0.6
gpt-5.1-2025-11-13 (low)
-4.5
+7.0
+3.9
+2.7
0.0
0.0
-0.3
gemini-2.5-flash (budget=16384)
-1.8
+6.1
+5.5
+3.3
0.0
-0.9
-0.9
Table 3: Impact of switching from a11y to HTML on task success rate by category (WorkArena L1). Values show the contribution to overall benchmark score (%) when using HTML instead of a11y. Task categories: Form (record creation), Filter (list filtering), Sort (list sorting), Dashboard (chart reading), Navi (menu navigation), Knowledge (KB search), Catalog (service ordering).
Figure 2: Grounding error counts and their breakdown for a11y and HTML across models (WorkArena L1).
Type
Description
API Format Errors
Scroll
Invalid argument passed to scroll()
Click
Invalid argument passed to click()
Select
Invalid argument passed to select_option()
Element Availability Errors
Not Found
The specified element does not exist in the DOM
Table 4: Grounding error categories.
hist0
hist4
hist9 full/diff
n_input_tokens
6720
26184
39011 / 13670
Higher-Capability Models (Proprietary)
gpt-5.1-2025-11-13 (high)
55.8
58.8 ( +3.0 )
58.8 ( +3.0 ) / 58.8 ( +3.0 )
gpt-5.1-2025-11-13 (low)
49.1
53.0 ( +3.9 )
50.9 ( +1.8 ) / 53.3 ( +4.2 )
o3-mini-2025-01-31 (high)
39.7
43.0 ( +3.3 )
43.3 ( +3.6 ) / 46.1 ( +6.4 )
gemini-2.5-flash (budget=16384)
45.5
48.2 ( +2.7 )
50.0 ( +4.5 ) / 48.2 ( +2.7 )
Table 5: Task success rate (%) by observation history setting and model, evaluated on Workarena L1. Numbers in parentheses indicate the difference from hist0. hist0: no history; hist4/hist9: history from the past 4/9 steps. In the hist9 column, full denotes adding the complete observation history, and diff denotes adding only the diff from the previous step. Values in parentheses after model names indicate reasoning_effort (high/low) or thinking_budget . Input token counts are provided as reference values for gpt-5.1 (high) .
Figure 3: Relationship between action repetition rate and task success rate (WorkArena L1). The x-axis shows the rate of actions identical to the previous step; the y-axis shows the task success rate. The number on each data point indicates the length of observation history included as input.