Autonomous web agents, powered by Large Language Models (LLMs), have garnered significant attention for automating various web-based tasks with multi-step reasoning and decision-making capabilities. An open research question in the development of these agents lies in the format of the webpage input. Raw HTML source code, with its extensive and often irrelevant details, poses difficulties for LLMs with limited context windows. To address this challenge, we first reproduce baseline models such as GPT-3.5 and LLaMA-2-70B on the WebArena (Zhou et al., 2023) benchmark, identifying common failure modes. We then propose two retrieval strategies to filter out irrelevant context for LLM agents. We develop DeBERTa-based and T5-based models that rank HTML elements by their relevance to the task. We fine-tune them on Mind2Web trajectory data and transfer them to WebArena. Experiments show that our DeBERTa-based model improves the success rate of the LLaMA-2-70B LLM agent on WebArena from 1.97% to 2.96%. Moreover, we develop a zero-shot ColBERT-based retriever that is able to retrieve the ground-truth element with a recall of 0.52 on Mind2Web and 0.47 on WebArena.
Figures & tables
Figure 1: Observation Space for WebArena: screenshot (left), HTML DOM tree (middle), and accessibility tree (right). Figure reproduced from webarena .
Model
Site
Success Rate (SR) ( ↑ )
Overall SR ( ↑ )
gpt-3.5
gitlab
3.33%
6.65%
reddit
3.77%
shopping_admin
4.94%
map
9.17%
shopping
13.37%
gpt-3.5 + DeBERTa + k = 50%
gitlab
6.21%
5.84%
Table 1: Comparison of GPT 3.5 and LLaMA model success rates on WebArena sites.
Figure 2: We fine-tune a DeBERTa model on Mind2Web deng2023mind2web to classify whether each HTML element is relevant to the task or not. During inference, we rank the logits of the elements in descending order and keep the top- k elements.
Figure 3: DeBERTa on cleaned WebArena trajectories
Figure 4: ColBERT on cleaned WebArena trajectories
Figure 5: Validation results of the three models we tried on Mind2Web.
Figure 6: Site Level Breakdown of distributions
Figure 7: Task 36 screenshot (left), raw accessibility tree with 1594 characters (center), and filtered accessibility tree to keep 50% of elements, resulting in 658 characters (right). Filtering the accessibility tree to keep just the top-50% items allows the model to focus on the relevant parts of the context. The original LLaMA-2-70B fails on this task, while the model with the filtered tree succeeds.
Figure 8: Task 132 screenshot (left), raw accessibility tree with 4319 characters (center), and filtered accessibility tree to keep 50% of elements, resulting in 2138 characters (right). The original LLaMA-2-70B fails on this task, while the model with the filtered tree succeeds.
Figure 9: Task 247 screenshot (left), raw accessibility tree with 5131 characters (center), and filtered accessibility tree to keep 50% of elements, resulting in 2841 characters (right). The original LLaMA-2-70B fails on this task, while the model with the filtered tree succeeds.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Task 152, the observation before (left) and after filtering (right). The action predicted by GPT-3.5 before was type [599] [Schenley park] , which was repeated thrice, leading to early stopping. This is improved after filtering of the context window, enabling the model to arrive at the correct answer.
Large language model (LLM) web agents are increasingly used for web navigation but remain far from human reliability on realistic, long-horizon tasks. Existing evaluations focus primarily on end-to-end success, offering limited insight into where failures arise. We propose a hierarchical planning framework to analyze web agents across three layers (i.e., high-level planning, low-level execution, and replanning), enabling process-based evaluation of reasoning, grounding, and recovery. Our experiments show that structured Planning Domain Definition Language (PDDL) plans produce more concise and goal-directed strategies than natural language (NL) plans, but low-level execution remains the dominant bottleneck. These results indicate that improving perceptual grounding and adaptive control, not only high-level reasoning, is critical for achieving human-level reliability. This hierarchical perspective provides a principled foundation for diagnosing and advancing LLM web agents.
Mohamed Aghzal, Gregory J. Stein, Ziyu Yao
Department of Computer Science, George Mason University
Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io
Web agents based on large language models (LLMs) rely on observations of web pages -- commonly represented as HTML -- as the basis for identifying available actions and planning subsequent steps. Prior work has treated the verbosity of HTML as an obstacle to performance and adopted observation reduction as a standard practice. We revisit this trend and demonstrate that the optimal observation representation depends on model capability and thinking token budget: (1) compact observations (accessibility trees) are preferable for lower-capability models, while detailed observations (HTML) are advantageous for higher-capability models; moreover, increasing thinking tokens further amplifies the benefit of HTML. (2) Our error analysis suggests that higher-capability models exploit layout information in HTML for better action grounding, while lower-capability models suffer from increased hallucination under longer inputs. We also find that incorporating observation history improves performance across most models and settings, and a diff-based representation offers a token-efficient alternative. Based on these findings, we suggest practical guidelines: adaptively select observation representations based on model capability and thinking token budget, and incorporate observation history using diff-based representations.