cs.AIAug 6, 2025

Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents

Authors: Thassilo M. Schiepanski, Nicholas Piël

Organizations: Surfly BV

Abstract

The advent of large language models (LLMs) has sparked an evolution of autonomous web browsing agents: given a web browsing task and serialised user interface (UI) state, an LLM is expected to suggest input actions that incrementally solve the given task. The central challenge lies in serialising UI state for LLMs. Web agents have increasingly relied on grounded graphical UI (GUI) snapshots - screenshots augmented with visual cues - favoured for their modest input token footprint. Document object model (DOM) snapshots, serialised as HTML, represent a compelling alternative that leverages previously demonstrated HTML interpretation capabilities of LLMs. Their excessive token footprint, however, has precluded reliable deployment with web agents to date. We propose D2Snap, an algorithm to downsample the DOM, premised on preserving actionability and actionability-discriminating features. We evaluate D2Snap-downsampled DOM snapshots using a snapshot-variant web agent (GPT-4o) on a dataset sampled from Online-Mind2Web. Whilst 42% of raw DOM snapshots exceed the model context window (128 x 10^3 tokens), all D2Snap-downsampled DOM snapshots of our reference configuration fit, at a mean context utilisation of 16.5%. Against the 67% success rate of a grounded GUI snapshot baseline, our configuration attains 73% (+5.8%pt; 95% CI -13.6 to +26.0%pt; McNemar, p = 0.47), excluding a deficit (one-sided 95%) beyond 11%pt. Image input moreover appears to add little to snapshot utility; grounding text alone attains 62% (-5.8%pt; McNemar, p = 0.37).

Explore similar work

CardsList
Mar 5, 2026cs.AI

TimeWarp: Evaluating Web Agents by Revisiting the Past

As web agents close the gap with humans on benchmarks, one question arises: Do today's agents perform just as well on tomorrow's web? We introduce TimeWarp, a benchmark that emulates the evolving web. TimeWarp consists of three web environments, each with six UI versions spanning UI design, frontend code, and workflows from different eras of the internet. We pair TimeWarp with a set of complex, realistic tasks covering different forms of web navigation. Our experiments reveal that vision-based agents are vulnerable to changes, while text-based agents become brittle once fine-tuned on a single version. To address this, we propose TimeTraj, a new annotation method that uses plan distillation to collect trajectories across multiple versions. By training agents on teacher rollouts using our BC-variant, we achieve substantial performance gains: 20.4% to 37.7% for Qwen-3 4B and 0% to 27.0% for Llama-3.1 8B models. Our work helps study generalization across web designs and opens a new paradigm for collecting plans rather than trajectories to improve the robustness of web agents.
Md Farhan Ishmam, Kenneth Marino
Web AgentsAgentic BenchmarksWebUser Interface Design
AskPDF
May 28, 2026cs.CL

Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework

HTML observations in LLM-based web agents are extremely long, and while many reduction methods have been proposed, it remains unclear which methods reduce overall agent latency while maintaining performance. The main obstacle is the high cost of end-to-end evaluation: in our experiments, evaluating 11 methods across 32 configurations on 33 tasks of WorkArena L1 required 232.4 cumulative hours. To address this, we propose a lightweight evaluation framework based on the Minimal Failure Set (MFS), the minimal set of HTML elements whose removal causes task failure. We define coverage as the fraction of instances in which a reduction method fully retains the MFS, which serves as a proxy metric that requires neither web access nor LLM inference. We validate that coverage strongly correlates with end-to-end success rate, with over 100×\times speedup in cumulative evaluation time on both benchmarks. Using this framework, we find that extractive HTML reduction methods require either high computation cost or domain-specific optimization to reduce agent latency while maintaining performance. Building on this, we optimize a pruning program on MFS training data, achieving 2.2×\times faster per-step latency on WorkArena L1 while retaining 84% of the original success rate, and 3.1×\times faster on WebLinx while retaining 89%.
Masafumi Enomoto, Ryoma Obara, Haochen Zhang +1
Web AgentsLow-Latency SystemsLLM Inference Efficiency
AskPDF
May 8, 2026cs.CL

Region4Web: Rethinking Observation Space Granularity for Web Agents

Web agents perceive web pages through an observation space, yet its granularity has remained an underexamined design choice. Existing work treats observation at the same element-level granularity as the action space, leaving the page's functional organization implicit and forcing the agent to infer it from element-level signals at every step. We argue observation should instead operate at the granularity of functional regions, parts of the page that each serve a distinct purpose. We propose Region4Web, a framework that reorganizes the AXTree into functional regions through hierarchical decomposition and semantic abstraction, exposing the page's functional organization as the basis for page state understanding. Moreover, we propose PageDigest, a web-specific inference pipeline that delivers this region-level observation to the actor agent as a compact per-page digest that persists across steps. On the WebArena benchmark, PageDigest substantially reduces observation length while improving overall task success rate across diverse backbone large language models (LLMs) and established agent methods, regardless of backbone capacity. These results show that operating at the granularity of functional regions delivers a more compact and informative basis for the actor agent than element-level processing alone.
Donguk Kwon, Dongha Lee
Web AgentsGranularity
AskPDF