Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Figures & tables
Figure 1: In this paper, we propose Ego2Web , a new benchmark introducing a novel web-agent task grounded in users’ real-world visual surroundings. The tasks span diverse domains, like e-commerce, media retrieval, knowledge lookup, and local/maps services. Given an egocentric video and an instruction, the agent must first perform spatio-temporal grounding to identify the relevant visual cue (e.g., the fourth snack picked up in the video), and then execute corresponding web actions based on both the grounded visual evidence and the instruction.
Figure 2: The workflow overview for agent action and evaluation in real-world, egocentric video perception. The agent operates using both egocentric video and textual instructions, and outputs a series of web actions, screenshots and a final response to the task. To enable automatic evaluation in a live, unconstrained web environment, we further introduce a new LLM-as-a-Judge framework tailored for this real-world visually grounded web task. Our LLM-as-a-Judge framework takes the instruction, action history, screenshots, and a final response, and compares them with the annotated visual evidence (video clip) to assess whether the task is successfully completed.
Benchmarks
Visual Grounding
Egocentric Video Perception
Web Task
Online Evaluation
EgoThink ( Cheng et al., 2024 )
✓
✓
✗
✗
EgoSchema ( Mangalam et al., 2023 )
✓
✓
✗
✗
WebArena ( Zhou et al., 2023 )
✗
✗
✓
✗
Mind2Web ( Zhou et al., 2023 )
✗
✗
✓
✗
VisualWebArena ( Koh et al., 2024 )
✓
✗
✓
✗
VideoWebArena ( Jang et al., 2024 )
✓
✗
✓
✗
Table 1: Benchmark comparison between Ego2Web and representative video reasoning and multimodal web-agent benchmarks. Unlike pure video reasoning datasets (e.g., EgoThink, EgoSchema) that focus on visual understanding alone, and prior web-agent benchmarks (e.g., WebVoyager, VisualWebArena) that emphasize online interaction with only web screenshot perception , Ego2Web uniquely connects real-world egocentric video with executable web tasks under an unconstrained online evaluation setting, forming a new testbed for multimodal agents grounded in the real-world visual perception.
Figure 3: Left: Overview of the semi-automatic data generation pipeline proposed in our Ego2Web . We first build video profiles via a frozen MLLM that converts video clips into structured captions, then prompt an LLM to automatically generate web task instructions. Human annotators are required to verify and refine generated tasks to ensure quality. Right: A detailed view of our proposed automatic evaluation method, Ego2WebJudge , for the egocentric video grounded web agent tasks.
Figure 4: (a) Task type distribution of Ego2Web , showing the high-level task category composition across major web platforms. (b) Fine-grained website domain distribution in Ego2Web (we omit websites whose count < 4 for visualization), highlighting coverage across e-commerce, media retrieval, knowledge bases, local/map services and others (indicated by color coding).
Figure 5: Some examples of Ego2Web , including sampled frames of an egocentric video and paired web task instructions. We annotate essential video perception (e.g. the mint toaster with brand name and the black backpack in the middle of the video) with red boxes.
Evaluation
Base MLLM
Claude 3.7
Claude 4.5
GPT-5.4
SeeAct
BU-GPT-4.1
BU-Gemini-3-Flash
Ego2WebJudge
Qwen3-VL-Flash
20.8
32.2
38.8
29.6
34.6
57.2
Gemini-2.5 Pro
17.8
24.8
23.6
25.2
34.6
48.2
GPT-4o
19.4
27.2
26.8
26.8
47.6
51.4
Human Eval
–
26.4
32.8
30.6
34.2
44.4
58.6
Table 2: Success Rate (SR) measured by human evaluation and Ego2WebJudge using different Multimodal LLMs.
Domains / Agents
Claude 3.7
Claude 4.5
GPT 5.4
SeeAct
BU-GPT-4.1
BU-Genimi-3-Flash
Avg. SR
E-Commerce
13.0
18.2
14.3
19.5
26.9
38.2
21.7
Media Retrieval
19.6
26.5
29.5
24.2
30.3
50.7
30.1
Knowledge Lookup
33.6
45.6
39.1
43.4
63.0
75.0
50.0
Local / Maps
6.4
12.9
29.0
19.3
22.5
48.3
23.1
Others
0.0
6.6
6.6
20.0
40.0
13.3
14.4
Total
17.8
24.8
23.6
25.2
34.6
48.2
29.0
Table 3: Fine-grained Success Rate (SR) per task domain across different models, evaluated by Ego2WebJudge with Gemini-2.5 Pro.
Auto Evaluation Methods
Base MLLM
Claude 3.7
Claude 4.5
GPT 5.4
SeeAct
BU-GPT-4.1
BU-Gemini-3-Flash
Avg. AR
WebVoyager ( He et al., 2024 )
Gemini-2.5-Pro
77.8
73.6
75.2
70.3
61.0
66.2
70.7
WebJudge ( Xue et al., 2025 )
82.2
75.8
79.2
76.0
69.2
74.2
76.1
Ego2WebJudge (Ours)
85.4
80.6
82.4
80.2
78.4
78.0
80.8
WebVoyager ( He et al., 2024 )
GPT-4o
77.0
76.2
78.6
75.4
72.6
68.4
74.7
WebJudge ( Xue et al., 2025 )
74.2
80.6
82.4
80.0
74.3
78.2
78.4
Ego2WebJudge (Ours)
86.0
84.2
84.0
85.6
83.6
80.4
84.0
Table 4: Agreement Rate (AR) between human evaluation and automatic evaluation methods across agents.
Raw Video
Detailed Caption
E-Commerce
Media Retrieval
Knowledge Lookup
Local / Maps
Others
Total
✗
✗
2.6
7.5
5.4
3.2
0.0
4.4
✗
✓
13.0
29.5
39.1
38.7
6.6
23.6
✓
✗
38.2
50.7
75.0
48.3
13.3
48.2
Table 5: Ablation studies on the impact of video perception in our Ego2Web task. We report the Successful Rate (SR). We use Gemini-3.1-Pro to generate structured and detailed captions to represent an egocentric video. The experiment is conducted with Browser-Use (Gemini-3-Flash) and evaluated with Ego2WebJudge (Gemini-2.5-Pro).
Figure 6: Visualization of a web agent (BU-Gemini-3-Flash) failure case. The agent is required to identify the second picked-up sauce from the egocentric video and retrieve its product page. The agent incorrectly identifies the target item due to temporal misunderstanding and fails to verify the required information on the webpage.
Mininglamp Technology · School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences