See and Remember: A Multimodal Agent for Web Traversal

Xinjun Wang; Shengyao Wang; Aimin Zhou; Hao Hao

cs.AIMar 3, 2026

See and Remember: A Multimodal Agent for Web Traversal

Authors: Xinjun Wang, Shengyao Wang, Aimin Zhou, Hao Hao

Organizations: Shanghai Institute of AI for Education, East China Normal University · Shanghai Innovation Institute

Abstract

Autonomous web navigation requires agents to perceive complex visual environments and maintain long-term context, yet current Large Language Model (LLM) based agents often struggle with spatial disorientation and navigation loops. In this paper, we propose generally applicable V-GEMS(Visual Grounding and Explicit Memory System), a robust multimodal agent architecture designed for precise and resilient web traversal. Our agent integrates visual grounding to resolve ambiguous interactive elements and introduces an explicit memory stack with state tracking. This dual mechanism allows the agent to maintain a structured map of its traversal path, enabling valid backtracking and preventing cyclical failures in deep navigation tasks. We also introduce an updatable dynamic benchmark to rigorously evaluate adaptability. Experiments show V-GEMS significantly dominates the WebWalker baseline, achieving a substantial 28.7% performance gain. Code is available at https://github.com/Vaultttttttttttt/V-GEMS.

Figures & tables

Figure 1: The framework of V-GEMS. Our system augments the dual-agent architecture (Explorer and Critic) with a specialized Tools suite consisting of a Counter, URL Stack, and US Calculator. The US Calculator adaptively scores page content to decide between LLM and VLM processing. The URL Stack enables stateful backtracking, while the Counter ensures arithmetic precision across multiple navigation steps. For a comprehensive visualization of the end-to-end execution workflow, please refer to Appendix B .

Figure 2: The framework of test dataset generation

Difficulty	Single-source	Multi-source
Easy	80	80
Medium	140	140
Hard	120	120
Total	340	340

Table 1: Dataset Distribution by Question Type and Difficulty

Panel of Figure 3 — Figure 3: Data distribution of the EverWebQA dataset across domains and languages.

Panel of Figure 4 — Figure 4: The comparison in type and domain.

Modules			Fine-grained Tasks (F1)						Cross-domain Tasks (F1)						Avg
VLM	URL	Count	Single-Source			Multi-Source			CN	EN	Game	Conf	Org	Edu
VLM	URL	Count	Easy	Med	Hard	Easy	Med	Hard	CN	EN	Game	Conf	Org	Edu
-	-	-	0.60	0.50	0.48	0.50	0.45	0.44	0.50	0.49	0.32	0.51	0.63	0.54	0.50
-	-	✓	0.63	0.52	0.48	0.49	0.46	0.46	0.51	0.50	0.34	0.51	0.63	0.55	0.51
-	✓	-	0.63	0.60	0.60	0.66	0.56	0.51	0.64	0.48	0.51	0.58	0.67	0.61	0.59
-	✓	✓	0.65	0.61	0.59	0.67	0.57	0.52	0.63	0.50	0.52	0.59	0.69	0.61	0.59

Table 2: Ablation Test Results. The Modules columns indicate enabled components. The results are categorized by task granularity and domain type.

Panel of Figure 5 — Figure 5: Three methods’ contribution in different task

Figure 6: Fine-grained error distribution across model variants. Our full system ( All ) significantly improves the correctness rate while suppressing severe failures like Refusal and Hallucination.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: A comprehensive workflow demonstration.

Figure 8: Operational flowchart of the Symbolic Counter.

Root Url	https://www.fiba.basketball/en
Question	Which team won the second place in the FIBA Asia Cup 2025?
Answer	China
Source Website	https://www.fiba.basketball/en/history/195-fiba-asia-cup
Website Information	(Visual Context: The 2025 row shows 1st: AUS, 2nd: CHN, 3rd: IRI)
Prediction (Hallucination)	"Japan won the second place." [Analysis]: The entity "Japan" is fabricated . It does not exist in the 2025 data row shown above.
Prediction (Totally Incorrect)	"Australia won the second place." [Analysis]: "Australia" exists in the row (as 1st place) but is logically misattributed to the 2nd place.

Appendix

Table 3: Case study comparing Hallucination and Totally Incorrect errors on the FIBA Asia Cup task.

Question	Which team won the FIBA Asia Cup 2025, and who was named the TISSOT MVP of the tournament?
Answer	Australia won the FIBA Asia Cup 2025, and Jaylin Galloway was named the TISSOT MVP.
Prediction	"Australia won the FIBA Asia Cup 2025." [Analysis]: The answer is factually correct but incomplete . The agent correctly processed the first intent (Winning Team) but completely ignored the second intent (MVP) of the compound query, demonstrating a failure in comprehensive information recall.

Appendix

Table 4: Case study of a Missing Key Info error on a multi-source query.

Question	When was the last update made to the PSLE Secondary 1 Posting information?
Answer	The last update to the PSLE Secondary 1 Posting information was made on 21 Jul 2023 .
Prediction (Imprecise)	"The last update to the PSLE Secondary 1 Posting information was made in Jul 2023." [Analysis]: The answer is factually consistent with the truth but fails to meet the required temporal granularity . While the source provides a specific timestamp (Day-Month-Year), the model generalizes it to a broader scope (Month-Year), resulting in a loss of information fidelity.

Appendix

Table 5: Case study of an Imprecise error.

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

WebNavigator: Global Web Navigation via Interaction Graph Retrieval
Mar 20, 2026Xuanwang Zhang, Yuteng Han, Jinnan Qi +5Web Agents Graph-Based Planning
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Fellou AI
MELLON - Multimodal Enhanced LLM for Online Navigation
Aug 10, 2026Ruiyu Li, Haoyang Cai, Zhitong Guo +1Multimodal Reasoning Vision-Language Alignment
Carnegie Mellon University, Pittsburgh, PA, United States.
Mango: Multi-Agent Web Navigation via Global-View Optimization
Apr 20, 2026Weixi Tong, Yifeng Di, Tianyi ZhangMulti-Agent Systems Web Agents
Purdue University

Mar 20, 2026J/K move · Enter open · S save

WebNavigator: Global Web Navigation via Interaction Graph Retrieval

Xuanwang Zhang, Yuteng Han, Jinnan Qi, Xinyu Liu+4

National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Fellou AI

Web Agents Graph-Based Planning

Open paper AskPDF arXiv

Despite significant advances in autonomous web navigation, current methods remain far from human-level performance in complex web environments. We argue that this limitation stems from Topological Blindness, where agents are forced to explore via trial-and-error without access to the global topological structure of the environment. To overcome this limitation, we introduce WebNavigator, which reframes web navigation from probabilistic exploration into deterministic retrieval and pathfinding. WebNavigator constructs Interaction Graphs via zero-token cost heuristic exploration offline and implements a Retrieve-Reason-Teleport workflow for global navigation online. WebNavigator achieves state-of-the-art performance on WebArena and OnlineMind2Web. On WebArena multi-site tasks, WebNavigator achieves a 72.9% success rate, more than doubling the performance of enterprise-level agents. This work reveals that Topological Blindness, rather than model reasoning capabilities alone, is an underestimated bottleneck in autonomous web navigation.

Abstract

Web Agents

Explore similar work

WebNavigator: Global Web Navigation via Interaction Graph Retrieval

MELLON - Multimodal Enhanced LLM for Online Navigation

Mango: Multi-Agent Web Navigation via Global-View Optimization