Autonomous web navigation requires agents to perceive complex visual environments and maintain long-term context, yet current Large Language Model (LLM) based agents often struggle with spatial disorientation and navigation loops. In this paper, we propose generally applicable V-GEMS(Visual Grounding and Explicit Memory System), a robust multimodal agent architecture designed for precise and resilient web traversal. Our agent integrates visual grounding to resolve ambiguous interactive elements and introduces an explicit memory stack with state tracking. This dual mechanism allows the agent to maintain a structured map of its traversal path, enabling valid backtracking and preventing cyclical failures in deep navigation tasks. We also introduce an updatable dynamic benchmark to rigorously evaluate adaptability. Experiments show V-GEMS significantly dominates the WebWalker baseline, achieving a substantial 28.7% performance gain. Code is available at https://github.com/Vaultttttttttttt/V-GEMS.
Figures & tables
Figure 1: The framework of V-GEMS. Our system augments the dual-agent architecture (Explorer and Critic) with a specialized Tools suite consisting of a Counter, URL Stack, and US Calculator. The US Calculator adaptively scores page content to decide between LLM and VLM processing. The URL Stack enables stateful backtracking, while the Counter ensures arithmetic precision across multiple navigation steps. For a comprehensive visualization of the end-to-end execution workflow, please refer to Appendix B .
Figure 2: The framework of test dataset generation
Difficulty
Single-source
Multi-source
Easy
80
80
Medium
140
140
Hard
120
120
Total
340
340
Table 1: Dataset Distribution by Question Type and Difficulty
Figure 3: Data distribution of the EverWebQA dataset across domains and languages.
Figure 4: The comparison in type and domain.
Modules
Fine-grained Tasks (F1)
Cross-domain Tasks (F1)
Avg
VLM
URL
Count
Single-Source
Multi-Source
CN
EN
Game
Conf
Org
Edu
Easy
Med
Hard
Easy
Med
Hard
-
-
-
0.60
0.50
0.48
0.50
0.45
0.44
0.50
0.49
0.32
0.51
0.63
0.54
0.50
-
-
✓
0.63
0.52
0.48
0.49
0.46
0.46
0.51
0.50
0.34
0.51
0.63
0.55
0.51
-
✓
-
0.63
0.60
0.60
0.66
0.56
0.51
0.64
0.48
0.51
0.58
0.67
0.61
0.59
-
✓
✓
0.65
0.61
0.59
0.67
0.57
0.52
0.63
0.50
0.52
0.59
0.69
0.61
0.59
Table 2: Ablation Test Results. The Modules columns indicate enabled components. The results are categorized by task granularity and domain type.
Figure 5: Three methods’ contribution in different task
Figure 6: Fine-grained error distribution across model variants. Our full system ( All ) significantly improves the correctness rate while suppressing severe failures like Refusal and Hallucination.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A comprehensive workflow demonstration.
Figure 8: Operational flowchart of the Symbolic Counter.
Root Url
https://www.fiba.basketball/en
Question
Which team won the second place in the FIBA Asia Cup 2025?
(Visual Context: The 2025 row shows 1st: AUS, 2nd: CHN, 3rd: IRI)
Prediction (Hallucination)
"Japan won the second place." [Analysis]: The entity "Japan" is fabricated . It does not exist in the 2025 data row shown above.
Prediction (Totally Incorrect)
"Australia won the second place." [Analysis]: "Australia" exists in the row (as 1st place) but is logically misattributed to the 2nd place.
Appendix
Table 3: Case study comparing Hallucination and Totally Incorrect errors on the FIBA Asia Cup task.
Question
Which team won the FIBA Asia Cup 2025, and who was named the TISSOT MVP of the tournament?
Answer
Australia won the FIBA Asia Cup 2025, and Jaylin Galloway was named the TISSOT MVP.
Prediction
"Australia won the FIBA Asia Cup 2025." [Analysis]: The answer is factually correct but incomplete . The agent correctly processed the first intent (Winning Team) but completely ignored the second intent (MVP) of the compound query, demonstrating a failure in comprehensive information recall.
Appendix
Table 4: Case study of a Missing Key Info error on a multi-source query.
Question
When was the last update made to the PSLE Secondary 1 Posting information?
Answer
The last update to the PSLE Secondary 1 Posting information was made on 21 Jul 2023 .
Prediction (Imprecise)
"The last update to the PSLE Secondary 1 Posting information was made in Jul 2023." [Analysis]: The answer is factually consistent with the truth but fails to meet the required temporal granularity . While the source provides a specific timestamp (Day-Month-Year), the model generalizes it to a broader scope (Month-Year), resulting in a loss of information fidelity.
Despite significant advances in autonomous web navigation, current methods remain far from human-level performance in complex web environments. We argue that this limitation stems from Topological Blindness, where agents are forced to explore via trial-and-error without access to the global topological structure of the environment. To overcome this limitation, we introduce WebNavigator, which reframes web navigation from probabilistic exploration into deterministic retrieval and pathfinding. WebNavigator constructs Interaction Graphs via zero-token cost heuristic exploration offline and implements a Retrieve-Reason-Teleport workflow for global navigation online. WebNavigator achieves state-of-the-art performance on WebArena and OnlineMind2Web. On WebArena multi-site tasks, WebNavigator achieves a 72.9% success rate, more than doubling the performance of enterprise-level agents. This work reveals that Topological Blindness, rather than model reasoning capabilities alone, is an underestimated bottleneck in autonomous web navigation.
Xuanwang Zhang, Yuteng Han, Jinnan Qi +5
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Fellou AI
Web navigation agents are capable of addressing various types of tasks on different websites. Current baselines on web navigation are either unimodal or lack strong reasoning abilities given multimodal inputs. Focusing on the WebShop benchmark, a real-world website simulation, we explore the alignment of text and images, as well as multimodal reasoning and planning abilities, to enhance the performance of web navigation agents. We propose three innovative multimodal enhancements: Multimodal Enhanced LLM for Online Navigation (MELLON), VQAgent, and Multimodal Ranker. MELLON demonstrates a significant improvement in task completion accuracy, with a 9.26% increase after just one epoch of training. Our findings suggest the necessity of further exploration into multimodal approaches, with a focus on more extensive training and alignment strategies to enhance the effectiveness of web navigation agents.
Ruiyu Li, Haoyang Cai, Zhitong Guo +1
Carnegie Mellon University, Pittsburgh, PA, United States.
Existing web agents typically initiate exploration from the root URL, which is inefficient for complex websites with deep hierarchical structures. Without a global view of the website's structure, agents frequently fall into navigation traps, explore irrelevant branches, or fail to reach target information within a limited budget. We propose Mango, a multi-agent web navigation method that leverages the website structure to dynamically determine optimal starting points. We formulate URL selection as a multi-armed bandit problem and employ Thompson Sampling to adaptively allocate the navigation budget across candidate URLs. Furthermore, we introduce an episodic memory component to store navigation history, enabling the agent to learn from previous attempts. Experiments on WebVoyager demonstrate that Mango achieves a success rate of 63.6% when using GPT-5-mini, outperforming the best baseline by 7.3%. Furthermore, on WebWalkerQA, Mango attains a 52.5% success rate, surpassing the best baseline by 26.8%. We also demonstrate the generalizability of Mango using both open-source and closed-source models as backbones. Our data and code are open-source and available at https://github.com/VichyTong/Mango.