cs.LGDec 2, 2025

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization

Authors: Zhengcheng Wang, Zichuan Lin, Yijun Yang, Haobo Fu, Deheng Ye

Organizations: Tencent AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Existing Vision-Language Navigation (VLN) agents based on Large Vision-Language Models (LVLMs) often suffer from errors of perception, reasoning, and planning, which significantly hinder their navigation performance. To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed. First, to reduce hallucinations of the visual module of the VLN agent, a dual-view Visual Prompt (VP) technique is introduced, which can also improve the agent's understanding of current spatial states. Subsequently, a novel step-level Reinforcement Fine-Tuning (RFT) method, Step Reward Group Policy Optimization (SRGPO), is designed for post-training. In SRGPO, we first define verifiable process rewards for navigation, and then perform efficient step-level advantage estimation by randomly grouping navigation steps. SRGPO provides dense reward signals for the RFT of the VLN agent and enhances its planning capability. Experiments on both the EmbodiedBench Navigation (Embench-Nav) and VLN-CE validate the effectiveness of the proposed method. By introducing the zero-shot VP module, the GPT-4.1 achieves a navigation success rate of 86.7%, surpassing the current best LVLM by approximately 20 percentage points (pp) on Embench-Nav and achieves an 11.1 pp improvement on VLN-CE compared to the base model. With VP+SRGPO, the Qwen2.5-VL-3B model reaches a navigation success rate of 72.3%, outperforming the best existing LVLM model by 5.6 pp on Embench-Nav and achieves 8.9 pp improvement on VLN-CE compared to the baseline method. Moreover, compared to GRPO and GiGPO, the proposed SRGPO demonstrates significant improvements in training stability, convergence efficiency, and generalization capability.

Figures & tables

Figure 1: The key motivations. Left: Analysis for different error types from VLA models. Right: Examples of different error types in the navigation task.
Figure 2: Overview of SeeNav-Agent. Different from previous VLN works tend to use single view image as input and use method like GRPO or GiGPO for RFT, our SeeNav-Agent designs a dual-view input with visual prompt to enhance the visual module in a zero-shot manner, and proposes SRGPO to efficiently introduce process reward signals by randomly grouping steps. The upper-middle schematic shows the dual-view visual prompt, with FV on the left and BEV on the right.
Figure 3: i.d and o.o.d Curves on Embench-Nav. Left: i.d training curves, where the solid line and the shaded region represent the mean and standard deviation obtained from multiple training runs. right: o.o.d. validation curves, where all algorithms are trained on o.o.d. scenes and tested on base scenes of the Embench-Nav.
Figure 4: Case Comparison of vanilla-GPT 4.1 and GPT4.1 with VP on Embench-Nav. The bottom part shows the image and text feedback from the environment after taking actions. This case clearly shows that the dual-view VP technique reduces visual hallucination and strengthens the VLN agent’s spatial understanding.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

The action history: <action info> Step 0, action id 0, Move forward by 0.25. <env feedback> Last action MoveAhead executed successfully. <action info> Step 1, action id 0, Move forward by 0.25. <env feedback> Last action MoveAhead executed successfully. <action info> Step 2, action id 0, Move forward by 0.25. <env feedback> Last action MoveAhead executed successfully. <action info> Step 3, action id 2, Move rightward by 0.25. <env feedback> Last action MoveRight is invalid. Cube.668 is blocking Agent 0 from moving by (0.0000, 0.0000, 0.2500). <action info> Step 4, action id 3, Move leftward by 0.25. <env feedback> Last action MoveLeft is invalid. Cube.690 is blocking Agent 0 from moving by (0.0000, 0.0000, -0.2500).
Appendix
Table 4: An Example of the Interaction History.
<image> ## You are a robot operating in a home. You can do various tasks and output a sequence of actions to accomplish a given task with images of your status. Your input is a concatenation of your top-down view and the first-person view image, with the top-down image on the left and the first-person view on the right. The colored circle in the top-down view represents your current position, with the YELLOW side indicating your LEFT and the PURPLE side indicating your RIGHT. The GREEN arrow shows your current camera orientation. In the first person view and the overhead view, the red bounding box in both views highlights the object you need to navigate to. The candidate actions are shown in the overhead view and the first person view images, and each action is represented by a blue arrow, with the corresponding action ID at the end of the arrow. Now the human instruction is: {user_instruction} . ## The available action id (0 to 7) and action names are: action id 0: Move forward by 0.25,
action id 1: Move backward by 0.25,
action id 2: Move rightward by 0.25,
action id 3: Move leftward by 0.25,
action id 4: Rotate to the right by 90 degrees,
action id 5: Rotate to the left by 90 degrees,
action id 6: Tilt the camera upward by 30 degrees,
Appendix
Table 5: Language Prompt Template for SeeNav-Agent with Visual Prompts.
More in this topic

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

Explore similar work

CardsList
May 17, 2026

SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation

Vision-Language Navigation (VLN) approaches have currently followed two primary paradigms: the end-to-end Vision-Language Model (VLM) policy fine-tuned on navigation trajectories to directly predict actions, and the zero-shot modular pipeline integrating pre-trained Multimodal Large Language Model (MLLM) for training-free generalization to unseen environments. However, end-to-end methods struggle with long-horizon navigation and lack dynamic reasoning ability, whereas zero-shot methods are constrained by limited spatial grounding for reliable planning and require substantial reasoning time. To bridge this gap, we introduce SEDualVLN, a spatially-enhanced dual-system VLN framework. System 1 is a VLM model enhanced with both global and local spatial awareness, tailored for high-frequency action generation. System 2 integrates a general MLLM with a mapping module, wherein the MLLM performs low-frequency waypoint planning by leveraging top-down views of the real-time 3D map alongside streams of rendered path images. Both systems exploit complementary spatial representations to improve the agent's spatial understanding in VLN tasks. Ultimately, they cooperate to complete the navigation task through a fast-slow coordinated approach. SEDualVLN achieves state-of-the-art performance on VLN-CE benchmarks, and further ablation studies demonstrate the effectiveness of each system and module.

Jingzhi Huang, Junkai Huang, Wenxuan Song +5

Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)

Visual Spatial ReasoningVision-Language NavigationVision-Language GroundingVLM Reasoning
AskPDF
Mar 18, 2026

P3^{3}Nav: End-to-End Perception, Prediction and Planning for Vision-and-Language Navigation

In Vision-and-Language Navigation (VLN), an agent is required to plan a path to the target specified by the language instruction, using its visual observations. Consequently, prevailing VLN methods primarily focus on building powerful planners through visual-textual alignment. However, these approaches often bypass the imperative of comprehensive scene understanding prior to planning, leaving the agent with insufficient perception or prediction capabilities. Thus, we propose P3^{3}Nav, a novel end-to-end framework integrating perception, prediction, and planning in a unified pipeline to strengthen the VLN agent's scene understanding and boost navigation success. Specifically, P3^{3}Nav augments perception by extracting complementary cues from object-level and map-level perspectives. Subsequently, our P3^{3}Nav predicts waypoints to model the agent's potential future states, endowing the agent with intrinsic awareness of candidate positions during navigation. Conditioned on these future waypoints, P3^{3}Nav further forecasts semantic map cues, enabling proactive planning and reducing the strict reliance on purely historical context. Integrating these perceptual and predictive cues, a holistic planning module finally carries out the VLN tasks. Extensive experiments demonstrate that our P3^{3}Nav achieves new state-of-the-art performance on the REVERIE, R2R-CE, and RxR-CE benchmarks.

Tianfu Li, Wenbo Chen, Haoxuan Xu +2

HKUST(GZ), Guangzhou, China

Vision-Language Navigation
AskPDF
Mar 16, 2026

Trajectory-Diversity-Driven Robust Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires agents to navigate photo-realistic environments following natural language instructions. Current methods predominantly rely on imitation learning, which suffers from limited generalization and poor robustness to execution perturbations. We present NavGRPO, a reinforcement learning framework that learns goal-directed navigation policies through Group Relative Policy Optimization. By exploring diverse trajectories and optimizing via within-group performance comparisons, our method enables agents to distinguish effective strategies beyond expert paths without requiring additional value networks. Built on ScaleVLN, NavGRPO achieves superior robustness on R2R and REVERIE benchmarks with +3.0% and +1.71% SPL improvements in unseen environments. Under extreme early-stage perturbations, we demonstrate +14.89% SPL gain over the baseline, confirming that goal-directed RL training builds substantially more robust navigation policies. Code and models will be released.

Jiangyang Li, Cong Wan, SongLin Dong +4

Xi’an Jiaotong University · Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology

Vision-Language Navigation
AskPDF