P$^{3}$Nav: End-to-End Perception, Prediction and Planning for Vision-and-Language Navigation

Tianfu Li; Wenbo Chen; Haoxuan Xu; Xinhu Zheng; Haoang Li

cs.ROMar 18, 2026

P $^{3}$ Nav: End-to-End Perception, Prediction and Planning for Vision-and-Language Navigation

Authors: Tianfu Li, Wenbo Chen, Haoxuan Xu, Xinhu Zheng, Haoang Li

Organizations: HKUST(GZ), Guangzhou, China

Abstract

In Vision-and-Language Navigation (VLN), an agent is required to plan a path to the target specified by the language instruction, using its visual observations. Consequently, prevailing VLN methods primarily focus on building powerful planners through visual-textual alignment. However, these approaches often bypass the imperative of comprehensive scene understanding prior to planning, leaving the agent with insufficient perception or prediction capabilities. Thus, we propose P $^{3}$ Nav, a novel end-to-end framework integrating perception, prediction, and planning in a unified pipeline to strengthen the VLN agent's scene understanding and boost navigation success. Specifically, P $^{3}$ Nav augments perception by extracting complementary cues from object-level and map-level perspectives. Subsequently, our P $^{3}$ Nav predicts waypoints to model the agent's potential future states, endowing the agent with intrinsic awareness of candidate positions during navigation. Conditioned on these future waypoints, P $^{3}$ Nav further forecasts semantic map cues, enabling proactive planning and reducing the strict reliance on purely historical context. Integrating these perceptual and predictive cues, a holistic planning module finally carries out the VLN tasks. Extensive experiments demonstrate that our P $^{3}$ Nav achieves new state-of-the-art performance on the REVERIE, R2R-CE, and RxR-CE benchmarks.

Figures & tables

Figure 1: Motivation of our method. (a) Scene information, such as objects, maps, and waypoints are crucial for grounding instructions to visual observations. (b.1) Early models are of a “planning-only” structure, with limited scene understanding due to implicit feature extraction and aggregation. (b.2) Recent methods build external perception/prediction modules, but suffer from information loss and error accumulation. (b.3) Our method unifies perception, prediction, and planning in a single network, through a unified environment representation with end-to-end feature propagation between each stage.

Figure 2: Pipeline of our P 3 Nav model. The agent first encodes its discrete observations into a BEV representation, upon which two perception decoders output object and map features, in parallel. Subsequently, two predictors sequentially decode waypoint cues and future scene semantics. Ultimately, a planning decoder incorporates the features from all the preceding modules to produce a comprehensive navigation decision.

Figure 3: Ground truth map semantics generation. First, we crop nearby regions from the point clouds with semantic annotations in the Matterport3D [ 8 ] dataset, and project them onto a plane. Second, we generate a template-based description of the map. Then we prompt the VLM to refine the description. The last token from the last VLM decoder layer is defined as the ground truth map semantics.

Figure 4: Pipeline of waypoint-level prediction. (a) Waypoint feature is decoded via the multi-attention deformable transformer decoder; after that, a heatmap is generated through up-sampling. (b) NMS and depth-based post-processing are performed to identify candidate waypoints, addressing the discrete-to-continuous transfer problem.

	Validation Unseen				Test Unseen
Methods	SR $\uparrow$	SPL $\uparrow$	RGS $\uparrow$	RGSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	RGS $\uparrow$	RGSPL $\uparrow$
HAMT [ 11 ]	32.95	30.20	18.92	17.28	30.40	26.67	14.88	13.08
DUET [ 12 ]	46.98	33.73	32.15	23.03	52.51	36.06	31.88	22.06
DSRG [ 60 ]	47.83	34.02	32.69	23.37	54.04	37.09	32.49	22.18
GridMM [ 66 ]	51.37	36.47	34.57	24.56	53.13	36.60	34.87	23.45
BEVBert [ 1 ]	51.78	36.37	34.71	24.44	52.81	36.41	32.06	22.09

Table 1: Comparison with SOTA methods on REVERIE dataset [ 45 ] . The best and second-best results are marked as bold and underline , respectively.

Methods	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
	R2R-CE				RxR-CE
CMA [ 19 ]	6.20	52	41	36	8.76	26.50	22.12	47.03	23.74
VLN $\circlearrowright$ BERT [ 19 ]	5.74	53	44	39	8.98	27.08	22.66	46.71	24.79
Reborn [ 3 ]	5.40	57	50	46	5.98	48.60	42.05	63.35	41.82
ETPNav [ 2 ]	4.71	65	57	49	5.64	54.79	44.89	61.90	45.33
BEVBert [ 1 ]	4.57	67	59	50	5.54	55.47	45.32	62.45	46.01

Table 2: Comparison with SOTA methods on Validation Unseen splits of R2R-CE [ 28 ] and RxR-CE [ 3 ] datasets. The best and second-best results are marked as bold and underline , respectively.

Figure 5: Analysis of (a) object-level perception (mAP and mAR at IoU=0.5) and (b) waypoint-level prediction. BTG denotes [ 19 ] , with BTG-B and BTG-U indicating its baseline version and U-Net [ 49 ] variants, respectively.

Figure 6: Case study in the simulator. (a) Comparison of navigation results: BEVBert [ 1 ] (left) fails to find the target, while our P 3 Nav (right) leverages perception and prediction information for accurate waypoint selection. (b) Visualization of object-level perception.

Figure 7: Case study in the real world. (a) Visualization of the path from the third-person view. (b) The robot’s first-person observations during navigation.

	REVERIE		R2R-CE		RxR-CE
Model Configs	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
Base Model	50.96	36.02	59.02	50.07	55.46	45.32	62.45	45.99
w/o Object Decoder	54.13	36.58	61.79	51.89	57.45	46.67	63.38	47.53
w/o Map Decoder	53.85	36.53	60.83	50.94	56.89	46.45	63.41	47.46
w/o Waypoint Decoder	55.04	36.67	61.87	52.09	57.47	47.71	63.78	47.85
w/o Scene Decoder	53.68	36.17	61.22	51.45	57.14	47.10	62.98	47.06

Table 3: Ablation study of intermediate modules. When ablating the waypoint decoder, we employ an external waypoint predictor [ 19 ] for R2R-CE [ 28 ] and RxR-CE [ 3 ] .

	REVERIE		R2R-CE		RxR-CE
Scale	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
11 $\times$ 11	54.91	36.52	61.78	51.99	57.61	47.81	63.97	47.85
15 $\times$ 15	55.98	36.78	62.17	52.32	58.01	47.92	64.29	48.04
21 $\times$ 21	55.73	36.69	62.01	52.18	57.95	47.88	64.22	47.99

Table 4: Comparison of different BEV scales. Odd scales are adopted to guarantee that the agent resides in the center cell, with the cell size fixed at

0.5

	REVERIE		R2R-CE		RxR-CE
Design	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
Modular	54.15	36.29	61.73	51.81	57.42	47.57	63.59	47.90
End-to-End	55.98	36.78	62.17	52.32	58.01	47.92	64.29	48.04

Table 5: Comparison of modular and end-to-end designs for integrating perception and prediction intermediate modules in the navigation pipeline.

	REVERIE		R2R-CE		RxR-CE
Methods	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
$\mathcal{V}_{m}$	53.82	36.51	61.32	51.44	57.02	47.13	63.53	47.66
$\mathcal{T}$	54.46	36.59	61.41	51.56	57.21	47.25	63.60	47.72
$\mathcal{V}_{e}$	55.43	36.70	61.96	52.02	57.88	47.79	64.11	47.89
$\mathcal{V}_{i}$	55.98	36.78	62.17	52.32	58.01	47.92	64.29	48.04

Table 6: Comparison of different ground truth generation methods for map semantics.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Pipeline of our planning module. Immediate scene grounding evaluates local perceptual feasibility using BEV features and scene semantics. Prospective future evaluation reasons over candidate waypoint graphs to estimate future consistency with the instruction. Global memory correction incorporates historical information through a global graph representation. The resulting scores are progressively integrated through two fusion layers to produce the final navigation score.

Methods	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	nDTW $\uparrow$	SDTW $\uparrow$
	R2R			RxR
HAMT [ 11 ]	3.65	66	61	56.5	63.1	48.3
HOP [ 47 ]	3.80	64	57	42.3	52.0	33.0
HOP+ [ 48 ]	3.49	67	61	45.7	52.0	36.0
EnvEdit [ 30 ]	3.24	69	64	62.8	68.5	54.6
BEVBert [ 1 ]	2.81	75	64	68.5	69.6	58.6

Appendix

Table 7: Comparison with SOTA methods on Validation Unseen splits of R2R [ 4 ] and RxR [ 29 ] datasets. The best and second-best results are marked as bold and underline , respectively.

Figure 9: First case study in the simulation. The image sequence shows the robot’s panoramic observations during navigation.

Figure 10: Second case study in the simulation. The image sequence shows the robot’s panoramic observations during navigation.

Figure 11: First case study in the real world. (a) Visualization of the path from the third-person view. (b) The robot’s first-person observations during navigation.

Figure 12: Second case study in the real world. (a) Visualization of the path from the third-person view. (b) The robot’s first-person observations during navigation.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 308 papers

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
May 21, 2026Wenxuan Guo, Xiuwei Xu, Yichen Liu +7Vision-Language Navigation VLM Reasoning
Tsinghua University, China
SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation
May 17, 2026Jingzhi Huang, Junkai Huang, Wenxuan Song +5Visual Spatial Reasoning Vision-Language Navigation
Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)
AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University

May 21, 2026J/K move · Enter open · S save

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

Wenxuan Guo, Xiuwei Xu, Yichen Liu, Xiangyu Li+6

Tsinghua University, China

Vision-Language Navigation VLM Reasoning Embodied AI Spatial Reasoning Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment. While state-of-the-art methods leverage the reasoning capabilities of Vision-Language Models (VLMs) for end-to-end action prediction, they often lack an explicit and explainable understanding of the relationships between the agent, the instruction, and the scene. Conversely, explicitly building a scene map for heuristic planning is intuitively appealing but relies on additional 3D sensors and hinders large-scale vision-language pre-training. To bridge this gap, we propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism, enabling it to understand the agent's state and task progress in a fully end-to-end and data-driven manner. Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training. Extensive experiments on various datasets in Habitat simulator show our AwareVLN significantly outperforms previous state-of-the-art vision-language navigation methods. Project page: https://gwxuan.github.io/AwareVLN/.

Abstract

Vision-Language Navigation

Explore similar work

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation

AgentVLN: Towards Agentic Vision-and-Language Navigation