Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress

Jiagan Huang; Zikun Zhou; Hongpeng Wang; Guangming Lu; Jun Yu; Wenjie Pei

cs.CVOct 9, 2026

Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress

Authors: Jiagan Huang, Zikun Zhou, Hongpeng Wang, Guangming Lu, Jun Yu, Wenjie Pei

Organizations: Harbin Institute of Technology, Shenzhen

Abstract

In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point. Most graph-based methods commonly match candidate trajectories with the entire instruction, allowing already executed content to interfere with subsequent decisions. Although instruction-execution progress provides a direct cue for separating executed and pending content, existing progress estimation approaches often rely on explicit progress supervision, sophisticated progress-learning stages, or costly online reasoning with LLMs. In this paper, we propose a robust VLN framework with Awareness of Instruction-Execution Progress (VLN-AIEP), which couples progress estimation with navigation decision-making. VLN-AIEP associates the visited trajectory with the instruction to suppress embeddings corresponding to executed content, and then matches candidate locations with the pending content. This coupling makes correct navigation decisions depend on accurate progress estimation, allowing the progress estimator to learn directly from navigation decision labels without explicit progress annotations. To support progress estimation, we construct a landmark-anchored trajectory representation that encodes the semantic labels and viewpoint-relative geometry of instruction-referred landmarks. These landmarks serve as spatial references for associating the visited trajectory with executed instruction content. Extensive experiments on R2R, RxR, CVDN, and R2R-CE demonstrate that the proposed method achieves favorable performance across multiple metrics.

Figures & tables

Fig. 1: Illustration of our motivation. (a) Existing methods typically perform global coarse matching between candidate trajectories and the entire instruction. The agent is susceptible to interference from previously executed portions of the instruction, such as “turn left” in this case, leading to an incorrect decision. (b) Our method first estimates navigation progress by associating the visited trajectory with the instruction, where instruction-referred landmarks ( e.g. , “French doors”) serve as spatial references with semantic and spatial cues. It then performs fine-grained matching with the pending instruction parts, thereby avoiding interference from executed instructions.

Fig. 2: Overview of our method, which is based on a directed graph and comprises three components: (a) the landmark-anchored trajectory representation, (b) the progress-aware fine-grained matching module, and (c) the navigation decision-making module. Our method identifies the detour-free path from the initial node to the current node and then extracts the landmark-anchored trajectory representation of this path. With the historical trajectory embeddings, our method first estimates the navigation progress and then performs fine-grained matching to obtain the correlation embeddings for the new candidate locations. Finally, it makes a decision based on the correlation embeddings of both the new and historical candidate locations, as well as a stop embedding.

Fig. 3: Structure of the navigation progress estimator. It employs the instruction-to-trajectory (I2T) and trajectory-to-instruction (T2I) attentions to suppress the instruction embeddings that resemble the trajectory embeddings, yielding progress-aware instruction embeddings that indicate instruction-execution progress.

Methods	Publication	Progress awareness	Val Seen				Val Unseen				Test Unseen
Methods	Publication	Progress awareness	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$
Seq2Seq-SF [ 1 ]	CVPR’18	No	11.33	6.01	39	-	8.39	7.81	22	-	8.13	7.85	28	18
Self-Monitoring [ 9 ]	ICLR’19	Yes	-	3.22	67	58	-	5.52	45	32	-	5.99	43	32
Regretful [ 10 ]	CVPR’19	Yes	-	3.23	69	63	-	5.32	50	41	-	5.69	56	40
HAMT [ 26 ]	NIPS’21	No	11.15	2.51	76	72	11.46	2.29	66	61	12.27	3.93	65	60
DUET [ 6 ]	CVPR’22	No	12.32	2.28	79	73	13.94	3.31	72	60	14.73	3.65	69	59

TABLE I: Experimental results of VLN methods on R2R [ 1 ] dataset.

\downarrow

and

\uparrow

indicate that lower/higher values are preferred, respectively. Methods marked with ⧫ are based on large models. “Progress awareness” indicates whether navigation progress is estimated.

Methods	Progress awareness	Val Seen			Val Unseen
Methods	Progress awareness	NE $\downarrow$	SR $\uparrow$	sDTW $\uparrow$	NE $\downarrow$	SR $\uparrow$	sDTW $\uparrow$
LSTM [ 18 ]	No	10.7	25.2	20.7	10.9	22.8	18.2
HAMT [ 26 ]	No	-	59.4	50.9	-	56.5	48.3
MARVEL [ 51 ]	No	3.0	75.9	68.8	4.5	64.8	57.5
BEVBert [ 8 ]	No	3.2	75.0	66.7	4.0	68.5	58.6
PRET [ 7 ]	No	2.4	79.3	70.7	3.2	72.8	62.4

TABLE II: Experimental results of the VLN methods on RxR [ 18 ] dataset. “Progress awareness” indicates whether navigation progress is estimated.

Methods	Progress awareness	Val Seen	Val Unseen	Test Unseen	Params
HAMT [ 26 ]	No	6.91	5.13	5.58	-
VLN-PETL [ 52 ]	No	7.76	5.69	6.13	-
NaviLLM ⧫ [ 53 ]	No	-	6.16	7.90	$\sim 7.4B$
VLN-AIEP(Ours)	Yes	8.06	6.44	6.90	$\mathbf{<}0.5B$

TABLE III: Goal Progress (GP,

\uparrow

) of the VLN methods on CVDN [ 19 ] dataset. Methods marked with ⧫ are based on large models. “Progress awareness” indicates whether navigation progress is estimated.

Methods	Publication	Progress awareness	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Seq2Seq [ 20 ]	ECCV’20	No	7.37	40	32	30
BEVBert [ 8 ]	ICCV’23	No	4.57	67	59	50
ETPNav [ 29 ]	TPAMI’24	No	4.71	65	57	49
Open-Nav ⧫ [ 17 ]	ICRA’25	Yes	6.70	23	19	16
Fast-SmartWay ⧫ [ 42 ]	ICRA’26	Yes	7.72	-	28	25
StreamVLN ⧫ [ 36 ]	ICRA’26	No	4.98	64	57	52

TABLE IV: Experimental results on val-unseen split of R2R-CE [ 20 ] dataset. Methods marked with ⧫ are based on large models. “Progress awareness” indicates whether navigation progress is estimated.

Trajectory representation	PAM	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$
1) Panorama features	×	15.83	3.74	67.39	56.45
2) Panorama & OA features	×	13.68	3.12	72.92	62.44
3) Landmark-anchored panorama & OA features	×	13.31	2.88	74.15	64.05
4) Panorama features	✓	13.91	3.32	69.47	60.02
5) Panorama & OA features	✓	13.44	2.85	74.02	63.82
6) Landmark-anchored panorama & OA features	✓	13.21	2.76	76.40	65.73

TABLE V: Ablation study on the core designs of VLN-AIEP. PAM denotes progress-aware fine-grained matching; OA denotes orientation-aware.

Variants	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$
1) $w/o$ I2T attention	12.71	3.04	72.62	63.30
2) $w/o$ T2I attention	12.41	2.85	73.81	64.92
3) full model (I2T+T2I)	13.21	2.76	76.40	65.73

TABLE VI: Ablation study on combining I2T and T2I attention.

Variants	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$
1) $w/o$ landmark-anchored MLM	12.64	3.02	72.32	63.77
2) $w/$ landmark-anchored MLM	13.21	2.76	76.40	65.73

TABLE VII: Ablation study on landmark-anchored MLM.

Variants	TL	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$
1) $w/o$ rethink	13.21	2.76	76.40	65.73
2) $w/$ rethink	13.28	2.74	76.81	65.85

TABLE VIII: Ablation study on rethinking strategy for termination decision.

Fig. 4: An R2R example of instruction-execution progress estimation. The left panel shows the navigation trajectory. The middle panel shows observations at key steps 1, 3, and 5. Landmark labels identify spatial references mentioned in the instruction. The right panel shows the average pending score for each of four sub-instructions. Each score decreases after the corresponding sub-instruction is executed. This indicates that VLN-AIEP accurately perceives instruction-execution progress. For visualization, the four sentences are treated as four sub-instructions. The model does not use this division during training.

Fig. 5: An R2R example of instruction-execution progress estimation. The left panel shows the navigation trajectory, with the instruction divided into three sub-instructions for visualization. The right panel shows their average pending scores. Each score decreases after the corresponding sub-instruction is executed, indicating accurate execution progress estimation.

Fig. 6: Navigation trajectories of PRET [ 7 ] and VLN-AIEP under the same instruction. At the key step, the pending instruction requires a left turn beside the photos landmark. PRET remains influenced by the executed phrase, turns right again, and stops incorrectly at the red node. VLN-AIEP uses photos as a spatial reference, focuses on the pending instruction, turns left into the target bedroom, and terminates correctly at the yellow node.

Fig. 7: Navigation trajectories of PRET [ 7 ] and VLN-AIEP under the same instruction. At the key step, both agents have completed the required left turn and should walk straight down the hallway . PRET remains influenced by the executed instruction and turns left again, resulting in the incorrect trajectory shown in red. VLN-AIEP focuses on the pending instruction and walks down the hallway, resulting in the correct trajectory shown in yellow.

w/o PAM	w/o PAM (perturbed)	w/ PAM	w/ PAM (perturbed)
74.2	72.3 (-1.9)	76.4	75.5 (-0.9)

TABLE IX: SR on val-unseen split of R2R [ 1 ] with and without PAM under perturbation.

Fig. 8: A representative case showing how PAM supports recovery from a manually induced erroneous deviation. After the agent is forced to move from node 2 to node 3, the agent still focuses on the pending sub-instruction “walk straight following cabinet”. Since no cabinet is observed around node 3, the agent backtracks to historical candidate node 4, which is aligned with the pending instruction at node 2. Notably, after backtracking, the historical trajectory is corrected to the detour-free path 2 $\rightarrow$ 4, rather than 2 $\rightarrow$ 3 $\rightarrow$ 2 $\rightarrow$ 4, for subsequent decision-making.

Fig. 9: A representative visualization example showing the effect of landmarks as spatial references for progress estimation. Compared with the setting without landmarks, incorporating landmarks produces a clearer boundary between executed and pending sub-instructions, indicating more accurate progress perception.

Explore similar work

From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation
Aug 4, 2026Xiangyun Huang, Xiangchen Wang, Runfeng Lin +5Vision-Language Navigation VLMs for Robotics
1Beihang University · 2Southern University of Science and Technology · 3Central South University +2
GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance
Oct 9, 2026Jingyu Guo, Jiaxin Huang, Chengxing Lin +6Vision-Language Navigation Visual Navigation
University of Melbourne · AgiBot · Mohamed bin Zayed University of Artificial Intelligence +3
StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation
Oct 5, 2026Anh Dao, Quan-Dung Pham, Le Danh Vinh +6Vision-Language Navigation
VinMotion, Inc., Vietnam · University of Southern California, USA

Aug 4, 2026J/K move · Enter open · S save

From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation

Xiangyun Huang, Xiangchen Wang, Runfeng Lin, Yihao Xu+4

1Beihang University · 2Southern University of Science and Technology · 3Central South University+2

Vision-Language Navigation VLMs for Robotics

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) requires an agent to follow a route-level instruction by executing its constituent steps from egocentric visual observations. Existing VLM-based navigators typically supervise both capabilities through next-action prediction alone, making progress-tracking errors difficult to distinguish from execution errors. When an agent deviates from the route, a corrective action label may recover the next movement but does not indicate whether the agent selected the wrong sub-instruction or failed to execute the correct one. Consequently, the agent may continue making decisions from an erroneous progress state. To resolve this ambiguity, we propose Route2Step, a framework that decouples semantic progress tracking from action generation through an explicit step-level interface. The Instruction Analysis Module (

\mathcal{M}_{\mathrm{IA}}

) predicts this state from the global instruction and visual history. Conditioned on the predicted state and recent observations, the Action Generation Module (

\mathcal{M}_{\mathrm{AG}}

) generates local action chunks. To supervise the progress state without manual temporal labels, E-SPA, a step-alignment procedure, associates sub-instructions with their corresponding portions of route-level demonstrations. These alignments enable state supervision for incorrect progress estimates, while direct action supervision is reserved for rollout groups that repeatedly fail under the correct active sub-instruction. On R2R-CE, Route2Step improves SR from 48.1% to 55.3% and SPL from 43.3% to 48.2%, using 190K state-level corrective samples while requiring only 11.5K directly action-supervised states. Experiments in real-world indoor and outdoor environments further demonstrate the practical applicability of Route2Step. The project page is: https://sisyphus-hxy.github.io/Route2Step/.

Abstract

Explore similar work

From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation

GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance

StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation