As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important for reliable deployment. Existing explainability methods overlook a key challenge in Vision-Language Navigation (VLN): evidence for the current action may be distributed across historical and current observations and associated with different instruction stages. We identify instruction-stage insensitivity, where existing methods produce similar explanations despite changes across different navigation stages. To address this issue, we propose a stage-aware attribution framework for VLN. We first construct action-conditioned temporal instruction--visual attributions across observations. These attributions are projected into instruction space to estimate stage associations, which guide the aggregation of relevant historical evidence for refining the current explanation. An instruction-guided spatial filter further suppresses isolated responses while preserving compact, instruction-relevant regions. Experiments across multiple VLN policies and benchmarks show improved attribution quality over existing methods. We further demonstrate that our method acts as a diagnostic tool for analyzing failure cases, and validate its applicability to real-world navigation on a Unitree G1 robot.
Figures & tables
Fig. 1 : Visual explanation challenges in VLN. (a) Explaining a generated token involves localizing its visual evidence while reducing interference from preceding tokens. (b) Explaining a navigation action requires recovering temporally distributed evidence across historical and current observations, while suppressing cross-stage interference from evidence associated with other instruction stages. Colored boxes schematically indicate stage-related regions. (c) Existing methods [ 14 ] are insensitive to instruction-stage transitions, exhibiting overlapping token attributions ( top ) and high cross-stage explanation similarity ( bottom ).
Fig. 2 : Overview of our framework. (a) We construct action-conditioned temporal instruction–visual attributions across historical and current observations. (b) We project each attribution into instruction space to estimate its associated instruction stage. (c) We weight historical observations by their stage correspondence with the current view and aggregate the resulting support to refine the current attribution. (d) We further use instruction–patch associations to suppress spatial noise while preserving decision-relevant visual evidence.
Fig. 3 : Qualitative comparison with state-of-the-art methods across navigation stages. The same instruction is used throughout navigation, while the highlighted tokens are automatically induced by the action-conditioned instruction attribution at each decision step. For Steps 1, 7, and 21, we visualize early-history, recent-history, and current observations together with their attribution maps. Compared with existing methods, our explanations better follow the evolving instruction stage while retaining relevant evidence across historical and current views.
Dataset
Method
Type
Δ NLL(%) ↑
Precision (%) ↑
F1-Score(%) ↑
R2R
DEX-AR [ 14 ]
Gradient
14.54
9.12
11.21
GradCAM [ 11 ]
Gradient
17.66
35.91
23.68
TAM [ 13 ]
Logit
14.92
7.13
9.65
Ours
Gradient
31.98
43.70
36.93
RxR
DEX-AR [ 14 ]
Gradient
21.13
7.04
10.56
GradCAM [ 11 ]
Gradient
27.83
28.62
28.22
TABLE I : Quantitative comparison on R2R and RxR using NaVILA under the VLN-CE setting. We report decision faithfulness ( ΔNLLcorr ), semantic alignment (Precision), and F1-Score. Higher is better for all metrics.
Dataset
VLN Policy
Ours over DEX-AR [ 14 ]
Δ NLL (%) ↑
Prec. (%) ↑
F1 (%) ↑
R2R
NaVid [ 6 ]
+14.23
+18.23
+16.50
NaVILA [ 5 ]
+17.44
+34.58
+25.72
StereoNav [ 33 ]
+5.16
+34.31
+9.17
RxR
NaVid [ 6 ]
+11.86
+37.50
+22.58
NaVILA [ 5 ]
+7.33
+35.37
+23.50
TABLE II : Performance gains over DEX-AR across different VLN policies. All values denote improvements.
Fig. 4 : Effect of TR. Comparisons before and after TR show that stage-guided historical support reduces cross-stage interference and yields more focused attribution on evidence relevant to the current navigation decision. Red boxes highlight the irrelevant regions.
Fig. 5 : Effect of SF. Visualization of the base rank filter [ 13 ] and our uni-/bidirectional SF variants (Section III-E ). Green and red boxes denote instruction-relevant and irrelevant regions, respectively.
TR
SF
ΔNLL(%)↑
Precision (%) ↑
F1-Score (%) ↑
×
×
26.61
41.80
32.52
✓
×
27.81
43.28
33.86
×
✓
30.82
41.93
35.53
✓
✓
31.98
43.70
36.93
TABLE III : Ablation of temporal refinement (TR, Sec. III-D ) and spatial filtering (SF, Sec. III-E ) on R2R.
Fig. 6 : Quantitative analysis of instruction-stage insensitivity. Top: token-level instruction attribution and frame-level similarity. Bottom: dataset-level comparison with the baseline, i.e. , TAM [ 13 ] . Our method shows stronger sensitivity to instruction-stage transitions and more concentrated visual explanations.
Fig. 7 : Real-world qualitative comparison on a Unitree G1 humanoid robot. Two representative navigation cases are shown across multiple decision steps. For each step, we compare egocentric attribution maps from DEX-AR [ 14 ] and our method, together with the corresponding third-person execution view. Our method better follows the evolving instruction stage and highlights decision-relevant visual evidence throughout real-world navigation
Fig. 8 : Our method as a diagnostic tool for VLN policies. Our attributions reveal stage- and token-level causes of VLN failures. (a) Premature subgoal execution : attention shifts to a later subgoal too early. (b) Action-phrase neglect : the landmark is emphasized while its action phrase is underweighted. (c) Partial instruction grounding : only part of the relevant instruction is grounded.
Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While this fusion of language instructions and visual observations allows multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual memory causally influence navigation decisions. Our results show that these navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these agents encode navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.
Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment. While state-of-the-art methods leverage the reasoning capabilities of Vision-Language Models (VLMs) for end-to-end action prediction, they often lack an explicit and explainable understanding of the relationships between the agent, the instruction, and the scene. Conversely, explicitly building a scene map for heuristic planning is intuitively appealing but relies on additional 3D sensors and hinders large-scale vision-language pre-training. To bridge this gap, we propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism, enabling it to understand the agent's state and task progress in a fully end-to-end and data-driven manner. Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training. Extensive experiments on various datasets in Habitat simulator show our AwareVLN significantly outperforms previous state-of-the-art vision-language navigation methods. Project page: https://gwxuan.github.io/AwareVLN/.
In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point. Most graph-based methods commonly match candidate trajectories with the entire instruction, allowing already executed content to interfere with subsequent decisions. Although instruction-execution progress provides a direct cue for separating executed and pending content, existing progress estimation approaches often rely on explicit progress supervision, sophisticated progress-learning stages, or costly online reasoning with LLMs. In this paper, we propose a robust VLN framework with Awareness of Instruction-Execution Progress (VLN-AIEP), which couples progress estimation with navigation decision-making. VLN-AIEP associates the visited trajectory with the instruction to suppress embeddings corresponding to executed content, and then matches candidate locations with the pending content. This coupling makes correct navigation decisions depend on accurate progress estimation, allowing the progress estimator to learn directly from navigation decision labels without explicit progress annotations. To support progress estimation, we construct a landmark-anchored trajectory representation that encodes the semantic labels and viewpoint-relative geometry of instruction-referred landmarks. These landmarks serve as spatial references for associating the visited trajectory with executed instruction content. Extensive experiments on R2R, RxR, CVDN, and R2R-CE demonstrate that the proposed method achieves favorable performance across multiple metrics.