Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

Yiyuan Yao; Xinpeng Ding; Minglei Chen; Fei Gao; Lihuo He; Jie Yuan; Aijun Li; Nannan Wang

cs.CVOct 25, 2026

Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

Authors: Yiyuan Yao, Xinpeng Ding, Minglei Chen, Fei Gao, Lihuo He, Jie Yuan, Aijun Li, Nannan Wang,

Organizations: Xidian University, Xi’an, China.

Abstract

As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important for reliable deployment. Existing explainability methods overlook a key challenge in Vision-Language Navigation (VLN): evidence for the current action may be distributed across historical and current observations and associated with different instruction stages. We identify instruction-stage insensitivity, where existing methods produce similar explanations despite changes across different navigation stages. To address this issue, we propose a stage-aware attribution framework for VLN. We first construct action-conditioned temporal instruction--visual attributions across observations. These attributions are projected into instruction space to estimate stage associations, which guide the aggregation of relevant historical evidence for refining the current explanation. An instruction-guided spatial filter further suppresses isolated responses while preserving compact, instruction-relevant regions. Experiments across multiple VLN policies and benchmarks show improved attribution quality over existing methods. We further demonstrate that our method acts as a diagnostic tool for analyzing failure cases, and validate its applicability to real-world navigation on a Unitree G1 robot.

Figures & tables

Fig. 1 : Visual explanation challenges in VLN. (a) Explaining a generated token involves localizing its visual evidence while reducing interference from preceding tokens. (b) Explaining a navigation action requires recovering temporally distributed evidence across historical and current observations, while suppressing cross-stage interference from evidence associated with other instruction stages. Colored boxes schematically indicate stage-related regions. (c) Existing methods [ 14 ] are insensitive to instruction-stage transitions, exhibiting overlapping token attributions ( top ) and high cross-stage explanation similarity ( bottom ).

Fig. 2 : Overview of our framework. (a) We construct action-conditioned temporal instruction–visual attributions across historical and current observations. (b) We project each attribution into instruction space to estimate its associated instruction stage. (c) We weight historical observations by their stage correspondence with the current view and aggregate the resulting support to refine the current attribution. (d) We further use instruction–patch associations to suppress spatial noise while preserving decision-relevant visual evidence.

Fig. 3 : Qualitative comparison with state-of-the-art methods across navigation stages. The same instruction is used throughout navigation, while the highlighted tokens are automatically induced by the action-conditioned instruction attribution at each decision step. For Steps 1, 7, and 21, we visualize early-history, recent-history, and current observations together with their attribution maps. Compared with existing methods, our explanations better follow the evolving instruction stage while retaining relevant evidence across historical and current views.

Dataset	Method	Type	$\Delta$ NLL(%) $\uparrow$	Precision (%) $\uparrow$	F1-Score(%) $\uparrow$
R2R	DEX-AR [ 14 ]	Gradient	14.54	9.12	11.21
	GradCAM [ 11 ]	Gradient	17.66	35.91	23.68
	TAM [ 13 ]	Logit	14.92	7.13	9.65
	Ours	Gradient	31.98	43.70	36.93
RxR	DEX-AR [ 14 ]	Gradient	21.13	7.04	10.56
	GradCAM [ 11 ]	Gradient	27.83	28.62	28.22

TABLE I : Quantitative comparison on R2R and RxR using NaVILA under the VLN-CE setting. We report decision faithfulness (

\Delta\mathrm{NLL}_{\mathrm{corr}}

), semantic alignment (Precision), and F1-Score. Higher is better for all metrics.

Dataset	VLN Policy	Ours over DEX-AR [ 14 ]
Dataset	VLN Policy	$\Delta$ NLL (%) $\uparrow$	Prec. (%) $\uparrow$	F1 (%) $\uparrow$
R2R	NaVid [ 6 ]	+14.23	+18.23	+16.50
	NaVILA [ 5 ]	+17.44	+34.58	+25.72
	StereoNav [ 33 ]	+5.16	+34.31	+9.17
RxR	NaVid [ 6 ]	+11.86	+37.50	+22.58
	NaVILA [ 5 ]	+7.33	+35.37	+23.50

TABLE II : Performance gains over DEX-AR across different VLN policies. All values denote improvements.

Fig. 4 : Effect of TR. Comparisons before and after TR show that stage-guided historical support reduces cross-stage interference and yields more focused attribution on evidence relevant to the current navigation decision. Red boxes highlight the irrelevant regions.

Fig. 5 : Effect of SF. Visualization of the base rank filter [ 13 ] and our uni-/bidirectional SF variants (Section III-E ). Green and red boxes denote instruction-relevant and irrelevant regions, respectively.

TR	SF	$\Delta\mathrm{NLL}(\%)\uparrow$	Precision (%) $\uparrow$	F1-Score (%) $\uparrow$
$\times$	$\times$	26.61	41.80	32.52
$\checkmark$	$\times$	27.81	43.28	33.86
$\times$	$\checkmark$	30.82	41.93	35.53
$\checkmark$	$\checkmark$	31.98	43.70	36.93

TABLE III : Ablation of temporal refinement (TR, Sec. III-D ) and spatial filtering (SF, Sec. III-E ) on R2R.

Fig. 6 : Quantitative analysis of instruction-stage insensitivity. Top: token-level instruction attribution and frame-level similarity. Bottom: dataset-level comparison with the baseline, i.e. , TAM [ 13 ] . Our method shows stronger sensitivity to instruction-stage transitions and more concentrated visual explanations.

Fig. 7 : Real-world qualitative comparison on a Unitree G1 humanoid robot. Two representative navigation cases are shown across multiple decision steps. For each step, we compare egocentric attribution maps from DEX-AR [ 14 ] and our method, together with the corresponding third-person execution view. Our method better follows the evolving instruction stage and highlights decision-relevant visual evidence throughout real-world navigation

Fig. 8 : Our method as a diagnostic tool for VLN policies. Our attributions reveal stage- and token-level causes of VLN failures. (a) Premature subgoal execution : attention shifts to a later subgoal too early. (b) Action-phrase neglect : the landmark is emphasized while its action phrase is underweighted. (c) Partial instruction grounding : only part of the relevant instruction is grounded.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 267 papers

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
Sep 21, 2026Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak +4Vision-Language Models VLM Interpretability
University of Technology Nuremberg. · ETH Zürich. · Inria, Ecole Normale Sup´erieure, CNRS, PSL Research University.
AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
May 21, 2026Wenxuan Guo, Xiuwei Xu, Yichen Liu +7Vision-Language Navigation VLM Reasoning
Tsinghua University, China
Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress
Oct 9, 2026Jiagan Huang, Zikun Zhou, Hongpeng Wang +3Vision-Language Navigation
Harbin Institute of Technology, Shenzhen

Sep 21, 2026J/K move · Enter open · S save

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

Débora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter+3

University of Technology Nuremberg. · ETH Zürich. · Inria, Ecole Normale Sup´erieure, CNRS, PSL Research University.

Vision-Language Models VLM Interpretability Vision-Language Navigation Activation Steering

Open paper AskPDF arXiv

Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While this fusion of language instructions and visual observations allows multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual memory causally influence navigation decisions. Our results show that these navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these agents encode navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.

Abstract

VLM Interpretability

Explore similar work

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress