GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance
Authors: Jingyu Guo, Jiaxin Huang, Chengxing Lin, Xiaoyu Luo, Cheng Wen, Shanshan Ye, Tingjin Chu, Shaoli Huang, +1 more
Organizations: University of Melbourne · AgiBot · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China · University of Macau · University of Technology Sydney
Vision-and-language navigation (VLN) requires an agent to track progress through an extended instruction, identify landmarks relevant to its current objective, and understand the surrounding spatial structure to act. End-to-end navigation policies typically rely on action supervision to implicitly learn these capabilities. Without explicit guidance on the evidence underlying each decision, however, policies can overfit to correlations predictive of training actions rather than learn the task-relevant grounding and geometry needed for generalization. Our insight is to guide the representations that connect instructions to actions: identifying what remains to be done determines which visual evidence matters, while understanding the geometry around that evidence informs how to move. We introduce GuidedNav, a unified framework that explicitly shapes these representations through training-time guidance. Subtask and landmark guidance steer native pre-action attention toward the active instruction segment and relevant visual regions, grounding action prediction in the current objective. Spatial guidance aligns the policy's internal representations with geometric features that capture long-horizon context and local spatial structure. Together, these objectives strengthen navigation capabilities while avoiding the inference overhead of intermediate textual reasoning and online geometric encoding. We also introduce Guided-R2R and Guided-RxR, two annotation-enriched navigation datasets that support more interpretable VLN research. GuidedNav establishes state-of-the-art performance among the compared monocular RGB policies, improving success rates by 8.1% and 13.1% relative to the strongest prior results on R2R-CE and RxR-CE val-unseen, respectively, while achieving faster inference. Extensive experiments further demonstrate the effectiveness and real-world transferability of our approach.
Figures & tables
Figure 1: Guidance Designs in GuidedNav. Subtask guidance focuses native pre-action attention on the active instruction span (left); landmark guidance directs it toward task-relevant visual regions (middle); spatial guidance shapes representations of the surrounding 3D structure using historical and nearby observations (right). Together, these training-time objectives connect instruction progress, visual grounding, and geometry to direct action prediction along the illustrated route.
Figure 2: Architecture of GuidedNav. Subtask and landmark guidance supervise native pre-action attention toward the active instruction span and relevant visual regions. Spatial guidance aligns current-frame representations with history-conditioned geometric features and supervises local fusion using geometric feature differences from nearby observations. All auxiliary objectives and the frozen spatial encoder are used only during training.
Figure 3: Landmark-Centric Data Engine for Guided-R2R and Guided-RxR. The pipeline combines subtask decomposition and temporal alignment, landmark selection and localization, and per-frame mask generation to produce annotations for navigation training.
Table 1: Monocular RGB Navigation on R2R-CE and RxR-CE Val-Unseen. External data counts samples beyond standard R2R/RxR-CE data.
Figure 5
Configuration
NE ↓
OSR ↑
SR ↑
SPL ↑
Base Model
4.87
63.8
59.3
56.1
Overall Guidance Ablation
Base + Subtask Attention Guidance
4.34
72.3
62.9
55.2
Base + Landmark Attention Guidance
4.38
72.1
62.6
55.8
Base + Spatial Guidance
4.33
68.5
62.4
57.2
GuidedNav (All)
3.83
71.5
65.4
59.1
Table 3: Ablation studies on R2R-CE val-unseen. SG denotes Spatial Guidance. The upper block evaluates individual modules against Base Model, defined in Section 4.1 ; the lower block progressively builds Spatial Guidance. The complete configuration in each block is shaded in blue.
Subtask Localization
Landmark Localization
Spatial Probing
Model
AUPRC (%) ↑
AUROC (%) ↑
PG (%) ↑
EBPG (%) ↑
AbsRel ↓
RMSE (m) ↓
Rank Acc. (%) ↑
Base Model
34.5
53.2
15.2
13.4
0.25
0.82
85.9
GuidedNav
56.3
76.1
49.1
42.6
0.19
0.64
91.7
Table 4: Attention Localization and Spatial Probing. Subtask and landmark localization are evaluated on R2R-CE val-unseen; EBPG is computed at the patch level, and Rank Acc. measures near–far depth ordering. Protocols appear in Appendices C.3 and C.4 .
Figure 5: Subtask Attention across Navigation Steps. We compare the Base Model and GuidedNav at sampled steps. Green underlines mark the GT active subtask, while darker blue indicates higher native pre-action attention.
Figure 9
Figure 8: Real-World Navigation without Real-World Fine-Tuning. Six decisions show third-person views, RGB observations, actions, decoded depth, and pre-action attention. Depth is for analysis only; attention is normalized within each displayed token group.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Stage 1
Stage 2
Stage 3 (GuidedNav)
Initialization
Qwen3-VL-8B-Instruct
Stage 1 checkpoint
Stage 2 checkpoint
Navigation data
R2R + RxR experts
DAgger
Annotated R2R + RxR experts
VQA data
Included
Included
Not included
Policy update
Full parameters
Full parameters
LoRA adapters
Peak policy LR
2×10−5
1×10−5
2×10−4
Our guidance
None
None
Subtask, landmark, spatial
Appendix
Table 5: Three-Stage Training Recipe. Every stage uses one epoch and global batch size 256. Stages 1 and 2 use full fine-tuning; Stage 3 adapts the policy with rank-32 LoRA and scaling factor 16.
Figure 9: Long-Horizon Spatial Guidance and Local Spatial Refinement. The upper branch supervises matched-token residuals with LLocal and fuses them into current visual tokens. The lower branch aligns current hidden states with history-conditioned teacher features through LLong . The teacher and auxiliary projectors are used only during training; temporal fusion remains active at inference.
Setting
Value
Decoding
Greedy
Maximum sampled context images
8
Executed action chunk
Up to 4 primitives
Forward displacement
0.25 m
Turn angle
15∘
Maximum episode actions
500
Appendix
Table 6: Simulator and Action Settings. The common action settings apply to both datasets; collision-body dimensions shown below are the RxR-specific settings.
Figure 10: Guided-R2R Annotation Example. Three subtask spans align hallway traversal, a right turn, and stopping by a black chair with their frame intervals. Representative observations show the selected landmarks and whether a mask was accepted.
Figure 11: Guided-RxR Annotation Example. Four subtask spans follow a route from a laundry room through a kitchen and bedroom to a glass door. Matching colors link instruction text, frame intervals, and representative landmark masks.
Method
SR ↑
SPL ↑
OSR ↑
NE ↓
Base Model
59.3
56.1
63.8
4.87
Sequential Generation
60.4
55.4
66.7
4.43
Auxiliary Generation
62.0
55.6
72.5
4.55
Attention Guidance
62.9
55.2
72.3
4.34
Appendix
Table 7: Subtask Supervision Strategies on R2R-CE Val-Unseen.
Method
SR ↑
SPL ↑
OSR ↑
NE ↓
Base Model
59.3
56.1
63.8
4.87
Distribution Matching
61.6
55.2
71.2
4.48
Access Regularization
62.4
55.8
71.5
4.40
Attention Guidance (Ours)
62.6
55.8
72.1
4.38
Appendix
Table 8: Landmark Attention Supervision on R2R-CE Val-Unseen. We compare Distribution Matching, Access Regularization, and our Attention Guidance objective. The latter maximizes attention mass supported by the landmark mask. SR, SPL, and OSR are percentages; NE is in meters. Bold denotes the best value in each column.
Figure 12: Landmark Attention Localization. Instruction excerpts and reference masks are compared with Base Model and GuidedNav pre-action attention for stairs, chairs, and an oven. Heatmaps cover the full current image.
Figure 13: Successful R2R-CE Rollouts: Cases 1-3. Columns follow six sampled decisions along each trajectory. Rows show observations, predicted action chunks, decoded depth, landmark attention, and instruction attention; darker instruction highlighting indicates greater relative attention.
Figure 14: Successful R2R-CE Rollouts: Cases 4-6. Six sampled decisions per trajectory, following the conventions in Figure 13 .
Figure 15: Additional Real-World Navigation Examples. Six sampled decisions per trajectory show observations, predicted actions, decoded depth, and native pre-action attention. The visualization follows the conventions in Figure 8 .
Method
Inputs
NE ↓
OSR ↑
SR ↑
SPL ↑
HPN+DN ( Krantz et al., 2021 )
P+O+D
6.31
40.0
36.0
34.0
CMA ( Hong et al., 2022 )
P+O+D
6.20
52.0
41.0
36.0
Sim2Sim ( Krantz and Lee, 2022 )
P+O+D
6.07
52.0
43.0
36.0
VLN ↻ BERT ( Hong et al., 2022 )
P+O+D
5.74
53.0
44.0
39.0
Ego 2 -Map ( Hong et al., 2023 )
P+O+D
5.54
56.0
47.0
41.0
DreamWalker ( Wang et al., 2023a )
P+O+D
5.53
59.0
49.0
44.0
Appendix
Table 9: Additional Navigation Comparisons. Methods using panoramic observations or additional sensory inputs. P: panoramic RGB; R: monocular RGB; O: odometry; D: depth. NE is in meters; all other metrics are percentages.
Vision-and-Language Navigation (VLN) policies increasingly benefit from strong semantic priors provided by large vision-language models (VLMs). However, standard action supervision does not explicitly encourage intermediate representations to preserve scene geometry, relative orientation, or global episode progress. Incorporating depth estimators, explicit maps, point clouds, or geometry foundation models at inference can provide such structure but introduces additional computation, memory overhead, and architectural dependence during deployment. We introduce StageVLN, a training framework that shapes navigation representations through privileged spatial and trajectory guidance while preserving the original inference pathway. A frozen geometry foundation model provides multi-level spatial guidance to hierarchical navigator states, while relative-heading and expert-route progress objectives provide complementary trajectory-state supervision. All auxiliary components are used only during training and removed at deployment. On R2R-CE validation-unseen, StageVLN achieves 56.3% SR and 51.4% SPL with a 4B-parameter backbone, without an additional geometry encoder at inference. On RxR-CE, it achieves 54.3% SR without additional navigation training data or a geometry encoder at inference.
Anh Dao, Quan-Dung Pham, Le Danh Vinh +6
VinMotion, Inc., Vietnam · University of Southern California, USA
In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point. Most graph-based methods commonly match candidate trajectories with the entire instruction, allowing already executed content to interfere with subsequent decisions. Although instruction-execution progress provides a direct cue for separating executed and pending content, existing progress estimation approaches often rely on explicit progress supervision, sophisticated progress-learning stages, or costly online reasoning with LLMs. In this paper, we propose a robust VLN framework with Awareness of Instruction-Execution Progress (VLN-AIEP), which couples progress estimation with navigation decision-making. VLN-AIEP associates the visited trajectory with the instruction to suppress embeddings corresponding to executed content, and then matches candidate locations with the pending content. This coupling makes correct navigation decisions depend on accurate progress estimation, allowing the progress estimator to learn directly from navigation decision labels without explicit progress annotations. To support progress estimation, we construct a landmark-anchored trajectory representation that encodes the semantic labels and viewpoint-relative geometry of instruction-referred landmarks. These landmarks serve as spatial references for associating the visited trajectory with executed instruction content. Extensive experiments on R2R, RxR, CVDN, and R2R-CE demonstrate that the proposed method achieves favorable performance across multiple metrics.
Although vision-language models (VLMs) possess strong visual understanding and reasoning capabilities, existing vision-and-language navigation (VLN) agents struggle to connect semantic reasoning with spatial execution. Two coupled gaps remain in this connection, as intermediate reasoning is not explicitly anchored to visual evidence and high-level decisions lack precise spatial goals to guide low-level motion. Cognitive science suggests that human navigation bridges these levels hierarchically by anchoring cognition to relevant landmarks and guiding locomotion toward spatial goals. Motivated by this principle, we propose GroundingVLN, which uses visual grounding as a shared interface between reasoning and action. GroundingVLN first reasons with grounding by anchoring task-relevant visual evidence to precise image locations throughout structured reasoning. It then acts through grounding by predicting a progress-aligned pixel goal that a geometric planner translates into primitive actions. To learn these capabilities, we construct GroundingCOTVLN-188K, a dataset of temporally aligned grounded reasoning traces, and introduce Grounded and Execution-Aware Reinforcement Learning (GEAR), which aligns grounded reasoning and spatial decisions with downstream execution. Experiments demonstrate that GroundingVLN achieves state-of-the-art performance (69.9% SR on R2R-CE and 75.1% SR on RxR-CE) with high sample efficiency, using just 0.9% as much training data as the strongest baseline. It also generalizes strongly across datasets, attaining 59.9% SR on RxR-CE when trained solely on R2R, a gain of 20.1% over the strongest baseline. Code and models will be released after review. Code is available at https://github.com/Teacher-Tom/GroundingVLN.
Kailing Li, Yu Han, Tianwen Qian +4
School of Computer Science and Technology, East China Normal University · NeoteAI · King Abdullah University of Science and Technology