GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance
Authors: Jingyu Guo, Jiaxin Huang, Chengxing Lin, Xiaoyu Luo, Cheng Wen, Shanshan Ye, Tingjin Chu, Shaoli Huang, +1 more
Organizations: University of Melbourne · AgiBot · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China · University of Macau · University of Technology Sydney
Vision-and-language navigation (VLN) requires an agent to track progress through an extended instruction, identify landmarks relevant to its current objective, and understand the surrounding spatial structure to act. End-to-end navigation policies typically rely on action supervision to implicitly learn these capabilities. Without explicit guidance on the evidence underlying each decision, however, policies can overfit to correlations predictive of training actions rather than learn the task-relevant grounding and geometry needed for generalization. Our insight is to guide the representations that connect instructions to actions: identifying what remains to be done determines which visual evidence matters, while understanding the geometry around that evidence informs how to move. We introduce GuidedNav, a unified framework that explicitly shapes these representations through training-time guidance. Subtask and landmark guidance steer native pre-action attention toward the active instruction segment and relevant visual regions, grounding action prediction in the current objective. Spatial guidance aligns the policy's internal representations with geometric features that capture long-horizon context and local spatial structure. Together, these objectives strengthen navigation capabilities while avoiding the inference overhead of intermediate textual reasoning and online geometric encoding. We also introduce Guided-R2R and Guided-RxR, two annotation-enriched navigation datasets that support more interpretable VLN research. GuidedNav establishes state-of-the-art performance among the compared monocular RGB policies, improving success rates by 8.1% and 13.1% relative to the strongest prior results on R2R-CE and RxR-CE val-unseen, respectively, while achieving faster inference. Extensive experiments further demonstrate the effectiveness and real-world transferability of our approach.
Figures & tables
Figure 1: Guidance Designs in GuidedNav. Subtask guidance focuses native pre-action attention on the active instruction span (left); landmark guidance directs it toward task-relevant visual regions (middle); spatial guidance shapes representations of the surrounding 3D structure using historical and nearby observations (right). Together, these training-time objectives connect instruction progress, visual grounding, and geometry to direct action prediction along the illustrated route.
Figure 2: Architecture of GuidedNav. Subtask and landmark guidance supervise native pre-action attention toward the active instruction span and relevant visual regions. Spatial guidance aligns current-frame representations with history-conditioned geometric features and supervises local fusion using geometric feature differences from nearby observations. All auxiliary objectives and the frozen spatial encoder are used only during training.
Figure 3: Landmark-Centric Data Engine for Guided-R2R and Guided-RxR. The pipeline combines subtask decomposition and temporal alignment, landmark selection and localization, and per-frame mask generation to produce annotations for navigation training.
Table 1: Monocular RGB Navigation on R2R-CE and RxR-CE Val-Unseen. External data counts samples beyond standard R2R/RxR-CE data.
Figure 5
Configuration
NE ↓
OSR ↑
SR ↑
SPL ↑
Base Model
4.87
63.8
59.3
56.1
Overall Guidance Ablation
Base + Subtask Attention Guidance
4.34
72.3
62.9
55.2
Base + Landmark Attention Guidance
4.38
72.1
62.6
55.8
Base + Spatial Guidance
4.33
68.5
62.4
57.2
GuidedNav (All)
3.83
71.5
65.4
59.1
Table 3: Ablation studies on R2R-CE val-unseen. SG denotes Spatial Guidance. The upper block evaluates individual modules against Base Model, defined in Section 4.1 ; the lower block progressively builds Spatial Guidance. The complete configuration in each block is shaded in blue.
Subtask Localization
Landmark Localization
Spatial Probing
Model
AUPRC (%) ↑
AUROC (%) ↑
PG (%) ↑
EBPG (%) ↑
AbsRel ↓
RMSE (m) ↓
Rank Acc. (%) ↑
Base Model
34.5
53.2
15.2
13.4
0.25
0.82
85.9
GuidedNav
56.3
76.1
49.1
42.6
0.19
0.64
91.7
Table 4: Attention Localization and Spatial Probing. Subtask and landmark localization are evaluated on R2R-CE val-unseen; EBPG is computed at the patch level, and Rank Acc. measures near–far depth ordering. Protocols appear in Appendices C.3 and C.4 .
Figure 5: Subtask Attention across Navigation Steps. We compare the Base Model and GuidedNav at sampled steps. Green underlines mark the GT active subtask, while darker blue indicates higher native pre-action attention.
Figure 9
Figure 8: Real-World Navigation without Real-World Fine-Tuning. Six decisions show third-person views, RGB observations, actions, decoded depth, and pre-action attention. Depth is for analysis only; attention is normalized within each displayed token group.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Stage 1
Stage 2
Stage 3 (GuidedNav)
Initialization
Qwen3-VL-8B-Instruct
Stage 1 checkpoint
Stage 2 checkpoint
Navigation data
R2R + RxR experts
DAgger
Annotated R2R + RxR experts
VQA data
Included
Included
Not included
Policy update
Full parameters
Full parameters
LoRA adapters
Peak policy LR
2×10−5
1×10−5
2×10−4
Our guidance
None
None
Subtask, landmark, spatial
Appendix
Table 5: Three-Stage Training Recipe. Every stage uses one epoch and global batch size 256. Stages 1 and 2 use full fine-tuning; Stage 3 adapts the policy with rank-32 LoRA and scaling factor 16.
Figure 9: Long-Horizon Spatial Guidance and Local Spatial Refinement. The upper branch supervises matched-token residuals with LLocal and fuses them into current visual tokens. The lower branch aligns current hidden states with history-conditioned teacher features through LLong . The teacher and auxiliary projectors are used only during training; temporal fusion remains active at inference.
Setting
Value
Decoding
Greedy
Maximum sampled context images
8
Executed action chunk
Up to 4 primitives
Forward displacement
0.25 m
Turn angle
15∘
Maximum episode actions
500
Appendix
Table 6: Simulator and Action Settings. The common action settings apply to both datasets; collision-body dimensions shown below are the RxR-specific settings.
Figure 10: Guided-R2R Annotation Example. Three subtask spans align hallway traversal, a right turn, and stopping by a black chair with their frame intervals. Representative observations show the selected landmarks and whether a mask was accepted.
Figure 11: Guided-RxR Annotation Example. Four subtask spans follow a route from a laundry room through a kitchen and bedroom to a glass door. Matching colors link instruction text, frame intervals, and representative landmark masks.
Method
SR ↑
SPL ↑
OSR ↑
NE ↓
Base Model
59.3
56.1
63.8
4.87
Sequential Generation
60.4
55.4
66.7
4.43
Auxiliary Generation
62.0
55.6
72.5
4.55
Attention Guidance
62.9
55.2
72.3
4.34
Appendix
Table 7: Subtask Supervision Strategies on R2R-CE Val-Unseen.
Method
SR ↑
SPL ↑
OSR ↑
NE ↓
Base Model
59.3
56.1
63.8
4.87
Distribution Matching
61.6
55.2
71.2
4.48
Access Regularization
62.4
55.8
71.5
4.40
Attention Guidance (Ours)
62.6
55.8
72.1
4.38
Appendix
Table 8: Landmark Attention Supervision on R2R-CE Val-Unseen. We compare Distribution Matching, Access Regularization, and our Attention Guidance objective. The latter maximizes attention mass supported by the landmark mask. SR, SPL, and OSR are percentages; NE is in meters. Bold denotes the best value in each column.
Figure 12: Landmark Attention Localization. Instruction excerpts and reference masks are compared with Base Model and GuidedNav pre-action attention for stairs, chairs, and an oven. Heatmaps cover the full current image.
Figure 13: Successful R2R-CE Rollouts: Cases 1-3. Columns follow six sampled decisions along each trajectory. Rows show observations, predicted action chunks, decoded depth, landmark attention, and instruction attention; darker instruction highlighting indicates greater relative attention.
Figure 14: Successful R2R-CE Rollouts: Cases 4-6. Six sampled decisions per trajectory, following the conventions in Figure 13 .
Figure 15: Additional Real-World Navigation Examples. Six sampled decisions per trajectory show observations, predicted actions, decoded depth, and native pre-action attention. The visualization follows the conventions in Figure 8 .
Method
Inputs
NE ↓
OSR ↑
SR ↑
SPL ↑
HPN+DN ( Krantz et al., 2021 )
P+O+D
6.31
40.0
36.0
34.0
CMA ( Hong et al., 2022 )
P+O+D
6.20
52.0
41.0
36.0
Sim2Sim ( Krantz and Lee, 2022 )
P+O+D
6.07
52.0
43.0
36.0
VLN ↻ BERT ( Hong et al., 2022 )
P+O+D
5.74
53.0
44.0
39.0
Ego 2 -Map ( Hong et al., 2023 )
P+O+D
5.54
56.0
47.0
41.0
DreamWalker ( Wang et al., 2023a )
P+O+D
5.53
59.0
49.0
44.0
Appendix
Table 9: Additional Navigation Comparisons. Methods using panoramic observations or additional sensory inputs. P: panoramic RGB; R: monocular RGB; O: odometry; D: depth. NE is in meters; all other metrics are percentages.