What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.
Figures & tables
Figure 1: See what I need, dream what I want. RepresentNav learns a shared visual interface over frozen visual priors and rolls out latent predictions before generating waypoints. Left: real-world person tracking and post-hoc readouts of semantics, occupancy, and motion. Right: model overview and benchmark results.
Figure 2: RepresentNav overview. (a) Representation Residuals attend over layers spanning the full depth of three frozen vision foundation models and build the self-constructed representation, summarized per view by a learnable [CLS] token. (b) The policy dreams in this space: a latent chain-of-thought predicts the [CLS] tokens of future observations, regularized by VISReg, before the planning head acts; QA shares the backbone, and decoded latents are for visualization only.
STT
DT
Method
TR ↑
SR ↑
CR ↓
TR ↑
SR ↑
CR ↓
Monocular
Uni-NaVid
39.5
25.7
41.90
27.4
11.3
43.50
TrackVLA
78.6
85.1
1.65
63.2
57.6
5.80
NavFoM
80.5
85.0
–
68.2
61.4
–
TrackVLA++
81.0
86.0
2.10
68.8
66.5
4.71
Table 1: Benchmark results. Baselines are from the cited works; bold : best per column. (a) EVT-Bench single-target (STT) and distracted (DT) tracking [ 31 , 60 , 12 , 1 ] ; best per observation setting. (b) InternVLA-N1 point-goal [ 32 , 55 ] . (c) Indoor ABotN-PointBench, collision-free success [ 1 ] . (d) VLN-CE Val-Unseen [ 32 , 53 , 9 , 1 , 59 , 60 ] ; best per observation setting; ∗ : waypoint-predictor-based; Qwen-RobotNav at 4B, the scale of our backbone. (e) VLNVerse, 3 m threshold, Full Test [ 60 ] and Val-Unseen [ 30 , 27 ] ; best per split.
R2R
RxR
Method
NE ↓
OSR ↑
SR ↑
SPL ↑
NE ↓
SR ↑
SPL ↑
nDTW ↑
Monocular
NaVid
5.72
49.2
41.9
36.5
5.72
45.7
38.2
–
Uni-NaVid
5.58
53.3
47.0
42.7
6.24
48.7
40.9
–
NaVILA
5.22
62.5
54.0
49.0
6.77
49.3
44.0
58.8
StreamVLN
4.98
64.2
56.9
51.9
6.22
52.9
46.0
61.9
Table 2: Benchmark results (continued). Vision-and-language navigation. Panel descriptions and evaluation conventions are given on the preceding page.
Figure 3: Real-world navigation and representation readouts. RepresentNav performs point-goal navigation, person following, and vision-and-language navigation in real environments. Insets show occupancy, target-box, and semantic readouts.
General description
Sem.
Geo.
Dyn.
Interface
Dream target
SR (%) ↑
SPL ↑
CR (%) ↓
(a) What the policy sees (no dream)
–
–
–
native ViT
none
79.65
73.43
40.80
–
✓
–
concat.
none
83.25
76.08
35.40
✓
✓
–
concat.
none
86.70
79.57
30.90
✓
✓
✓
concat.
none
87.30
80.20
28.70
Table 3: What the policy sees and where it dreams. Point-goal ablations; no row in (a) dreams. General description: semantic (DINOv3), geometric (DA3), and dynamic (OpenD4RT) vision foundation models. W/o supervision: the latents are rolled out with no loss on them.
Figure 4: What the constructed space draws on. (a) Attention shares of Representation Residuals by encoder during training; the running state, initialized from the geometric encoder, counts as geometric. (b) Where Representation Residuals in the final model prefer each encoder, in six selected VLNVerse views; colors show contrast-enhanced relative preferences, not absolute weights. Details and shares by depth in Appendix B.2 .
Figure 5: What one dreamed latent holds. Post-hoc readouts from the +0.4 s latent ((a) depth, (b) semantics, (c) 3D occupancy) and all four latents ((d) ego motion), on 1,000 test observations. Encoder features and the [CLS] tokens of Representation Residuals from real future images are reference inputs, not predicted latents; full results are in Table 8 . Examples were selected to show the model’s strengths; more in Appendix Figure 9 .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training data composition (Section 3.4 ).
Restricted stage
Open stage
SR (%) ↑
SPL ↑
CR (%) ↓
–
✓
92.40
83.68
15.15
✓
✓
95.00
86.70
13.45
Appendix
Table 4: Effect of the two-stage curriculum. Same model and data; the single-stage variant opens attention from the start instead of first restricting the action position to the latent chain-of-thought.
Training
Inference condition
Affected (%)
Shift (cm)
GT ADE (cm)
Single-stage
Normal
0.0
0.00
3.93
Single-stage
Cross-scene swap
0.0
1.38
4.32
Single-stage
Zero inputs
2.0
2.68
4.98
Two-stage
Normal
0.0
0.00
4.38
Two-stage
Cross-scene swap
27.0
4.95
7.06
Two-stage
Zero inputs
33.0
8.24
9.76
Appendix
Table 5: The two-stage policy responds more strongly to changes in its latent inputs. Each model and condition uses 200 observations.
Figure 7: Latent-input interventions and action sensitivity. Left: GT and normal, cross-scene-swap, and zero-input predictions from our two-stage point-goal policy; axes are left and forward (meters), with equal XY scales. Right: mean XY change from each policy’s normal prediction over eight waypoints on the same 200 observations. All five models use their own predicted latents; Qwen native dreams in the backbone ViT’s pooled features and Free run rolls out the same latents with no loss on them (w/o supervision in Table 3 (b)), while VICReg and SIGReg replace VISReg (Table 7 ). Only the four recurrent inputs are swapped within each checkpoint or zeroed; other inputs stay fixed, with no future images. These open-loop diagnostics on navigation-training observations measure sensitivity, not navigation performance.
Figure 8: Attention shares by encoder and by depth. Attention shares of Representation Residuals during training over a subset of layers spaced along each encoder’s depth, averaged over its three layers and all positions. (a) By encoder; the running state, initialized from the geometric encoder, counts as geometric. (b) By depth within each encoder; dashed lines mark final layers.
Representation Residuals layers
1
2
4
8
16
SR (%) ↑
92.75
91.15
92.25
91.70
93.30
SPL (%) ↑
84.1
82.6
83.0
82.6
84.1
CR (%) ↓
18.40
18.25
17.25
16.10
16.40
Appendix
Table 6: Capacity of Representation Residuals is not the source of the gain. Point-goal navigation without the dream, all three encoders, varying the number of Representation Residuals layers. The default uses three layers (SR 93.45%, SPL 84.2%, CR 18.05%); all variants lie within 2.2 points of one another in SR and SPL, so the gain in Table 3 is not a matter of capacity.
Regularizer
SR (%) ↑
SPL ↑
CR (%) ↓
None
92.90
84.09
16.40
SIGReg
93.85
84.39
17.15
VICReg
93.25
83.98
16.40
VISReg (ours)
95.00
86.70
13.45
Appendix
Table 7: Choice of the target regularizer. Same model and data; only the regularizer on the dream targets changes; None keeps the prediction loss and drops only the regularizer.
Depth
Occupancy
Semantic
Pixel
Ego motion
Representation
MAE ↓
IoU (%) ↑
mIoU (%) ↑
acc. (%) ↑
ADE (m) ↓
Backbone ViT on real future
0.16257
25.47
16.05
57.32
0.17932
DA3 on real future
0.14472
26.14
15.25
59.99
0.14095
DINOv3 on real future
0.15206
25.85
16.84
58.72
0.14209
openD4RT on real future
0.10253
29.79
20.34
66.67
0.04713
Representation Residuals [CLS] on real future
0.07520
35.95
24.82
73.45
0.02962
Appendix
Table 8: Full readout comparison , including the alternative regularizers and the single-stage model. Each metric uses 1,000 test observations.
Figure 9: Future scenes and ego motion can be read from the dream. Expanded comparison for Figure 5 , using the same observations and selected camera views. (a) The camera depth uses original 32×32 relative inverse-depth maps and GT references on a common color scale. (b) The camera semantics uses original 64×64 labels and GT references with a common class palette. Both target +0.4 s; future RGB provides visual context. (c) Occupancy uses native 120×80×20 voxels with identical viewpoints, spatial bounds, and height colors. (d) Ego motion shows six future positions with shared ranges and equal axis scales; dashed and solid curves denote past and future motion. GT sources and full-test metrics are specified in Table 8 . Groups use different observations, selected post hoc to illustrate RepresentNav’s strengths.
Target localization (%)
Person motion (m)
Ego motion (m)
Model
Front IoU ↑
All IoU ↑
AP50 ↑
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Backbone ViT on real future
33.69
32.74
8.67
1.09701
1.27292
0.33404
0.54533
SIGReg (two-stage)
59.33
50.04
36.56
0.78507
0.90616
0.22126
0.37249
RepNav (ours)
72.81
61.66
69.44
0.44450
0.55498
0.15455
0.25190
Appendix
Table 9: RepresentNav improves target localization and motion readouts on the expanded Track test set. All three models are evaluated on the same 5,268 observations.
Figure 10: RepresentNav localizes the target more accurately than the baselines. Frozen-model bounding-box readouts on two expanded Track test observations. Each row shares the current-view RGB image and GT across models. Green: GT reference. Magenta: original decoded box coordinates before visibility-confidence filtering, shown for all models to compare localization on GT-visible targets, not thresholded detections. RepresentNav’s boxes lie closer to GT; the baselines are displaced. Examples are selected post hoc; aggregate results are in Table 9 .
Figure 11: RepresentNav recovers the target’s position and subsequent motion. Frozen-model readouts on two observations from the expanded Track test set. Each column compares one model with GT: cyan solid curves are GT references; orange dashed curves are readouts. Circles, diamonds, and triangles mark past, current, and future positions. Each row shares metric bounds across models, with left horizontal and forward vertical in the fixed current-robot frame. Examples are selected post hoc; aggregate results and model conventions are in Table 9 .
Figure 12: RepresentNav recovers future ego displacement underestimated by the baselines. Frozen-model readouts on two observations from the expanded Track test set. Colors, markers, and BEV axes follow Figure 11 ; each row shares metric bounds across models. Ego t=0 is fixed at the origin. RepresentNav follows the GT displacement, while the backbone ViT and SIGReg predict shorter travel. Examples are selected post hoc. These curves are readouts, not executed trajectories; evaluation conventions are in Table 9 .
Figure 14: Real-world robot platform and inference latency. (a) The Go2 quadruped and Bumi humanoid, with camera locations marked. The Go2 carries four SENSING GMSL2 cameras and a reComputer Robotics J3011 with an NVIDIA Jetson Orin Nano 8GB module. (b) RGB observations are sent to an RTX 4090 server running RepresentNav; predicted waypoints are returned to the onboard controller for robot execution. (c) Inference latency of Native Qwen, Qwen with Representation Residuals, and our full model.
Figure 15: Visualization of real-world experiments across tasks.
Vision-language navigation requires aligning language with visual observations while maintaining spatial understanding over time. Geometry foundation models (GFMs) expose intermediate representations throughout their hierarchy, but how navigation policies should use these features and retain historical geometric evidence remains unresolved. We introduce \method{}, a streaming VLN framework that addresses these questions across representation depth and navigation time. Hierarchical GFM--VLM fusion couples earlier, intermediate, and later GFM representations to successive policy stages instead of repeatedly injecting a terminal feature. Navigation-aware GFM memory retains historical VGGT global-attention KV states according to instruction relevance, geometric confidence, and transition novelty under a bounded per-layer budget. Retained states provide geometric context for subsequent observations before fusion with the policy. Across R2R-CE and RxR-CE, \method{} achieves strong performance using a single RGB stream without additional navigation-specific external data. Controlled ablations show that multi-depth coupling substantially outperforms repeated terminal-feature injection at matched fusion locations. Bounded navigation-aware retention preserves navigation performance while considerably reducing GFM-KV memory relative to larger-memory temporal retention. These findings support jointly examining the geometric representations exposed to the policy and the historical evidence retained for future inference. Code will be released upon acceptance at https://humanoid-research.github.io/adageovln/.
Quan-Dung Pham, Anh Dao, Danh Vinh Le +7
VinMotion, Inc., Vietnam · Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA · University of Southern California, USA
Vision-and-language navigation (VLN) requires an agent to track progress through an extended instruction, identify landmarks relevant to its current objective, and understand the surrounding spatial structure to act. End-to-end navigation policies typically rely on action supervision to implicitly learn these capabilities. Without explicit guidance on the evidence underlying each decision, however, policies can overfit to correlations predictive of training actions rather than learn the task-relevant grounding and geometry needed for generalization. Our insight is to guide the representations that connect instructions to actions: identifying what remains to be done determines which visual evidence matters, while understanding the geometry around that evidence informs how to move. We introduce GuidedNav, a unified framework that explicitly shapes these representations through training-time guidance. Subtask and landmark guidance steer native pre-action attention toward the active instruction segment and relevant visual regions, grounding action prediction in the current objective. Spatial guidance aligns the policy's internal representations with geometric features that capture long-horizon context and local spatial structure. Together, these objectives strengthen navigation capabilities while avoiding the inference overhead of intermediate textual reasoning and online geometric encoding. We also introduce Guided-R2R and Guided-RxR, two annotation-enriched navigation datasets that support more interpretable VLN research. GuidedNav establishes state-of-the-art performance among the compared monocular RGB policies, improving success rates by 8.1% and 13.1% relative to the strongest prior results on R2R-CE and RxR-CE val-unseen, respectively, while achieving faster inference. Extensive experiments further demonstrate the effectiveness and real-world transferability of our approach.
Jingyu Guo, Jiaxin Huang, Chengxing Lin +6
University of Melbourne · AgiBot · Mohamed bin Zayed University of Artificial Intelligence +3
Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.
Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2