Existing vision-and-language navigation (VLN) models primarily reason over past and current visual observations, while largely ignoring the future visual dynamics induced by actions. As a result, they often lack an effective understanding of the causal relationship between actions and how the visual world changes, limiting robust decision-making. Humans, in contrast, can imagine the near future by leveraging action-dynamics causality, which improves both environmental understanding and navigation choices. Inspired by this capability, we propose LatentPilot, a new paradigm that exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics, while requiring no access to future frames at inference. Concretely, we propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution, with an expert takeover triggered when the agent deviates excessively. LatentPilot further learns visual latent tokens without explicit supervision; these latent tokens attend globally in a continuous latent space and are carried across steps, serving as both the current output and the next input, thereby enabling the agent to dream ahead and reason about how actions will affect subsequent observations. Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene. Project page:https://abdd.top/latentpilot/
Figures & tables
Figure 1: Prior VLN pipelines optimize next-action prediction from past/current observations, leaving future observations in trajectories largely unused as supervision. LatentPilot uses future observations only as training-time privileged supervision to internalize action-conditioned visual dynamics while inference remains strictly causal and requires neither future frames nor an external world-model rollout.
Figure 2: Overview of LatentPilot. The vision encoder extracts visual tokens from the current observation, the LLM backbone predicts the action, and a lightweight Pilot module updates a propagated Pilot Token that is cached and reused as the Pilot slot input at the next step, enabling strictly causal dreaming-ahead.
Figure 3: PilotLoop: flywheel-style closed-loop training. Each round iterates rollout collection, expert takeover for deviation-aware correction, future-privileged target construction, and fine-tuning with joint action imitation and Pilot supervision.
Figure 4: Visibility matrix of LatentPilot.
Method
Observation
R2R Val-Unseen
RxR Val-Unseen
Pano.
Odo.
D.
SRGB
NE ↓
OS ↑
SR ↑
SPL ↑
NE ↓
SR ↑
SPL ↑
nDTW ↑
CMA [ 14 ]
✓
✓
✓
6.20
52.0
41.0
36.0
8.76
26.5
22.1
47.0
VLN-BERT [ 14 ]
✓
✓
✓
5.74
53.0
44.0
39.0
8.98
27.0
22.6
46.7
Sim2Sim [ 24 ]
✓
✓
✓
6.07
52.0
43.0
36.0
–
–
–
–
Ego2-Map [ 15 ]
✓
✓
✓
5.54
56.0
47.0
41.0
–
–
–
–
DreamWalker [ 47 ]
✓
✓
✓
5.53
59.0
49.0
44.0
–
–
–
–
Table 1: Comparison with state-of-the-art methods on VLN-CE R2R and RxR Val-Unseen splits. Pano, Odo and D respectively represent panoramic view, odometry and depth, SRGB denotes monocular RGB.
Method
R2R Validation Seen
R2R Validation Unseen
NE ↓
FR ↓
StR ↓
OS ↑
SR ↑
SPL ↑
NE ↓
FR ↓
StR ↓
OS ↑
SR ↑
SPL ↑
Train-free Map-based Exploration and Navigation
VLMaps [ 16 ]
–
–
–
–
–
–
6.98
23.00
0.00
20.00
20.00
12.70
Train on VLN-PE
Seq2Seq [ 25 ]
7.73
22.19
3.04
30.55
19.60
15.67
7.91
19.67
3.71
27.62
15.89
12.58
Seq2Seq+ [ 25 ]
7.54
26.11
5.59
31.93
19.58
15.13
7.64
21.82
5.12
30.47
18.13
14.06
Table 2: Evaluation Metrics on VLN-PE benchmark with physical locomotion controller. +: model is first trained on Habitat and fine-tuned on VLN-PE. † : model is trained with data augmentation.
Modality
R2R Validation Unseen
R2R Validation Seen
SR ↑
SPL ↑
NE ↓
OS ↑
SR ↑
SPL ↑
NE ↓
OS ↑
NaN
51.7
47.1
5.3
57.0
53.1
46.9
4.6
60.7
3D
50.3
45.6
5.4
55.8
51.5
45.1
4.9
59.9
Text
53.2
48.7
5.2
59.6
54.9
48.6
4.5
63.9
Vision(ours)
62.0
58.0
4.4
66.3
65.1
59.6
4.3
71.6
Table 3: Ablation study of supervision modality for the Pilot Token. NaN removes any extra Pilot supervision and optimizes only the action cross-entropy, serving as a baseline that tests whether the propagated latent becomes predictive by itself.
World Model
T/Act (ms) ↓
Mem ↓ (GB)
R2R Val-Seen
SR ↑
SPL ↑
NE ↓
OS ↑
Wan2.1 1.3B [ 46 ]
2040
41.5
56.6
51.7
5.1
64.5
CogVideoX1.5 5B [ 57 ]
5460
32.4
59.3
53.9
4.9
66.8
Ours
130
22.8
65.1
59.6
4.3
71.6
Table 4: Internalized future supervision (ours) vs. plug-in video world models on R2R Val-Seen. T/Act: ms per action; Mem: peak GPU memory (GB).
Figure 5: Performance over PilotLoop.
Figure 6: Latent collapse Analysis via PCA.
Figure 7: Real-world Cross-Embodiment Experiments: Wheeled and Quadruped Robots. LatentPilot successfully follows long, multi-step instructions on both Robots.
Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.
Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2
Adapting vision-language navigation (VLN) policies to new environments is expensive because every additional route and instruction requires an embodied demonstration. Yet standard observation-to-action training uses only a small fraction of the information already contained in each trajectory. In particular, future observations reveal the instruction-relevant landmarks that the agent will encounter, including what they look like and how they are arranged in 3D. We introduce FINE, a Future-Informed Navigation Encoding framework that extracts this latent supervision from existing demonstrations. FINE equips a VLN backbone with two complementary auxiliary representations. First, explicit landmark tokens follow the ordered landmarks specified by the instruction and are trained to predict future landmark regions in both semantic 2D patch-feature space and viewpoint-dependent 3D geometric feature space. Second, an implicit future token learns to distinguish the landmark state that is actually reached from plausible same-scene counterfactual futures generated by a video world model. On R2R-CE and RxR-CE val-unseen, FINE improves InternVLA-N1 by 2.6 and 4.5 success-rate points, respectively, at full training data. More importantly, as demonstrations become limited, the benefit grows: at a 70% demonstration budget, FINE improves success rate by 6.8 points, recovering roughly one-third of the performance lost by reducing the training demonstrations. Project page is available at https://finevln.github.io/.
Khang H. Nguyen, Hoang Pham Quang Nguyen, Ha Phuong Nguyen +6
VinRobotics · University of California, Los Angeles (UCLA) · Center for AI Research, VinUniversity, Vietnam +4
End-to-end vision-language navigation (VLN) with causal vision-language models maps instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly encourage the policy state to be predictive of future visual outcomes, limiting long-horizon decision making. A privileged-input diagnostic shows that access to an expert-trajectory future image can substantially improve navigation, indicating that future observations contain rich, actionable cues, though such inputs are unavailable at deployment. Motivated by this signal, we propose Future-State-Conditioned VLN (FSC-VLN), a deployable model that augments a causal policy with a future-query token and uses training-only future-state supervision to distill information from future observations into the policy state. Concretely, during training we align the future-query representation to a frozen visual embedding Δ steps ahead, while inference requires only past and current observations. This design preserves the baseline inference pattern and adds only two learned prefix tokens, implying minimal overhead. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design that separates future and action queries.
Lingfeng Zhang, Zhanguang Zhang, Liheng Ma +2
Noah’s Ark Lab, 2012 Labs, Huawei · Department of Foundation model, 2012 Labs, Huawei