Navigation world models predict future observations conditioned on navigation actions to support planning, but their generalization across real-world environments remains constrained by the limited coverage of action-labeled data. Action-free navigation videos provide diverse visual experience, yet lack the recorded actions required for action-conditioned training. We introduce OpenNWM, a generalist navigation world model with latent action pretraining. It learns universal latent actions from large-scale action-free videos through visual prediction pretraining, and further aligns them with physical motions via action-conditioned post-training. To support video pretraining, we curate NavAnywhere, a large-scale action-free navigation video dataset comprising 17.5 million visual observations from 15 heterogeneous sources. It captures navigation by ground robots, humans, and drones, with diverse viewpoints and motion patterns across five broad scene types: residential areas, public and commercial spaces, urban settings, parks, and natural landscapes. Experiments demonstrate improved visual prediction on in-domain and out-of-domain benchmarks, together with improved navigation performance. These results show that latent-action pretraining enables diverse, action-free videos to improve dynamics learning and generalization for open-world navigation.
Figures & tables
Figure 1: Overview of OpenNWM. OpenNWM learns navigation dynamics from heterogeneous action-free videos via latent-action pretraining, then adapts to real waypoint actions for action-conditioned prediction and planning.
Figure 2: Framework of OpenNWM.
Figure 3: Three-stage training pipeline of OpenNWM. (a) Latent-action pretraining on action-free NavAnywhere videos. (b) Action-encoder warmup with the CDiT backbone frozen. (c) Joint post-training of the entire model.
Table 4Figure 5
Figure 5: Latent-action clustering under four LAM supervision paradigms.
Table 7
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Overview of NavAnywhere and its complementary out-of-domain evaluation sets. NavAnywhere covers diverse everyday navigation environments. The complementary evaluation sets are separate from NavAnywhere and are described in Section B.2 .
Figure 7: Source composition of NavAnywhere. Statistics cover the 15 action-free sources.
Dataset
Sequences
Frames
Hours
Ego4D
1,619
13,088,098
908.95
DL3DV-10K
20,352
1,180,382
81.36
KrishnaCam
436
996,666
69.21
RealEstate10K
45,010
712,145
49.44
LAVN
312
443,244
30.78
Walking Tours
10
198,097
14.40
Appendix
Table 6: NavAnywhere dataset statistics. Sequences denote processed video clips or trajectories, and frames denote sampled RGB images. Hours measure source-video duration. All 15 sources are used as RGB-only, action-free data.
Dataset
Sequences
Frames
Data used
RECON
11,835
610,911
RGB + position/yaw
HuRoN
2,839
230,911
RGB + position/yaw
Go Stanford
3,696
198,126
RGB + position/yaw
SCAND
604
103,457
RGB + position/yaw
TartanDrive
1,251
62,884
RGB + position/yaw
Appendix
Table 7: Statistics of the action-annotated datasets. Sequences denote processed trajectories, and frames denote sampled RGB images. Go Stanford is used only for evaluation.
Module
Configuration
Value
Latent Action Model
Architecture
Pixel+Action spatiotemporal Transformer
Model size
∼709 M
Encoder / decoder blocks
24 / 24
Hidden width / attention heads
1,024 / 16
Latent-action dimension
32
Input frames
2
Appendix
Table 8: Training and inference configurations of OpenNWM. Batch sizes are global, and image resolutions are reported as height × width.
Figure 8: Scaling unlabeled data ablation. The horizontal axis gives the percentage of the unlabeled data pool used for latent-action pretraining, with action-label availability fixed.
Method
RECON
SCAND
HuRoN
TartanDrive
LPIPS ↓
DS ↓
PSNR ↑
LPIPS ↓
DS ↓
PSNR ↑
LPIPS ↓
DS ↓
PSNR ↑
LPIPS ↓
DS ↓
PSNR ↑
NWM
0.323
0.120
15.487
0.406
0.300
12.778
0.463
0.275
13.377
0.427
0.223
13.602
NWM XL (with Ego4D)
0.359
0.140
14.850
0.394
0.260
13.068
0.501
0.295
12.604
0.424
0.213
13.569
RAE-NWM
0.353
0.134
15.491
0.413
0.190
14.419
0.339
0.171
15.079
0.662
0.519
10.840
OpenNWM
0.320
0.116
15.614
0.385
0.271
13.247
0.362
0.193
14.329
0.405
0.203
14.005
Appendix
Table 9: In-distribution (ID) direct visual prediction at the nominal 4 s horizon. DS: DreamSim. Best and second-best results are bold and underlined, respectively.
Method
RECON
SCAND
TartanDrive
ATE ↓
RPE ↓
ATE ↓
RPE ↓
ATE ↓
RPE ↓
GNM
1.87
0.73
2.12
0.61
6.65
1.62
NoMaD
1.95
0.53
2.24
0.49
6.32
1.31
NWM
1.13
0.35
1.01
0.28
5.72
1.19
RAE-NWM
1.36
0.37
1.14
0.28
5.99
1.25
Garrido et al.
1.40
0.40
–
–
–
–
Appendix
Table 10: Navigation performance on RECON, SCAND, and TartanDrive. Best and second-best available results are bold and underlined, respectively.
Figure 9: Four latent action learning paradigms. An encoder maps a visual transition to a variational latent zt . The prediction targets are (a) the observed action at , (b) the future image xt+1 , (c) frozen DINOv3 features ft+1 , and (d) both the image and action. Reconstruction decoders additionally receive the current image or its features. In (d), pixel reconstruction uses all samples, while the action loss uses only labeled samples. All variants include the same form of KL regularization.
Figure 10: Visualization of the semantic separability of latent actions learned by four LAM paradigms across different navigation action categories. TartanDrive and Go Stanford are out-of-domain datasets.
Figure 11: Video generation examples on RECON. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.
Figure 12: Video generation examples on HuRoN. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.
Figure 13: Video generation examples on TartanDrive. NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 0 to 16 seconds, every 2 second.
Figure 14: Video generation examples on Go Stanford (OOD). NWM, RAE-NWM and OpenNWM are conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 4 seconds at 4 FPS. We plot the generated results from 0 to 4 seconds, every 1 second.
Figure 15: Qualitative results of our LAM on HuRoN. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error ( ∘ ) are reported for each example.
Figure 16: Qualitative results of our LAM on RECON. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error ( ∘ ) are reported for each example.
Figure 17: Qualitative results of our LAM on SCAND. Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error ( ∘ ) are reported for each example.
Figure 18: Qualitative results of our LAM on Go Stanford (OOD). Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error ( ∘ ) are reported for each example.
Figure 19: Qualitative results of our LAM on TartanDrive (OOD). Each row shows the input image, ground-truth target image, reconstruction, and ground-truth versus decoded actions. Endpoint error (EPE, m) and yaw error ( ∘ ) are reported for each example.
Figure 20: Qualitative navigation results of OpenNWM on HuRoN. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 21: Qualitative navigation results of OpenNWM on HuRoN. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 22: Qualitative navigation results of OpenNWM on Office-go2 (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 23: Qualitative navigation results of OpenNWM on Office-go2 (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 24: Qualitative navigation results of OpenNWM on RECON. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 25: Qualitative navigation results of OpenNWM on RECON. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 26: Qualitative navigation results of OpenNWM on SCAND. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 27: Qualitative navigation results of OpenNWM on SCAND. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 28: Qualitative navigation results of OpenNWM on Go Stanford (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 29: Qualitative navigation results of OpenNWM on Go Stanford (OOD). The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 30: Qualitative navigation results of OpenNWM on TartanDrive. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Figure 31: Qualitative navigation results of OpenNWM on TartanDrive. The top row shows the start and goal images alongside ground-truth and predicted trajectories, with ATE and goal error reported in meters. The bottom rows compare ground-truth and predicted observations at 0.5-second intervals over a 2-second horizon.
Direct visual navigation policies generate trajectories efficiently but do not explicitly evaluate their future consequences. Generative navigation world models provide this foresight through visual rollouts, which are costly when evaluating multiple candidates. We present LiteNWM, a latent navigation world model that shares visual encoding across candidates and jointly predicts their action-conditioned future representations at multiple horizons, while a learned scorer uses these predictions to select trajectories. In offline evaluations on RECON, SCAND, and SACSoN, LiteNWM reduces macro-averaged trajectory error by 17.56% relative to NoMaD+NWM-XL and achieves a 128.00-fold end-to-end speedup on an RTX 5090. The same evaluator transfers from NoMaD to MBRA without proposer-specific retraining, reducing MBRA's macro-averaged trajectory error by 16.2%. In real-robot experiments in unseen indoor and outdoor environments, LiteNWM improves navigation success from 43.3% to 83.3% relative to NoMaD. These results demonstrate that LiteNWM can be deployed for future-aware planning and closed-loop navigation on a physical robot.
Linkai Liu, Yuntian Zhang, Zhenshan Bing +5
Nanjing University · Imperial College London · Sony AI +1
Goal-conditioned visual navigation requires a robot to act under partial observability by anticipating how its motion will change the future egocentric view and whether that change brings it closer to the goal. Navigation world models provide such visual foresight, but they remain prediction modules that require an external planner to convert predicted futures into closed-loop control. We propose Navigation World Action Model (NavWAM), a diffusion-transformer policy that turns navigation world-model prediction into executable action by representing future observations, goal-progress values, and action chunks in a shared latent sequence. By learning future prediction jointly with the action and value targets that determine closed-loop behavior, NavWAM makes visual foresight directly usable for robot control. We build NavWAM through simulation pretraining and real-robot adaptation, and evaluate it on image-goal navigation against planning-based world models and a representative direct navigation policy. Across offline benchmarks and closed-loop real-robot deployment, NavWAM improves over planning-based world-model baselines in our evaluations while using the default policy mode without CEM-style action search. Project page: https://dachii-azm.github.io/navwam/
Daichi Azuma, Taiki Miyanishi, Koya Sakamoto +6
The University of Tokyo · National Institute of Informatics · AIRoA +1
World action models integrate future visual dynamics with robot action prediction, but their scalability remains limited by the need for action-annotated robot trajectories. Observation-only videos contain rich evidence about interaction dynamics, but existing approaches typically use them either to pretrain visual representations that must later be adapted for control, or to infer latent actions that are subsequently grounded to robot commands. We present NAVA-WAM, which introduces native action-prior learning by directly pretraining the action policy from observation-only videos, avoiding indirect representation-to-control transfer or a separate latent-action model. Our training consists of two stages. First, we pretrain on observation-only videos, where future-video flow-matching supervision over visual transitions is propagated through transition-structured joint attention to optimize the Action-DiT and learn action-relevant priors. Second, we use action-labeled demonstrations to post-train the Action-DiT for robot control through joint video--action flow matching, while asymmetric attention decouples the visual branch from iterative action denoising and enables efficient action-only inference. Extensive experiments show that NAVA-WAM consistently outperforms prior approaches under both in-distribution and out-of-distribution settings, while demonstrating strong action-label efficiency and effective real-robot generalization. These results establish native action-prior learning as an effective approach to directly pretrain action policies from observation-only videos, providing a scalable path beyond action-labeled robot data.
Zhaochong An, Fei Zhang, Menglin Jia +10
Meta AI · University of Copenhagen · Physical Intelligence +1