Enabling embodied agents to imagine future states is essential for robust and generalizable visual navigation. Yet, state-of-the-art systems typically rely on modular designs that decouple navigation planning from visual world modeling, which often induces state-action misalignment and weak adaptability in novel or dynamic scenarios. We propose UniWM, a unified, memory-augmented world model that integrates egocentric visual foresight and planning within a single multimodal autoregressive backbone. UniWM explicitly grounds action selection in visually imagined outcomes, tightly aligning prediction with control. Meanwhile, a hierarchical memory mechanism fuses short-term perceptual cues with longer-term trajectory context, supporting stable and coherent reasoning over extended horizons. Extensive experiments on four challenging benchmarks (Go Stanford, ReCon, SCAND, HuRoN) and the 1X Humanoid Dataset show that UniWM improves navigation success rates by up to 30%, substantially reduces trajectory errors against strong baselines, generalizes zero-shot to the unseen TartanDrive dataset, and scales naturally to high-dimensional humanoid navigation. These results position UniWM as a principled step toward unified, imagination-driven embodied navigation. The code and models are available at https://github.com/UWMILab/UniWM.
Figures & tables
Figure 1: Comparisons among goal-conditioned visual navigation methods. (a) Navigation policy methods like NoMaD ( Sridhar et al., 2024 ) directly predict action sequences AT . (b) World model for navigation like NWM ( Bar et al., 2025 ) uses a world model to visualize future observations, enhancing a separate navigation planner. (c) U n i W M (no memory) unifies planning and visualization within one multimodal backbone, and actions are grounded in the imagined next observation while generating AT autoregressively. (d) U n i W M (with hierarchical memory) adds intra-step and cross-step memory banks, stabilizing longer-horizon rollouts and consistently yielding the highest SR and lowest errors. All panels use the same start/goal observations; headers report navigation SR ↑ , ATE ↓ , and RPE ↓ on HuRoN ( Hirose et al., 2023 ) .
Figure 2: Overview of the U n i W M framework. (a) Training: planner and world-model samples are interleaved within a single unified multimodal autoregressive backbone, optimized jointly with the discretized bin-token loss Lplan and the reconstruction loss Lworld ; bin/text/image tokenizers map actions, pose, and observations to tokens. (b) Inference: a hierarchical memory supplies intra- and cross-step KV states ( Mtintra caches the current observation; Mtcross accumulates prior steps) to augment attention, yielding robust trajectory-consistent alternating predictions of a^t (next action) and o^t (next observation). See Fig. 3 for the detailed memory mechanism.
Figure 3: Overview of hierarchical memory bank mechanism ( Mtintra & Mtcross ). (a) KV (keys/values) extracted from selected layers are deposited into Mtintra at the beginning of each step t (Eq. 7 ). (b)(c) Mtintra is merged with the accumulated cross-step memory Mtcross via top- k similarity gating (Eq. 8 ) and exponential temporal decay (Eq. 9 ), yielding a fused memory (Eq. 10 ) that augments attention for both the planner and the world-model substeps (Eq. 11 ) to promote trajectory-consistent predictions. At the end of step t , Mtintra (with timestamp t ) is appended to Mtcross for reliable reuse at step t+1 , enabling robustly efficient rollouts.
Method
Go Stanford
ReCon
SCAND
HuRoN
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
GNM ( Shah et al., 2022 )
0.27
1.11
0.31
0.72
0.70
0.20
0.49
0.51
0.21
0.36
1.07
0.35
VINT ( Shah et al., 2023 )
0.29
1.09
0.35
0.68
0.84
0.28
0.45
0.58
0.28
0.30
1.19
0.43
NoMaD ( Sridhar et al., 2024 )
0.33
0.94
0.30
0.71
0.77
0.21
0.50
0.54
0.23
0.37
0.92
0.33
Anole-7B ( Chern et al., 2024 )
0.18
2.18
0.73
0.41
1.74
0.69
0.29
1.37
0.71
0.20
1.92
0.78
NWM ( Bar et al., 2025 )
0.45
0.80
0.27
0.79
0.58
0.17
0.55
0.41
0.19
0.41
0.73
0.28
Table 1: Comparisons with SOTA Methods upon Goal-Conditioned Visual Navigation on evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN with SR, ATE, and RPE. Best results for each metric are in bold.
Figure 4: Qualitative Comparisons on Go Stanford and HuRoN datasets across UniWM, NWM, and NoMaD. Qualitative results here include both static indoor environments and outdoor scenarios with moving pedestrians. The central trajectory plots highlight the difference between predicted AT and the ground-truth.
Method
SSIM ↑
PSNR ↑
LPIPS ↓
DreamSIM ↓
SSIM @5↑
PSNR @5↑
LPIPS @5↓
DreamSIM @5↓
Diamond ( Alonso et al., 2024 )
0.311
9.837
0.410
0.131
0.186
6.352
0.582
0.252
Aether ( Zhu et al., 2025b )
0.373
10.952
0.271
0.068
0.235
7.183
0.514
0.158
NWM ( Bar et al., 2025 )
0.389
11.420
0.318
0.089
0.256
7.755
0.494
0.174
U n i W M
0.457
13.607
0.254
0.041
0.350
10.874
0.435
0.126
Table 2: Comparisons with SOTA methods on visual quality assessment , averaged over evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN. Best results for each metric are in bold.
Context
Token Len.
Navigation
Visualization
SR ↑
ATE ↓
RPE ↓
SSIM ↑
LPIPS ↓
DreamSIM ↓
SSIM @5↑
LPIPS @5↓
DreamSIM @5↓
1
784
0.71
0.36
0.13
0.457
0.254
0.041
0.350
0.435
0.126
2
625
0.68
0.39
0.15
0.448
0.247
0.051
0.336
0.451
0.137
2
484
0.55
0.53
0.26
0.365
0.328
0.084
0.258
0.515
0.192
4
484
0.64
0.44
0.19
0.425
0.285
0.052
0.315
0.462
0.141
Table 3: Impact of Context Size and Image Token Length on both navigation and visualization performance, averaged over four datasets. All settings are evaluated without memory banks. Best results for each metric are highlighted in bold .
Method
SR ↑
ATE ↓
RPE ↓
SSIM ↑
LPIPS ↓
DreamSIM ↓
SSIM @5↑
LPIPS @5↓
DreamSIM @5↓
UniWM (Separate)
0.65
0.41
0.16
0.443
0.280
0.055
0.329
0.470
0.154
UniWM
0.71
0.36
0.13
0.457
0.254
0.041
0.350
0.435
0.126
Table 4: Comparison of the unified UniWM and the separate planner + world-model setting on both navigation (SR, ATE, RPE) and visualization metrics (SSIM, LPIPS, DreamSim). Best results for each metric are highlighted in bold .
Figure 5: Impact of discretized bin-token loss ( Lplan ) and reconstruction Loss ( Lworld ) on navigation (left) and visualization (right) performance, averaged over evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN. X-axis arrows indicate whether higher or lower values are preferable.
Layer Num
Go Stanford
ReCon
SCAND
HuRoN
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
1
0.71
0.30
0.10
0.84
0.36
0.12
0.62
0.35
0.14
0.72
0.41
0.14
3
0.74
0.27
0.09
0.89
0.35
0.11
0.66
0.33
0.13
0.75
0.39
0.14
5
0.75
0.22
0.09
0.93
0.34
0.11
0.68
0.32
0.13
0.76
0.38
0.13
7
0.74
0.25
0.09
0.91
0.35
0.12
0.69
0.31
0.13
0.74
0.38
0.14
16
0.61
0.46
0.23
0.70
0.58
0.26
0.52
0.49
0.24
0.57
0.55
0.22
Table 5: Impact of number of selected layers included in memory bank on navigation performance of UniWM (with Mtintra & Mtcross ) on evaluation splits of four in-domain datasets. Best results for each metric are in bold.
Step Strategy
Go Stanford
ReCon
SCAND
HuRoN
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
SR ↑
ATE ↓
RPE ↓
Predict both
0.65
0.38
0.13
0.80
0.41
0.15
0.57
0.39
0.17
0.63
0.47
0.20
Interleave
0.71
0.32
0.10
0.82
0.35
0.12
0.61
0.36
0.14
0.70
0.42
0.15
Table 6: Comparison of navigation performance under different step strategies across four datasets.
Figure 12Figure 13
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Prompt design details and examples on action prediction (context size = 1).
Table A1 : Comparison of average inference time per trajectory together with navigation metrics (SR, ATE, RPE) across the four datasets (Go Stanford, ReCon, SCAND, HuRoN).
Figure A3 : Qualitative Comparisons on Go Stanford across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted AT and GT.
Figure A4 : Qualitative Comparisons on ReCon and Scand across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted AT and GT.
Figure A5 : Qualitative Comparisons on HuRoN across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted AT and GT.
Conventional visual navigation policies often struggle with myopic decision-making and mode collapse in complex environments. While world models offer a promising alternative, existing paradigms typically isolate perception, generation, and control, failing to capture their shared spatio-temporal dynamics. In this paper, we propose NavWM, a unified navigation world model that seamlessly integrates latent world reasoning, multimodal action prediction, and controllable visual generation. At its core, NavWM leverages latent world tokens to distill geometric and semantic priors, endowing the agent with robust structural understanding. To overcome the limitations of deterministic policies, we introduce an anchor-based multimodal trajectory forecasting framework that generates a diverse action space. This inherent diversity explicitly empowers the generative world model to act as a robust closed-loop planner, utilizing visual foresight to evaluate and select the optimal path. Extensive experiments across diverse robotics datasets demonstrate that NavWM significantly advances the state-of-the-art, delivering remarkable improvements in both high-fidelity future state generation and zero-shot navigation success.
Yanghong Mei, Longteng Guo, Ming-Ming Yu +3
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beihang University
Image-goal visual navigation is a fundamental capability for embodied agents. Existing navigation policies efficiently predict waypoint trajectories but lack visual foresight, while navigation world models can anticipate future observations but often require costly planning rollouts. We present UniNav, a unified world-action model that generates future visual observations and continuous waypoint trajectories through a single diffusion process. Given history frames and a goal image, UniNav jointly denoises visual and waypoint tokens within a single transformer, unifying future prediction and action generation in a shared framework. To improve spatial grounding, we incorporate geometry-aware camera tokens. We also train on both trajectory-labeled navigation data and video-only data, enabling the model to benefit from diverse videos without waypoint annotations. Based on this unified framework, we introduce two variants: UniNav-Full jointly predicts interpretable future observations and their corresponding trajectories, while UniNav-Fast removes future-image tokens at inference for efficient trajectory prediction. Experiments on navigation benchmarks show that UniNav outperforms the strongest baseline in ATE across all datasets. With one-step inference, UniNav-Fast achieves a latency of 0.1s without a substantial accuracy drop. Code will be released.
Changqing Zhou, Yueru Luo, Zeyu Jiang +1
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Chinese University of Hong Kong, Shenzhen
Visual navigation requires generating smooth and collision-free trajectories under complex geometric and physical constraints. Existing reactive policies that directly map observations to actions lack anticipatory reasoning, limiting their ability to proactively avoid obstacles. While visual imagination offers predictive foresight, conventional modular approaches separate scene prediction from policy learning, often leading to error accumulation and inefficient inference. To address these limitations, we propose WAM-Nav, a Latent World-Action Model for embodied visual navigation that jointly learns action generation and latent visual foresight, enabling more robust and foresighted navigation decisions without compromising inference efficiency. Specifically, WAM-Nav utilizes a shared Diffusion Transformer for asymmetric joint diffusion to concurrently generate long-horizon actions and short-horizon visual foresight, reducing the inference latency and visual error accumulation inherent in multi-step autoregressive rollouts. To further encourage smooth and consistent trajectory generation, we introduce a dual-stream contextual conditioning mechanism that integrates episode-level ego-motion history with sequential visual observations. Combined with a unified goal alignment module that preserves balanced representations across goal types, WAM-Nav naturally supports Image-Goal, Point-Goal, and No-Goal exploration within a single policy. Extensive experiments on the challenging ClutterScenes and InternScenes benchmarks demonstrate strong generalization of WAM-Nav, particularly on Image-Goal and Point-Goal navigation, where it improves success rates by 15.7% and 3.3%, respectively. Real-world deployment further validates effective zero-shot sim-to-real transfer, achieving an average 85% task success rate across diverse indoor and outdoor environments.
Ning Yang, Yan Huang, Kaiwen Peng +9
1Nanjing University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +4