Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight

Yifei Dong; Fengyi Wu; Guangyu Chen; Lingdong Kong; Qiyu Hu; Yuxuan Zhou; Xu Zhu; Jingdong Sun

cs.AIOct 9, 2025

Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight

Authors: Yifei Dong, Fengyi Wu, Guangyu Chen, Lingdong Kong, Qiyu Hu, Yuxuan Zhou, Xu Zhu, Jingdong Sun,

Organizations: University of Washington · National University of Singapore · Apple · Microsoft Research ·

Abstract

Enabling embodied agents to imagine future states is essential for robust and generalizable visual navigation. Yet, state-of-the-art systems typically rely on modular designs that decouple navigation planning from visual world modeling, which often induces state-action misalignment and weak adaptability in novel or dynamic scenarios. We propose UniWM, a unified, memory-augmented world model that integrates egocentric visual foresight and planning within a single multimodal autoregressive backbone. UniWM explicitly grounds action selection in visually imagined outcomes, tightly aligning prediction with control. Meanwhile, a hierarchical memory mechanism fuses short-term perceptual cues with longer-term trajectory context, supporting stable and coherent reasoning over extended horizons. Extensive experiments on four challenging benchmarks (Go Stanford, ReCon, SCAND, HuRoN) and the 1X Humanoid Dataset show that UniWM improves navigation success rates by up to 30%, substantially reduces trajectory errors against strong baselines, generalizes zero-shot to the unseen TartanDrive dataset, and scales naturally to high-dimensional humanoid navigation. These results position UniWM as a principled step toward unified, imagination-driven embodied navigation. The code and models are available at https://github.com/UWMILab/UniWM.

Figures & tables

Figure 1: Comparisons among goal-conditioned visual navigation methods. (a) Navigation policy methods like NoMaD ( Sridhar et al., 2024 ) directly predict action sequences $A_{T}$ . (b) World model for navigation like NWM ( Bar et al., 2025 ) uses a world model to visualize future observations, enhancing a separate navigation planner. (c) U n i W M (no memory) unifies planning and visualization within one multimodal backbone, and actions are grounded in the imagined next observation while generating $A_{T}$ autoregressively. (d) U n i W M (with hierarchical memory) adds intra-step and cross-step memory banks, stabilizing longer-horizon rollouts and consistently yielding the highest SR and lowest errors. All panels use the same start/goal observations; headers report navigation SR $\uparrow$ , ATE $\downarrow$ , and RPE $\downarrow$ on HuRoN ( Hirose et al., 2023 ) .

Figure 2: Overview of the U n i W M framework. (a) Training: planner and world-model samples are interleaved within a single unified multimodal autoregressive backbone, optimized jointly with the discretized bin-token loss $\mathcal{L}_{\text{plan}}$ and the reconstruction loss $\mathcal{L}_{\text{world}}$ ; bin/text/image tokenizers map actions, pose, and observations to tokens. (b) Inference: a hierarchical memory supplies intra- and cross-step KV states ( $\mathcal{M}^{\text{intra}}_{t}$ caches the current observation; $\mathcal{M}^{\text{cross}}_{t}$ accumulates prior steps) to augment attention, yielding robust trajectory-consistent alternating predictions of $\hat{a}_{t}$ (next action) and $\hat{o}_{t}$ (next observation). See Fig. 3 for the detailed memory mechanism.

Figure 3: Overview of hierarchical memory bank mechanism ( $\mathcal{M}^{\text{intra}}_{t}$ & $\mathcal{M}^{\text{cross}}_{t}$ ). (a) $KV$ (keys/values) extracted from selected layers are deposited into $\mathcal{M}^{\text{intra}}_{t}$ at the beginning of each step $t$ (Eq. 7 ). (b)(c) $\mathcal{M}^{\text{intra}}_{t}$ is merged with the accumulated cross-step memory $\mathcal{M}^{\text{cross}}_{t}$ via top- $k$ similarity gating (Eq. 8 ) and exponential temporal decay (Eq. 9 ), yielding a fused memory (Eq. 10 ) that augments attention for both the planner and the world-model substeps (Eq. 11 ) to promote trajectory-consistent predictions. At the end of step $t$ , $\mathcal{M}^{\text{intra}}_{t}$ (with timestamp $t$ ) is appended to $\mathcal{M}^{\text{cross}}_{t}$ for reliable reuse at step $t{+}1$ , enabling robustly efficient rollouts.

Method	Go Stanford			ReCon			SCAND			HuRoN
Method	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$
GNM ( Shah et al., 2022 )	0.27	1.11	0.31	0.72	0.70	0.20	0.49	0.51	0.21	0.36	1.07	0.35
VINT ( Shah et al., 2023 )	0.29	1.09	0.35	0.68	0.84	0.28	0.45	0.58	0.28	0.30	1.19	0.43
NoMaD ( Sridhar et al., 2024 )	0.33	0.94	0.30	0.71	0.77	0.21	0.50	0.54	0.23	0.37	0.92	0.33
Anole-7B ( Chern et al., 2024 )	0.18	2.18	0.73	0.41	1.74	0.69	0.29	1.37	0.71	0.20	1.92	0.78
NWM ( Bar et al., 2025 )	0.45	0.80	0.27	0.79	0.58	0.17	0.55	0.41	0.19	0.41	0.73	0.28

Table 1: Comparisons with SOTA Methods upon Goal-Conditioned Visual Navigation on evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN with SR, ATE, and RPE. Best results for each metric are in bold.

Figure 4: Qualitative Comparisons on Go Stanford and HuRoN datasets across UniWM, NWM, and NoMaD. Qualitative results here include both static indoor environments and outdoor scenarios with moving pedestrians. The central trajectory plots highlight the difference between predicted $A_{T}$ and the ground-truth.

Method	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	DreamSIM $\downarrow$	SSIM $@5$ $\uparrow$	PSNR $@5$ $\uparrow$	LPIPS $@5$ $\downarrow$	DreamSIM $@5$ $\downarrow$
Diamond ( Alonso et al., 2024 )	0.311	9.837	0.410	0.131	0.186	6.352	0.582	0.252
Aether ( Zhu et al., 2025b )	0.373	10.952	0.271	0.068	0.235	7.183	0.514	0.158
NWM ( Bar et al., 2025 )	0.389	11.420	0.318	0.089	0.256	7.755	0.494	0.174
U n i W M	0.457	13.607	0.254	0.041	0.350	10.874	0.435	0.126

Table 2: Comparisons with SOTA methods on visual quality assessment , averaged over evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN. Best results for each metric are in bold.

Context	Token Len.	Navigation			Visualization
Context	Token Len.	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DreamSIM $\downarrow$	SSIM $@5$ $\uparrow$	LPIPS $@5$ $\downarrow$	DreamSIM $@5$ $\downarrow$
1	784	0.71	0.36	0.13	0.457	0.254	0.041	0.350	0.435	0.126
2	625	0.68	0.39	0.15	0.448	0.247	0.051	0.336	0.451	0.137
2	484	0.55	0.53	0.26	0.365	0.328	0.084	0.258	0.515	0.192
4	484	0.64	0.44	0.19	0.425	0.285	0.052	0.315	0.462	0.141

Table 3: Impact of Context Size and Image Token Length on both navigation and visualization performance, averaged over four datasets. All settings are evaluated without memory banks. Best results for each metric are highlighted in bold .

Method	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DreamSIM $\downarrow$	SSIM $@5$ $\uparrow$	LPIPS $@5$ $\downarrow$	DreamSIM $@5$ $\downarrow$
UniWM (Separate)	0.65	0.41	0.16	0.443	0.280	0.055	0.329	0.470	0.154
UniWM	0.71	0.36	0.13	0.457	0.254	0.041	0.350	0.435	0.126

Table 4: Comparison of the unified UniWM and the separate planner + world-model setting on both navigation (SR, ATE, RPE) and visualization metrics (SSIM, LPIPS, DreamSim). Best results for each metric are highlighted in bold .

Panel of Figure 5 — Figure 5: Impact of discretized bin-token loss ( $\mathcal{L}_{\text{plan}}$ ) and reconstruction Loss ( $\mathcal{L}_{\text{world}}$ ) on navigation (left) and visualization (right) performance, averaged over evaluation splits of Go Stanford, ReCon, SCAND, and HuRoN. X-axis arrows indicate whether higher or lower values are preferable.

Layer Num	Go Stanford			ReCon			SCAND			HuRoN
	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$
1	0.71	0.30	0.10	0.84	0.36	0.12	0.62	0.35	0.14	0.72	0.41	0.14
3	0.74	0.27	0.09	0.89	0.35	0.11	0.66	0.33	0.13	0.75	0.39	0.14
5	0.75	0.22	0.09	0.93	0.34	0.11	0.68	0.32	0.13	0.76	0.38	0.13
7	0.74	0.25	0.09	0.91	0.35	0.12	0.69	0.31	0.13	0.74	0.38	0.14
16	0.61	0.46	0.23	0.70	0.58	0.26	0.52	0.49	0.24	0.57	0.55	0.22

Table 5: Impact of number of selected layers included in memory bank on navigation performance of UniWM (with

\mathcal{M}^{\text{intra}}_{t}

\mathcal{M}^{\text{cross}}_{t}

) on evaluation splits of four in-domain datasets. Best results for each metric are in bold.

Step Strategy	Go Stanford			ReCon			SCAND			HuRoN
Step Strategy	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$	SR $\uparrow$	ATE $\downarrow$	RPE $\downarrow$
Predict both	0.65	0.38	0.13	0.80	0.41	0.15	0.57	0.39	0.17	0.63	0.47	0.20
Interleave	0.71	0.32	0.10	0.82	0.35	0.12	0.61	0.36	0.14	0.70	0.42	0.15

Table 6: Comparison of navigation performance under different step strategies across four datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 14 — Figure A1 : Prompt design details and examples on action prediction (context size = 1).

Panel of Figure 15 — Figure A2 : Prompt design examples on navigation visualization (context size = 1).

Method	Avg SR	Avg ATE	Avg RPE	Avg Inference Time per Trajectory (second)
NoMaD ( Sridhar et al., 2024 )	0.48	0.79	0.26	0.6
NWM ( Bar et al., 2025 )	0.55	0.63	0.23	114 $\times$ 32
Anole-7B ( Chern et al., 2024 )	0.27	1.80	0.73	654
UniWM (with Memory)	0.78	0.32	0.12	82
UniWM + Quant. 4-bit (with Memory)	-	-	-	16 (est. ( Frantar et al., 2022 ) )

Appendix

Table A1 : Comparison of average inference time per trajectory together with navigation metrics (SR, ATE, RPE) across the four datasets (Go Stanford, ReCon, SCAND, HuRoN).

Figure A3 : Qualitative Comparisons on Go Stanford across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted $A_{T}$ and GT.

Figure A4 : Qualitative Comparisons on ReCon and Scand across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted $A_{T}$ and GT.

Figure A5 : Qualitative Comparisons on HuRoN across UniWM, NWM, and NoMaD. Central trajectory plots highlight the difference between predicted $A_{T}$ and GT.

Embodied Navigation

Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.

See all 138 papers

NavWM: A Unified Navigation World Model for Foresight-Driven Planning
Jun 23, 2026Yanghong Mei, Longteng Guo, Ming-Ming Yu +3Visual Navigation World Models for Robotics
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beihang University
UniNav: A Unified World-Action Diffusion Model for Visual Navigation
Aug 4, 2026Changqing Zhou, Yueru Luo, Zeyu Jiang +1Visual Navigation Trajectory Generation
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Chinese University of Hong Kong, Shenzhen
WAM-Nav: Asymmetric Latent World-Action Modeling for Unified Visual Navigation
Jun 3, 2026Ning Yang, Yan Huang, Kaiwen Peng +9Visual Navigation Robot Policy Learning
1Nanjing University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +4

Jun 23, 2026J/K move · Enter open · S save

NavWM: A Unified Navigation World Model for Foresight-Driven Planning

Yanghong Mei, Longteng Guo, Ming-Ming Yu, Guiyu Zhao+2

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beihang University

Visual Navigation World Models for Robotics Latent Visual Reasoning Model-Based Planning

Open paper AskPDF arXiv

Conventional visual navigation policies often struggle with myopic decision-making and mode collapse in complex environments. While world models offer a promising alternative, existing paradigms typically isolate perception, generation, and control, failing to capture their shared spatio-temporal dynamics. In this paper, we propose NavWM, a unified navigation world model that seamlessly integrates latent world reasoning, multimodal action prediction, and controllable visual generation. At its core, NavWM leverages latent world tokens to distill geometric and semantic priors, endowing the agent with robust structural understanding. To overcome the limitations of deterministic policies, we introduce an anchor-based multimodal trajectory forecasting framework that generates a diverse action space. This inherent diversity explicitly empowers the generative world model to act as a robust closed-loop planner, utilizing visual foresight to evaluate and select the optimal path. Extensive experiments across diverse robotics datasets demonstrate that NavWM significantly advances the state-of-the-art, delivering remarkable improvements in both high-fidelity future state generation and zero-shot navigation success.

Abstract

Embodied Navigation

Explore similar work

NavWM: A Unified Navigation World Model for Foresight-Driven Planning

UniNav: A Unified World-Action Diffusion Model for Visual Navigation

WAM-Nav: Asymmetric Latent World-Action Modeling for Unified Visual Navigation