Visual Representation and History Modeling for Navigation World Models

Guangfu Guo; Xiaoqian Lu; Rui Liu; Yutong Chen; Kunpeng Liu; Long Cheng

cs.CVAug 26, 2026

Visual Representation and History Modeling for Navigation World Models

Authors: Guangfu Guo, Xiaoqian Lu, Rui Liu, Yutong Chen, Kunpeng Liu, Long Cheng

Organizations: Clemson University Clemson, SC, USA · University of Ottawa Ottawa, ON, Canada

Abstract

Navigation World Models (NWMs) predict action-conditioned visual futures for planning. Two practical challenges are central to their design: selecting a suitable visual representation and efficiently modeling observation history for repeated candidate queries. Standard Global-Softmax attention provides flexible interactions but repeatedly processes the same history, leading to increasing computation and memory costs for long contexts and multi-query planning. We study both problems within a unified conditional flow-transformer framework. We first compare five frozen visual representations under the same dynamics model and evaluation. To reduce redundant history computation, we design Cached-Linear, a hybrid architecture that combines local and shifted-window attention for target mixing with linear attention for reusable history access. We further develop Balanced Gated Delta Network (GDN), which augments this design with frame-wise recurrent memory for temporal history modeling. Experiments on RECON, SACSoN, and SCAND show that representation choice depends on the prediction objective: PAE-L performs best for reconstruction, RAE-B for direct prediction, and V-JEPA for long-horizon rollout. Under shared-history workloads, Cached-Linear substantially reduces computation and memory compared with Global-Softmax, while Balanced GDN improves selected direct-prediction endpoints with efficient context reuse. Overall, we systematically study visual representation and history modeling for NWMs and develop hybrid reusable-history architectures for efficient long-context and multi-query prediction.

Figures & tables

Fig. 1: Overview of the Navigation World-model Framework and Visual-Representation Comparison. Visual history is encoded by a frozen encoder into latent states, and a conditional flow transformer predicts the future state using the observation history, candidate action, prediction horizon, and flow time. A matched frozen decoder maps the predicted latent back to image space. The same dynamics model and evaluation pipeline are used across five frozen visual representation—RAE-B, RAE-L, PAE-L, Cosmos, and V-JEPA—which differ in representation type and latent size.

Fig. 2: Balanced GDN Dynamics Architecture. Left: action-conditioned conditional flow prediction with frozen visual history latents. Center: each CDiT block separates target mixing, reusable history mixing, and feed-forward computation with frame-wise GDN memory updates. Right: rollout updates the latent history directly. The key denotes window attention ( $W$ ), shifted-window attention (SWA), cached-linear summary $(M,u)$ , and GDN state $(S,r)$ .

Component	Function
Window attention ( $W$ )	Exact self-attention within $8\times 8$ windows for fine-grained local mixing.
Shifted-window attention (SWA)	Shifts the window partition by half a window to exchange information across neighboring regions.
Cached linear attention ( $L$ )	Compresses the observation history into a reusable key–value summary for repeated queries.
Gated Delta Network (GDN)	Maintains a recurrent state across frames for temporal history modeling.
Full attention	Provides global mixing in the final anchor block to preserve a direct long-range communication path.

TABLE I: History-mixing components and their roles.

Study	Representation	History mixer	Train	Evaluation
Study I: Representation	5 states	Global-Softmax	10 epochs	16 s: 598/199/42; 32 s: 150/112/37
Main Experiments	RAE, PAE, V-JEPA	7 models	50 epochs	Direct and rollout @ 16/32
Ablation I: Components	RAE, PAE, V-JEPA	$3\times 4$ configurations	10 epochs	Direct and rollout @ 16/32
Ablation II: Systems	RAE-B	Softmax / Linear / GDN	Systems	8–128 frames; 1 or 120 queries

TABLE II: Experimental design and evaluation map.

Interface	Frozen source	Channels	Values/frame	Re. DINO $\downarrow$
RAE-B	DINOv2-B/C768	768	196,608	0.147
RAE-L	DINOv2-L/C1024	1,024	262,144	0.159
PAE-L	DINOv2-L/C32	32	8,192	0.069
Cosmos	Cosmos-CI16	16	4,096	0.125
V-JEPA	V-JEPA 2.1 ViT-L	96	24,576	0.335

TABLE III: Study I: visual representation and reconstruction DINO.

Fig. 3: Study I: Visual-Representation Horizons. Horizon-wise decoded quality for all five Representation under Global-Softmax. Rows show direct prediction and 4-fps autoregressive rollout; columns show DINO, LPIPS, and SSIM. Lower is better for DINO/LPIPS and higher is better for SSIM.

	LPIPS $\downarrow$		DreamSim $\downarrow$		DINO $\downarrow$		FID-199 $\downarrow$
Model	4s	16s	4s	16s	4s	16s	4s	16s
RAE-B	0.492	0.529	0.335	0.377	0.497	0.529	156.00	170.36
PAE-L	0.583	0.605	0.450	0.472	0.605	0.625	171.59	181.79
RAE-L	0.505	0.529	0.420	0.456	0.626	0.655	209.55	223.86
Cosmos	0.550	0.570	0.528	0.546	0.722	0.738	247.27	250.87
V-JEPA	0.466	0.499	0.353	0.386	0.571	0.593	133.82	145.96

TABLE IV: Study I: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.

	SACSoN		RECON		SCAND
Model	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$
RAE-B/C768	3.668	0.858	1.644	0.448	1.210	0.326
PAE-L/C32	4.016	0.907	1.765	0.476	1.170	0.316
RAE-L/C1024	3.916	0.914	1.689	0.443	1.214	0.330
Cosmos-CI16	4.369	0.993	1.803	0.468	1.154	0.316
V-JEPA/S-VAE96	3.885	0.909	1.754	0.448	0.968	0.300

TABLE V: Study I: open-loop trajectory prediction. Two-second, eight-step trajectory prediction on held-out indices, using CEM-120, DINO-token scoring, and Euler-50 generation.

(a) Conditioning corruption: DINO degradation
Model	Macro	RECON	SACSoN	SCAND	Clean DINO
RAE-B	+0.0369	+0.0255	+0.0442	+0.0411	0.5742
PAE-L	+0.0075	+0.0003	+0.0068	+0.0153	0.6520
RAE-L	+0.0222	+0.0186	+0.0231	+0.0250	0.6607
Cosmos	+0.0071	-0.0007	+0.0064	+0.0157	0.7355
V-JEPA	+0.0227	+0.0157	+0.0301	+0.0223	0.6181

TABLE VI: Study I: conditioning sensitivity and frozen motion probes.

Fig. 4: Main Experiment: RAE Horizon Quality. Macro-averaged Direct (top) and autoregressive Rollout (bottom) curves across RECON, SACSoN, and SCAND. Columns report DINO, LPIPS, and SSIM.

		Direct@16					Rollout@16					Direct@32					Rollout@32
State	Mixer	DINO	LPIPS	DreamSim	PSNR	SSIM	DINO	LPIPS	DreamSim	PSNR	SSIM	DINO	LPIPS	DreamSim	PSNR	SSIM	DINO	LPIPS	DreamSim	PSNR	SSIM
RAE	Softmax	0.482	0.451	0.235	12.98	0.359	0.530	0.505	0.314	11.77	0.330	0.554	0.534	0.311	11.47	0.319	0.554	0.531	0.337	11.31	0.316
RAE	Linear	0.483	0.456	0.241	12.91	0.356	0.539	0.515	0.321	11.72	0.323	0.546	0.529	0.306	11.49	0.319	0.556	0.532	0.347	11.11	0.313
RAE	GDN	0.480	0.455	0.235	12.88	0.355	0.542	0.522	0.327	11.53	0.319	0.540	0.521	0.303	11.71	0.322	0.563	0.540	0.353	11.05	0.305
PAE	GDN	0.478	0.436	0.231	12.86	0.375	0.535	0.508	0.319	11.50	0.332	0.567	0.538	0.334	11.20	0.333	0.565	0.540	0.350	10.94	0.319
PAE	GDN+iREPA	0.479	0.434	0.230	13.00	0.377	0.532	0.506	0.316	11.60	0.340	0.568	0.537	0.337	11.19	0.322	0.553	0.527	0.341	11.13	0.328

TABLE VII: Main experiment: endpoint results. Macro-averages across RECON, SACSoN, and SCAND.

Model	LPIPS $\downarrow$		DreamSim $\downarrow$		DINO $\downarrow$		FID $\downarrow$
	4s	16s	4s	16s	4s	16s	4s	16s
RAE-Softmax	0.339	0.382	0.165	0.198	0.359	0.401	63.43	69.61
RAE-Linear	0.344	0.381	0.167	0.194	0.368	0.396	62.98	68.55
RAE-GDN	0.351	0.384	0.170	0.193	0.365	0.396	62.70	68.56
PAE-GDN	0.304	0.354	0.139	0.172	0.343	0.382	58.14	64.55
PAE-GDN+iREPA	0.301	0.352	0.137	0.173	0.341	0.381	58.17	64.37

TABLE VIII: Main experiment: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.

	SACSoN		RECON		SCAND
Model	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$
RAE-Softmax	3.259	0.777	1.468	0.401	1.021	0.285
RAE-Linear	3.458	0.807	1.664	0.426	1.083	0.302
RAE-GDN	3.325	0.786	1.553	0.414	1.276	0.336
PAE-GDN	3.614	0.836	1.668	0.432	1.218	0.336
PAE-GDN+iREPA	3.578	0.818	1.633	0.424	1.220	0.325

TABLE IX: Main experiment: open-loop trajectory prediction. Two-second, eight-step planning. CEM uses 120 candidates, DINO-token scoring, and Euler-50 generation.

Fig. 5: Main Experiment: Seven-Model Subset by Dataset. Each cell shows DINO excess over the best row for one dataset and endpoint; black dots mark the column winners. R/S/C denote RECON/SACSoN/SCAND.

Model	Direct@16	Rollout@16	Direct@32	Rollout@32	Avg-4
RAE-Softmax	0.574	0.813	0.608	0.820	0.704
RAE-Linear	0.550	0.667	0.571	0.692	0.620
RAE-GDN	0.543	0.656	0.567	0.677	0.611
RAE-GDN+iREPA	0.564	0.681	0.581	0.702	0.632
PAE-Softmax	0.646	0.815	0.652	0.816	0.732
PAE-Linear	0.628	0.736	0.635	0.747	0.687

TABLE X: Ablation I: architecture matrix. The

3\times 4

endpoint and Avg-4 DINO results compare complete bundles.

Model	LPIPS $\downarrow$		DreamSim $\downarrow$		DINO $\downarrow$		FID $\downarrow$
	4s	16s	4s	16s	4s	16s	4s	16s
RAE-Softmax	0.492	0.529	0.335	0.377	0.497	0.529	156.00	170.36
RAE-Linear	0.483	0.508	0.307	0.340	0.470	0.493	135.11	147.78
RAE-GDN	0.487	0.526	0.305	0.349	0.460	0.497	134.71	154.36
RAE-GDN+iREPA	0.492	0.529	0.326	0.368	0.481	0.514	149.05	171.17
PAE-Softmax	0.570	0.590	0.437	0.449	0.609	0.615	161.57	159.14

TABLE XI: Ablation I: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.

Model	SACSoN		RECON		SCAND
	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$
RAE-Softmax	3.668	0.858	1.644	0.448	1.210	0.326
RAE-Linear	3.894	0.895	1.677	0.444	1.220	0.341
RAE-GDN	3.816	0.896	1.591	0.436	1.290	0.349
RAE-GDN+iREPA	3.748	0.880	1.567	0.422	1.227	0.343
PAE-Softmax	4.184	0.959	1.816	0.491	1.170	0.316

TABLE XII: Ablation I: open-loop trajectory prediction. Two-second, eight-step open-loop trajectory prediction on held-out indices, using CEM-120, top-3 selection, one update, DINO-token scoring, and Euler-50 generation.

Fig. 6: Ablation I: Conditioning Controls. Six-model subset. (a) DINO degradation under zero, shuffle, wrong-sign, and wrong-horizon inputs. (b) Four-control mean. Positive values indicate aggregate sensitivit.

Queries / frames	Global-Softmax	Cached-Linear	Balanced GDN
Single / 8	0.58 / 2.10	0.60 / 2.13	0.78 / 2.19
120 / 8	11.84 / 7.50	7.15 / 5.29	7.41 / 5.40
120 / 16	17.45 / 11.38	7.15 / 5.30	7.50 / 5.44
120 / 32	28.81 / 19.13	7.15 / 5.31	7.68 / 5.51
120 / 64	51.67 / 34.63	7.15 / 5.35	8.02 / 5.67
120 / 128	OOM / $>79.18$	7.19 / 5.55	8.75 / 5.97

TABLE XIII: Ablation II: repeated-query scaling. Cells report H100 BF16 latency (s) / peak memory (GiB).

Architecture	Train $\downarrow$ ms	Memory $\downarrow$ GiB	Prefill $\downarrow$ ms	Query $\downarrow$ ms
Global-Softmax	453.98	31.02	–	151.95
Cached-Linear	284.98	23.70	12.73	75.98
Linear + Cross-L3/MLA1	359.41	28.72	10.98	109.20
Linear + Cross-L3/SWA1	293.94	24.44	15.07	77.71
Balanced GDN	292.66	24.76	13.52	77.21
Dense Self-GDN	308.01	26.89	15.10	79.68

TABLE XIV: Ablation II: architecture efficiency test. BF16 model-core efficiency evaluation on one RTX PRO 6000 Blackwell GPU.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 366 papers

RAE-NWM: Navigation World Model in Dense Visual Representation Space
Mar 10, 2026Mingkun Zhang, Wangtian Shen, Fan Zhang +3Visual Navigation World Model Learning
Department of Precision Instrument, Tsinghua University, Beijing, China · University of Rochester, Rochester, NY, USA · Beijing Information Science and Technology University, Beijing, China
NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation
Jun 11, 2026Daichi Azuma, Taiki Miyanishi, Koya Sakamoto +6World Models for Robotics Robot Navigation
The University of Tokyo · National Institute of Informatics · AIRoA +1
LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild
Oct 8, 2026Linkai Liu, Yuntian Zhang, Zhenshan Bing +5Robot Navigation Latent World Models
Nanjing University · Imperial College London · Sony AI +1

Mar 10, 2026J/K move · Enter open · S save

RAE-NWM: Navigation World Model in Dense Visual Representation Space

Mingkun Zhang, Wangtian Shen, Fan Zhang, Haijian Qin+2

Department of Precision Instrument, Tsinghua University, Beijing, China · University of Rochester, Rochester, NY, USA · Beijing Information Science and Technology University, Beijing, China

Visual Navigation World Model Learning Visual Representation Learning Diffusion Transformer Action-Conditioned World Models

Open paper AskPDF arXiv

Visual navigation requires agents to reach goals in complex environments through perception and planning. World models address this task by simulating action-conditioned state transitions to predict future observations. Current navigation world models typically learn state evolution under actions within the compressed latent space of a Variational Autoencoder, where spatial compression often discards fine-grained structural information and hinders precise control. To better understand the propagation characteristics of different representations, we conduct a linear dynamics probe and observe that dense DINOv2 features exhibit stronger linear predictability for action-conditioned transitions. Motivated by this observation, we propose the Representation Autoencoder-based Navigation World Model (RAE-NWM), which models navigation dynamics in a dense visual representation space. We employ a Conditional Diffusion Transformer with Decoupled Diffusion Transformer head (CDiT-DH) to model continuous transitions, and introduce a separate time-driven gating module for dynamics conditioning to regulate action injection strength during generation. Extensive evaluations show that modeling sequential rollouts in this space improves structural stability and action accuracy, benefiting downstream planning and navigation.

Abstract

World Models

Explore similar work

RAE-NWM: Navigation World Model in Dense Visual Representation Space

NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation

LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild