Navigation World Models (NWMs) predict action-conditioned visual futures for planning. Two practical challenges are central to their design: selecting a suitable visual representation and efficiently modeling observation history for repeated candidate queries. Standard Global-Softmax attention provides flexible interactions but repeatedly processes the same history, leading to increasing computation and memory costs for long contexts and multi-query planning. We study both problems within a unified conditional flow-transformer framework. We first compare five frozen visual representations under the same dynamics model and evaluation. To reduce redundant history computation, we design Cached-Linear, a hybrid architecture that combines local and shifted-window attention for target mixing with linear attention for reusable history access. We further develop Balanced Gated Delta Network (GDN), which augments this design with frame-wise recurrent memory for temporal history modeling. Experiments on RECON, SACSoN, and SCAND show that representation choice depends on the prediction objective: PAE-L performs best for reconstruction, RAE-B for direct prediction, and V-JEPA for long-horizon rollout. Under shared-history workloads, Cached-Linear substantially reduces computation and memory compared with Global-Softmax, while Balanced GDN improves selected direct-prediction endpoints with efficient context reuse. Overall, we systematically study visual representation and history modeling for NWMs and develop hybrid reusable-history architectures for efficient long-context and multi-query prediction.
Figures & tables
Fig. 1: Overview of the Navigation World-model Framework and Visual-Representation Comparison. Visual history is encoded by a frozen encoder into latent states, and a conditional flow transformer predicts the future state using the observation history, candidate action, prediction horizon, and flow time. A matched frozen decoder maps the predicted latent back to image space. The same dynamics model and evaluation pipeline are used across five frozen visual representation—RAE-B, RAE-L, PAE-L, Cosmos, and V-JEPA—which differ in representation type and latent size.
Fig. 2: Balanced GDN Dynamics Architecture. Left: action-conditioned conditional flow prediction with frozen visual history latents. Center: each CDiT block separates target mixing, reusable history mixing, and feed-forward computation with frame-wise GDN memory updates. Right: rollout updates the latent history directly. The key denotes window attention ( W ), shifted-window attention (SWA), cached-linear summary (M,u) , and GDN state (S,r) .
Component
Function
Window attention ( W )
Exact self-attention within 8×8 windows for fine-grained local mixing.
Shifted-window attention (SWA)
Shifts the window partition by half a window to exchange information across neighboring regions.
Cached linear attention ( L )
Compresses the observation history into a reusable key–value summary for repeated queries.
Gated Delta Network (GDN)
Maintains a recurrent state across frames for temporal history modeling.
Full attention
Provides global mixing in the final anchor block to preserve a direct long-range communication path.
TABLE I: History-mixing components and their roles.
Study
Representation
History mixer
Train
Evaluation
Study I: Representation
5 states
Global-Softmax
10 epochs
16 s: 598/199/42; 32 s: 150/112/37
Main Experiments
RAE, PAE, V-JEPA
7 models
50 epochs
Direct and rollout @ 16/32
Ablation I: Components
RAE, PAE, V-JEPA
3×4 configurations
10 epochs
Direct and rollout @ 16/32
Ablation II: Systems
RAE-B
Softmax / Linear / GDN
Systems
8–128 frames; 1 or 120 queries
TABLE II: Experimental design and evaluation map.
Interface
Frozen source
Channels
Values/frame
Re. DINO ↓
RAE-B
DINOv2-B/C768
768
196,608
0.147
RAE-L
DINOv2-L/C1024
1,024
262,144
0.159
PAE-L
DINOv2-L/C32
32
8,192
0.069
Cosmos
Cosmos-CI16
16
4,096
0.125
V-JEPA
V-JEPA 2.1 ViT-L
96
24,576
0.335
TABLE III: Study I: visual representation and reconstruction DINO.
Fig. 3: Study I: Visual-Representation Horizons. Horizon-wise decoded quality for all five Representation under Global-Softmax. Rows show direct prediction and 4-fps autoregressive rollout; columns show DINO, LPIPS, and SSIM. Lower is better for DINO/LPIPS and higher is better for SSIM.
LPIPS ↓
DreamSim ↓
DINO ↓
FID-199 ↓
Model
4s
16s
4s
16s
4s
16s
4s
16s
RAE-B
0.492
0.529
0.335
0.377
0.497
0.529
156.00
170.36
PAE-L
0.583
0.605
0.450
0.472
0.605
0.625
171.59
181.79
RAE-L
0.505
0.529
0.420
0.456
0.626
0.655
209.55
223.86
Cosmos
0.550
0.570
0.528
0.546
0.722
0.738
247.27
250.87
V-JEPA
0.466
0.499
0.353
0.386
0.571
0.593
133.82
145.96
TABLE IV: Study I: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.
SACSoN
RECON
SCAND
Model
ATE ↓
RPE ↓
ATE ↓
RPE ↓
ATE ↓
RPE ↓
RAE-B/C768
3.668
0.858
1.644
0.448
1.210
0.326
PAE-L/C32
4.016
0.907
1.765
0.476
1.170
0.316
RAE-L/C1024
3.916
0.914
1.689
0.443
1.214
0.330
Cosmos-CI16
4.369
0.993
1.803
0.468
1.154
0.316
V-JEPA/S-VAE96
3.885
0.909
1.754
0.448
0.968
0.300
TABLE V: Study I: open-loop trajectory prediction. Two-second, eight-step trajectory prediction on held-out indices, using CEM-120, DINO-token scoring, and Euler-50 generation.
(a) Conditioning corruption: DINO degradation
Model
Macro
RECON
SACSoN
SCAND
Clean DINO
RAE-B
+0.0369
+0.0255
+0.0442
+0.0411
0.5742
PAE-L
+0.0075
+0.0003
+0.0068
+0.0153
0.6520
RAE-L
+0.0222
+0.0186
+0.0231
+0.0250
0.6607
Cosmos
+0.0071
-0.0007
+0.0064
+0.0157
0.7355
V-JEPA
+0.0227
+0.0157
+0.0301
+0.0223
0.6181
TABLE VI: Study I: conditioning sensitivity and frozen motion probes.
Fig. 4: Main Experiment: RAE Horizon Quality. Macro-averaged Direct (top) and autoregressive Rollout (bottom) curves across RECON, SACSoN, and SCAND. Columns report DINO, LPIPS, and SSIM.
Direct@16
Rollout@16
Direct@32
Rollout@32
State
Mixer
DINO
LPIPS
DreamSim
PSNR
SSIM
DINO
LPIPS
DreamSim
PSNR
SSIM
DINO
LPIPS
DreamSim
PSNR
SSIM
DINO
LPIPS
DreamSim
PSNR
SSIM
RAE
Softmax
0.482
0.451
0.235
12.98
0.359
0.530
0.505
0.314
11.77
0.330
0.554
0.534
0.311
11.47
0.319
0.554
0.531
0.337
11.31
0.316
RAE
Linear
0.483
0.456
0.241
12.91
0.356
0.539
0.515
0.321
11.72
0.323
0.546
0.529
0.306
11.49
0.319
0.556
0.532
0.347
11.11
0.313
RAE
GDN
0.480
0.455
0.235
12.88
0.355
0.542
0.522
0.327
11.53
0.319
0.540
0.521
0.303
11.71
0.322
0.563
0.540
0.353
11.05
0.305
PAE
GDN
0.478
0.436
0.231
12.86
0.375
0.535
0.508
0.319
11.50
0.332
0.567
0.538
0.334
11.20
0.333
0.565
0.540
0.350
10.94
0.319
PAE
GDN+iREPA
0.479
0.434
0.230
13.00
0.377
0.532
0.506
0.316
11.60
0.340
0.568
0.537
0.337
11.19
0.322
0.553
0.527
0.341
11.13
0.328
TABLE VII: Main experiment: endpoint results. Macro-averages across RECON, SACSoN, and SCAND.
Model
LPIPS ↓
DreamSim ↓
DINO ↓
FID ↓
4s
16s
4s
16s
4s
16s
4s
16s
RAE-Softmax
0.339
0.382
0.165
0.198
0.359
0.401
63.43
69.61
RAE-Linear
0.344
0.381
0.167
0.194
0.368
0.396
62.98
68.55
RAE-GDN
0.351
0.384
0.170
0.193
0.365
0.396
62.70
68.56
PAE-GDN
0.304
0.354
0.139
0.172
0.343
0.382
58.14
64.55
PAE-GDN+iREPA
0.301
0.352
0.137
0.173
0.341
0.381
58.17
64.37
TABLE VIII: Main experiment: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.
SACSoN
RECON
SCAND
Model
ATE ↓
RPE ↓
ATE ↓
RPE ↓
ATE ↓
RPE ↓
RAE-Softmax
3.259
0.777
1.468
0.401
1.021
0.285
RAE-Linear
3.458
0.807
1.664
0.426
1.083
0.302
RAE-GDN
3.325
0.786
1.553
0.414
1.276
0.336
PAE-GDN
3.614
0.836
1.668
0.432
1.218
0.336
PAE-GDN+iREPA
3.578
0.818
1.633
0.424
1.220
0.325
TABLE IX: Main experiment: open-loop trajectory prediction. Two-second, eight-step planning. CEM uses 120 candidates, DINO-token scoring, and Euler-50 generation.
Fig. 5: Main Experiment: Seven-Model Subset by Dataset. Each cell shows DINO excess over the best row for one dataset and endpoint; black dots mark the column winners. R/S/C denote RECON/SACSoN/SCAND.
Model
Direct@16
Rollout@16
Direct@32
Rollout@32
Avg-4
RAE-Softmax
0.574
0.813
0.608
0.820
0.704
RAE-Linear
0.550
0.667
0.571
0.692
0.620
RAE-GDN
0.543
0.656
0.567
0.677
0.611
RAE-GDN+iREPA
0.564
0.681
0.581
0.702
0.632
PAE-Softmax
0.646
0.815
0.652
0.816
0.732
PAE-Linear
0.628
0.736
0.635
0.747
0.687
TABLE X: Ablation I: architecture matrix. The 3×4 endpoint and Avg-4 DINO results compare complete bundles.
Model
LPIPS ↓
DreamSim ↓
DINO ↓
FID ↓
4s
16s
4s
16s
4s
16s
4s
16s
RAE-Softmax
0.492
0.529
0.335
0.377
0.497
0.529
156.00
170.36
RAE-Linear
0.483
0.508
0.307
0.340
0.470
0.493
135.11
147.78
RAE-GDN
0.487
0.526
0.305
0.349
0.460
0.497
134.71
154.36
RAE-GDN+iREPA
0.492
0.529
0.326
0.368
0.481
0.514
149.05
171.17
PAE-Softmax
0.570
0.590
0.437
0.449
0.609
0.615
161.57
159.14
TABLE XI: Ablation I: direct prediction. One-step SACSoN Direct@4 and Direct@16 on 199 held-out trajectories.
Model
SACSoN
RECON
SCAND
ATE ↓
RPE ↓
ATE ↓
RPE ↓
ATE ↓
RPE ↓
RAE-Softmax
3.668
0.858
1.644
0.448
1.210
0.326
RAE-Linear
3.894
0.895
1.677
0.444
1.220
0.341
RAE-GDN
3.816
0.896
1.591
0.436
1.290
0.349
RAE-GDN+iREPA
3.748
0.880
1.567
0.422
1.227
0.343
PAE-Softmax
4.184
0.959
1.816
0.491
1.170
0.316
TABLE XII: Ablation I: open-loop trajectory prediction. Two-second, eight-step open-loop trajectory prediction on held-out indices, using CEM-120, top-3 selection, one update, DINO-token scoring, and Euler-50 generation.
Visual navigation requires agents to reach goals in complex environments through perception and planning. World models address this task by simulating action-conditioned state transitions to predict future observations. Current navigation world models typically learn state evolution under actions within the compressed latent space of a Variational Autoencoder, where spatial compression often discards fine-grained structural information and hinders precise control. To better understand the propagation characteristics of different representations, we conduct a linear dynamics probe and observe that dense DINOv2 features exhibit stronger linear predictability for action-conditioned transitions. Motivated by this observation, we propose the Representation Autoencoder-based Navigation World Model (RAE-NWM), which models navigation dynamics in a dense visual representation space. We employ a Conditional Diffusion Transformer with Decoupled Diffusion Transformer head (CDiT-DH) to model continuous transitions, and introduce a separate time-driven gating module for dynamics conditioning to regulate action injection strength during generation. Extensive evaluations show that modeling sequential rollouts in this space improves structural stability and action accuracy, benefiting downstream planning and navigation.
Mingkun Zhang, Wangtian Shen, Fan Zhang +3
Department of Precision Instrument, Tsinghua University, Beijing, China · University of Rochester, Rochester, NY, USA · Beijing Information Science and Technology University, Beijing, China
Goal-conditioned visual navigation requires a robot to act under partial observability by anticipating how its motion will change the future egocentric view and whether that change brings it closer to the goal. Navigation world models provide such visual foresight, but they remain prediction modules that require an external planner to convert predicted futures into closed-loop control. We propose Navigation World Action Model (NavWAM), a diffusion-transformer policy that turns navigation world-model prediction into executable action by representing future observations, goal-progress values, and action chunks in a shared latent sequence. By learning future prediction jointly with the action and value targets that determine closed-loop behavior, NavWAM makes visual foresight directly usable for robot control. We build NavWAM through simulation pretraining and real-robot adaptation, and evaluate it on image-goal navigation against planning-based world models and a representative direct navigation policy. Across offline benchmarks and closed-loop real-robot deployment, NavWAM improves over planning-based world-model baselines in our evaluations while using the default policy mode without CEM-style action search. Project page: https://dachii-azm.github.io/navwam/
Daichi Azuma, Taiki Miyanishi, Koya Sakamoto +6
The University of Tokyo · National Institute of Informatics · AIRoA +1
Direct visual navigation policies generate trajectories efficiently but do not explicitly evaluate their future consequences. Generative navigation world models provide this foresight through visual rollouts, which are costly when evaluating multiple candidates. We present LiteNWM, a latent navigation world model that shares visual encoding across candidates and jointly predicts their action-conditioned future representations at multiple horizons, while a learned scorer uses these predictions to select trajectories. In offline evaluations on RECON, SCAND, and SACSoN, LiteNWM reduces macro-averaged trajectory error by 17.56% relative to NoMaD+NWM-XL and achieves a 128.00-fold end-to-end speedup on an RTX 5090. The same evaluator transfers from NoMaD to MBRA without proposer-specific retraining, reducing MBRA's macro-averaged trajectory error by 16.2%. In real-robot experiments in unseen indoor and outdoor environments, LiteNWM improves navigation success from 43.3% to 83.3% relative to NoMaD. These results demonstrate that LiteNWM can be deployed for future-aware planning and closed-loop navigation on a physical robot.
Linkai Liu, Yuntian Zhang, Zhenshan Bing +5
Nanjing University · Imperial College London · Sony AI +1