Organizations: Institute of Automation, Chinese Academy of Sciences · Chongqing Chang’an Technology Co., Ltd · Zhongguancun Academy · School of Artificial Intelligence, University of Chinese Academy of Sciences · College of AI, Tsinghua University
We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills geometric knowledge from a foundation model and compresses multi-view images into compact scene tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal Transformer to autoregressively predict future world status conditioned on historical visual and motion representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model.
Figures & tables
Figure 1: Performance vs. training data scale on NAVSIM v2. Bubble size indicates model parameters. Our Latent-WAM achieves the highest EPDMS with significantly fewer training data and smaller model size, demonstrating superior data efficiency over existing world-model-based methods. World4Drive is marked separately as it employs an additional ViT-L depth estimator.
Figure 2: Overview of the Latent-WAM architecture. See Sec. 3.1 for details.
Figure 3: Teacher Forcing Attention Mask.
Method
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Perception-based Methods
TransFuser [ 30 ]
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
DriveSuprim [ 42 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
ReCogDrive [ 21 ]
98.3
95.2
99.5
99.8
87.1
97.5
96.6
98.3
86.5
83.6
DiffusionDrive [ 23 ]
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
WorldRFT [ 40 ]
97.8
96.5
99.5
99.8
88.5
97.0
97.4
98.1
69.1
86.7
Table 1: Comparison with state-of-the-art methods on the NAVSIM v2 with extended metrics. We indicate the best and second best with bold and underlined respectively. †: The reported results are dependent on perception-based annotation.
Method
RC ↑
HD-Score ↑
E
M
H
X
Avg .
E
M
H
X
Avg .
UniAD
[ 13 ]
58.6
41.2
40.4
26.0
40.6
48.7
29.5
27.3
14.3
28.9
VAD
[ 15 ]
38.7
27.0
25.5
23.0
27.9
24.3
0 9.9
10.4
0 8.2
12.3
LTF
[ 4 ]
68.4
40.7
36.9
25.5
41.4
52.8
24.6
19.8
0 8.1
24.8
GTRS-Dense
[ 22 ]
64.2
50.0
20.7
22.3
38.0
55.5
39.0
11.7
14.3
28.6
Ours
84.2
42.5
30.6
35.5
45.9
72.5
24.0
12.2
18.1
28.9
Table 2: Photorealistic closed-loop evaluation on HUGSIM [ 48 ] . Zero-shot generalization using our model from the NAVSIM-v2 evaluation. Scores are per difficulty and overall average road completion (RC) and HD-Score, higher always better.
Table 3: Ablation study of each proposed component. + and - denote improvement/degradation relative to the baseline.
Table 7
Prediction Temporal Stride
EPDMS ↑
0→8
88.4
−3→0→4→8
89.3
−3→−2→−1→0→2→4→6→8
89.1
Table 6: Ablation study on world model prediction temporal stride
Figure 4: Visualization of planning trajectories, where the green line is the human trajectory, the yellow line is the predicted trajectory of corresponding method.
Figure 5: Visualization of attention maps between scene tokens and image patches. From top to bottom, the three groups correspond to going straight, turning right, and turning left respectively.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Module
Parameters
Memory Usage
Latency
World Encoder
86.6M
—
100ms
Trajectory Decoder
8.4M
—
6ms
All modules
104M
1.1GB
107ms
Appendix
Table 7: Per-module inference latency of Latent-WMA
World models in end-to-end autonomous driving predict future scene evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.
Jinchang Xu, Hongda Yu, Fengwei Dong +9
Southeast University · NIO · Nanyang Technological University +1
Driving world models learn rich predictive representations of the surrounding environment from visual observations, yet accurate visual prediction does not necessarily translate into effective trajectory planning. We argue that a key bottleneck lies in the mismatch between visual world states and raw geometric trajectories, which may limit the planner's ability to exploit action-relevant semantics encoded by the world model. To address this issue, we propose World-Model Alignment for Latent Trajectories (WALT), which learns a compact generative trajectory latent space by transferring information from a frozen pretrained driving world model without modifying the world model itself. Rather than directly generating raw waypoints, WALT maps them into compact representations through a dual-branch trajectory autoencoder and transfers semantic knowledge from the frozen visual world model into this trajectory space, encouraging the learned action representation to capture scene-level cues relevant to future motion and planning. Beyond our proposed formulation, we systematically study latent learning based on Joint-Embedding Predictive Architectures (JEPA) and feature alignment following Representation Alignment (REPA) to investigate how trajectory-only representation learning affects downstream planning. We evaluate WALT on the NAVSIM benchmarks. Relative to the raw-waypoint baseline, WALT improves PDMS from 89.4 to 89.8 on NAVSIMv1 and EPDMS from 87.3 to 87.9 on NAVSIMv2 while reducing trajectory planner FLOPs by 30.5%. These results suggest that preserving world representations while extracting action-relevant information provides an effective interface for world-model-based trajectory planning.
Mingkai Jia, Jiaxin Guo, Zhijian Shu +5
The Hong Kong University of Science and Technology. · Horizon Robotics. · The Chinese University of Hong Kong. +2
In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring video dynamics priors to action prediction, but many still couple planning with future-video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future-frame generation at inference. This design supports multiple pretrained video backbones and independent action-expert scaling within a shared attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves 91.9 PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
Zongchuang Zhao, Xin Zhou, Tianyang Xu +6
Huazhong University of Science & Technology · Dongfeng Research & Development Institute