End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.
Figures & tables
Figure 1: World models for end-to-end autonomous driving. (a) Planning with future scenes generated by a driving world model. (b) Planning with semantic representation extracted from a latent world model. (c) WorldDrive bridges planning and driving world model via unifying vision and motion representation.
Figure 2: Overall architecture of WorldDrive . WorldDrive is a holistic framework unifying vision and motion representation to bridge scene generation and planning. The training process includes Phase 1: WorldDrive for scene generation and Phase 2: WorldDrive for motion planning. The vision and motion representations are optimized through the scene generation task. In the planning stage, the planner utilizes the frozen vision and trajectory encoders and outputs top- K multi-modal trajectories. A future-aware rewarder is further designed to select the optimal trajectory from the candidates.
Figure 3: Detailed illustration of Future-aware Rewarder . During training, the frozen world model generates future latents. A distillation mechanism aligns the future scene queries with the generated future latents. During the inference phase, the distilled scene features are directly queried by the motion representation to compute future-aware rewards.
Method
Sensors
WM-Based
NC
DAC
TTC
Comf.
EP
PDMS
TransFuser [ 8 ]
MV & L
×
97.7
92.8
92.8
100.0
79.2
84.0
Hydra-MDP [ 32 ]
MV & L
×
98.3
96.0
94.6
100.0
78.7
86.5
DiffusionDrive [ 36 ]
MV & L
×
98.2
96.2
94.7
100.0
82.2
88.1
WoTE [ 31 ]
MV & L
✓
98.5
96.8
94.9
99.9
81.9
88.3
UniAD [ 21 ]
MV
×
97.8
91.9
92.9
100.0
78.8
83.4
PARA-Drive [ 49 ]
MV
×
97.9
92.4
93.0
99.8
79.3
84.0
Table 1: NAVSIM navtest split comparison . PDMS and sub-scores reflecting closed-loop performance. MV: multi-view cameras; SV: single-view camera; L: LiDAR. † : Training with full navtrain split. ‡ : Best-of-6 performance with oracle.
Table 3: nuScenes validation split open-loop planning comparison . We follow the SparseDrive evaluation metric. V: multi-view camera; SV: single-view camera
VAE
TA-DWM
TA-DWM
NC
DAC
TTC
EP
PDMS
Pretrain
Vision
Motion
×
×
×
69.0
57.6
57.2
28.7
31.4
✓
×
×
97.6
93.6
93.2
79.5
84.9
✓
✓
×
97.9
94.1
93.3
80.7
85.8
✓
✓
✓
98.4
95.1
94.7
80.4
86.9
Table 4: Planner with different pretrained representation . “VAE Pretrain” means initializing with the 3D VAE weights from CogVideoX. “TA-DWM Vision” and “TA-DWM Motion” mean initializing the vision and motion encoders from TA-DWM.
Traj feat.
Future feat.
NC
DAC
TTC
EP
PDMS
×
×
98.4
95.1
94.7
80.4
86.9
✓
×
98.3
95.3
94.3
81.2
87.0
✓
✓
98.4
96.2
95.1
81.9
88.1
Table 5: Future-aware rewarder design . “Traj feat.” means using candidate trajectory features from the planner. “Future feat.” means using future scene query and future latent distillation.
Method
NC
DAC
TTC
EP
PDMS
Latency
PWM †
98.0
95.1
94.1
82.4
87.3
570ms
PWM
98.6
95.9
95.4
81.8
88.1
850ms
WorldDrive
98.4
96.8
95.2
83.3
89.0
53ms
Table 6: Inference latency comparison . Latency is measured on a single NVIDIA A800 GPU. “ † ” means without future frame forecast.
Figure 4: Quantitative Analysis of Motion Sensitivity. The similarity between scene representations is inversely correlated with the geometric distance. The sensitivity to both large (a) and small (b) deviations is amplified with further training.
Figure 5: Qualitative planning result of WorldDrive on NAVSIM navtest split . (a) Planning result and the corresponding generated future scene with different trajectories. (b) Top-10 Multi-modal planning trajectories.
Figure 6: Qualitative simulation result on nuScenes . Scenes are generated with three trajectory conditions: the expert trajectory (green), and non-expert alternatives. The results show a tight coupling between the input motion and the generated scenes.
Figure 7: The details of (a) Trajectory-aware Driving World Model and (b) Trajectory-aware Diffusion Transformer block.
Top-K
NC
DAC
TTC
EP
PDMS
1
98.4
95.1
94.7
80.4
86.9
3
98.4
96.0
95.2
81.5
87.9
5
98.4
96.2
95.1
81.9
88.1
10
98.0
95.9
94.4
81.9
87.6
Table 8: Ablation on the number of trajectory candidates for Future-aware Rewarder. Top-K indicates the trajectories with the K highest predicted scores from the multi-modal planner.
Module
Encoders
Planner
FAR
Total
latency
17.9ms
18.9ms
16.2ms
53ms
Table 9: Inference latency comparison . Latency is measured on a single NVIDIA A800 GPU for the full forward pipeline of WorldDrive. “Encoders” include the 3D Causal VAE, visual adapter, and trajectory encoder.
Figure 8: Qualitative results of planning and future scene generation with WorldDrive on the NAVSIM navtest split.
Figure 9: Qualitative results of action-controllable future scene generation with WorldDrive on the nuScenes validation set.
Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.
Mengmeng Liu, Diankun Zhang, Jiuming Liu +7
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1
World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, yet existing approaches typically treat perception, prediction, and planning as separate modules. We propose UniDrive-WM, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture. UniDrive-WM's trajectory planner predicts a future trajectory, which conditions a VLM-based image generator to produce plausible future frames. These predictions provide additional supervisory signals that enhance scene understanding and iteratively refine trajectory generation. We further compare discrete and continuous output representations for future image prediction, analyzing their influence on downstream driving performance. Experiments on the challenging Bench2Drive benchmark show that UniDrive-WM produces high-fidelity future images and improves planning performance by 7.3% in L2 trajectory error and 10.4% in collision rate over the previous best method. These results demonstrate the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving. The project page is available at https://unidrive-wm.github.io/UniDrive-WM.
Zhexiao Xiong, Xin Ye, Burhan Yaman +5
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Shuzheng Wang, Haoxuan Li, Shengzhen Li +5
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.