DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning

Yang Zhou; Xiaofeng Wang; Hao Shao; Letian Wang; Guosheng Zhao; Jiangnan Shao; Jiagang Zhu; Tingdong Yu

cs.CVApr 2, 2026

DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning

Authors: Yang Zhou, Xiaofeng Wang, Hao Shao, Letian Wang, Guosheng Zhao, Jiangnan Shao, Jiagang Zhu, Tingdong Yu, +3 more

Organizations: GigaAI · University of Toronto · CUHK MMLab

Abstract

Recently, world-action models (WAM) have emerged to bridge vision-language-action (VLA) models and world models, unifying their reasoning and instruction-following capabilities and spatio-temporal world modeling. However, existing WAM approaches often focus on modeling 2D appearance or latent representations, with limited geometric grounding-an essential element for embodied systems operating in the physical world. We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning within a single modular architecture. The model employs a large language model to process language instructions, multi-view images, and actions, followed by three lightweight generators that produce depth, future video, and actions. By learning a geometry-aware world representation and using it to guide both future prediction and planning within a unified framework, the proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and controllable latency. Experiments on the Navsim v1 and v2 benchmarks demonstrate that DriveDreamer-Policy achieves strong performance on both closed-loop planning and world generation tasks. In particular, our model reaches 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2, outperforming existing world-model-based approaches while producing higher-quality future video and depth predictions. Ablation studies further show that explicit depth learning provides complementary benefits to video imagination and improves planning robustness.

Figures & tables

Figure 1: Comparison of our DriveDreamer-Policy with existing models. Items with dashed lines are optional. Vision-based and VLA planners directly map observations (and optional inputs) to actions without explicitly predicting the future world. World models generate future observations but often rely on external action signals. Recent world–action models unify future world generation and planning, but typically operate on image/video representations. DriveDreamer-Policy extends this line by explicitly generating depth alongside video and actions, enabling geometry-grounded imagination and planning within a unified model.

Figure 2: Overview of our DriveDreamer-Policy pipeline. The large language model takes the language instruction, multi-view images and current action, along with a set of learnable queries as inputs to reason and generate world and action embeddings. The generated embeddings are then passed into our three generative expert models as cross-attention conditions to generate depth, future images, and future action.

Methods	Venue	Sensors	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Human	–	–	100.0	100.0	100.0	99.9	87.5	94.8
Vision-Based End-to-End Methods
TransFuser ( Chitta et al., 2022 )	TPAMI’23	3 $\times$ C ${+}$ L	97.7	92.8	92.8	100.0	79.2	84.0
UniAD ( Hu et al., 2023 )	CVPR’23	6 $\times$ C	97.8	91.9	92.9	100.0	78.8	83.4
PARA-Drive ( Weng et al., 2024 )	CVPR’24	6 $\times$ C	97.9	92.4	93.0	99.8	79.3	84.0
DiffusionDrive ( Liao et al., 2025a )	CVPR’25	3 $\times$ C ${+}$ L	98.2	96.2	94.7	100.0	82.2	88.1

Table 1: Comparison with state-of-the-art methods on the the navtest of Navsim v1 ( Dauner et al., 2024 ) benchmark. The best result outlines as bold. “*” indicates results with imitation learning. Methods fall into three categories: vision-based E2E models, VLA-based models, and world-model-based models.

Methods	Venue	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Vision-Based End-to-End Methods
TransFuser ( Chitta et al., 2022 )	TPAMI’23	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
DiffusionDrive ( Liao et al., 2025a )	CVPR’25	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
Drivesuprim ( Yao et al., 2025 )	AAAI’26	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
ARTEMIS ( Feng et al., 2026 )	RAL’26	98.3	95.1	98.6	99.8	81.5	97.4	96.5	98.3	89.1	83.1
Vision-Language-Action Methods

Table 2: Comparison with state-of-the-art methods on the navtest of Navsim v2 ( Cao et al., 2025 ) .

Strategy	World Representation		Planning Metrics
	Depth	Video	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Without World Learning	✗	✗	98.0	96.3	94.4	100.0	82.5	88.0
With World Learning	✔	✗	98.1	96.7	94.9	100.0	82.8	88.5
	✗	✔	98.1	97.0	95.0	100.0	83.1	88.9
	✔	✔	98.4	97.1	95.1	100.0	83.5	89.2

Table 4: Ablations on World Learning for Planning. All world learning strategies effectively improve planning performance compared to training from scratch.

Strategy	World Representation	Video Metrics
	Depth	LPIPS $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$
Without Depth Learning	✗	0.22	19.89	65.82
With Depth Learning	✔	0.20	21.05	53.59

Table 5: Ablations on Depth Learning for Video Generation. Using depth as a prior in joint learning improves video generation accuracy.

Number of Queries			Metrics
Depth	Video	Action	Depth				Video			Action
Depth	Video	Action	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	$\delta_{2}$ $\uparrow$	$\delta_{3}$ $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
32	32	4	9.7	90.2	97.9	99.4	0.20	20.67	57.97	98.2	97.0	95.0	100.0	83.2	88.9
64	64	8	8.1	92.8	98.6	99.5	0.20	21.05	53.59	98.4	97.1	95.1	100.0	83.5	89.2

Table 6: Ablations on Number of Queries. More query tokens provide higher-capacity slots for storing relevant context, thereby enhancing both generation and planning.

Panel of Figure 3 — Figure 3: Visualization Results of our method. We show the generated depth, video, and actions, respectively. Depth is truncated to below 80 meters for better visualization. Our generation results remain spatially stable, and the planning performs well compared with human trajectories ( e.g. , aligns with human trajectories (top) and slows down more effectively than human trajectories (bottom)).

Figure 4: Visualization of world learning for planning. Columns compare Action-Only, Depth-Action, Video-Action, and Depth-Video-Action variants. Green denotes the human (expert) trajectory and red denotes the predicted trajectory. The three rows correspond to (top) avoiding potential collision by a slower trajectory, (middle) correcting an initially wrong maneuver, and (bottom) aligning more closely with the human trajectory. Depth and video provide complementary world cues that improve safety margins and trajectory consistency.

Explore similar work

Learning Vision-Language-Action World Models for Autonomous Driving
Apr 10, 2026Guoqing Wang, Pin Tang, Xiangxuan Ren +3World Models for Autonomous Driving Vision-Language-Action Models
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei
DriveVA: Video Action Models are Zero-Shot Drivers
Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video Generation World Models
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1
DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
Oct 13, 2026Shuzheng Wang, Haoxuan Li, Shengzhen Li +5Autonomous Driving Planning VLMs for Autonomous Driving
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.

Apr 10, 2026J/K move · Enter open · S save

Learning Vision-Language-Action World Models for Autonomous Driving

Guoqing Wang, Pin Tang, Xiangxuan Ren, Guodongfang Zhao+2

MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei

World Models for Autonomous Driving Vision-Language-Action Models Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io

Abstract

Explore similar work

Learning Vision-Language-Action World Models for Autonomous Driving

DriveVA: Video Action Models are Zero-Shot Drivers

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving