Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.
Figures & tables
Figure 1: Comparison between existing works and ours. Prior works either leverage LLMs/VLMs for multimodal understanding and reasoning, or adopt world models for video-based scene imagination, but treat these capabilities in isolation. In contrast, our proposed LMGenDrive unifies both within a closed-loop end-to-end framework: the LLM interprets and reasons over multimodal inputs, while the world model simulates future scene evolution, together enabling instruction-guided planning, spatio-temporal understanding, and robust long-horizon driving.
Figure 2: Overview of our unified understanding and generation architecture. We start by encoding the language instruction, multi-view RGB images, and the current action into the LLM. Two sets of learnable queries, world query and action query, are then fed into the LLM, and ultimately used to generate the future driving video and corresponding actions. The framework supports two operation modes: (1) offline data generation mode, an autoregressive generation process is adopted, where the last frame of the future video and the predicted action are used as inputs for the next timestep; (2) online planning mode, real-world data are provided as inputs for the following timestep.
Figure 3: Architecture of our world generator. We begin by fusing the world embedding obtained from the LLM with multi-view RGB images. The fused multi-view world embedding is then injected into the diffusion model with the cross-attention mechanism to generate multi-view future videos.
Methods
LangAuto
LangAuto-Short
LangAuto-Tiny
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
LMDrive [ 41 ]
10.7 ± 3.8
16.2 ± 4.9
0.63 ± 0.04
14.2 ± 4.4
20.1 ± 4.4
0.72 ± 0.04
20.1 ± 4.1
24.7 ± 5.1
0.75 ± 0.03
AD-H † [ 63 ]
44.0
53.2
0.83
56.1
68.0
0.78
77.5
85.1
0.91
BEVDriver [ 56 ]
48.9
59.7
0.82
66.7
77.8
0.87
70.2
81.3
0.87
Ours
62.2 ± 3.3
74.5 ± 4.1
0.85 ± 0.04
77.1 ± 4.1
87.9 ± 3.5
0.88 ± 0.03
84.1 ± 3.6
92.5 ± 4.0
0.92 ± 0.04
Table 1: Performance comparison on the LangAuto benchmark. We report the metrics for 3 evaluation runs. AD-H † leverages an extra model OPT-350M [ 62 ] for low-level control.
Table 5
Figure 4: Visualization of multi-view future scenes generated by LMGenDrive, showing consistent left-front-right views aligned with driving instructions.
Setting
LangAuto-Tiny
LangAuto-Short
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
baseline (Stage-2: 2w iter, Stage-3: 1w iter)
84.1
92.5
0.92
77.1
87.9
0.88
w/o Stage-2 (Stage-3: 3w iter)
78.0
87.6
0.90
67.9
81.0
0.85
w/o Stage-3 (Stage-2: 3w iter)
80.1
88.5
0.91
72.3
83.7
0.86
Table 4: Training-stage ablation under controlled iterations. We report DS/RC/IS on LangAuto-Tiny and LangAuto-Short.
Horizon (frames)
8
16
24
32
64
128
FID ↓
6.3
7.4
7.8
9.2
12.8
15.1
FVD ↓
286
340
371
435
610
981
Table 5: Scaling of long-horizon video generation. FID/FVD degrade with longer prediction horizons due to accumulated errors.
World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, yet existing approaches typically treat perception, prediction, and planning as separate modules. We propose UniDrive-WM, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture. UniDrive-WM's trajectory planner predicts a future trajectory, which conditions a VLM-based image generator to produce plausible future frames. These predictions provide additional supervisory signals that enhance scene understanding and iteratively refine trajectory generation. We further compare discrete and continuous output representations for future image prediction, analyzing their influence on downstream driving performance. Experiments on the challenging Bench2Drive benchmark show that UniDrive-WM produces high-fidelity future images and improves planning performance by 7.3% in L2 trajectory error and 10.4% in collision rate over the previous best method. These results demonstrate the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving. The project page is available at https://unidrive-wm.github.io/UniDrive-WM.
Zhexiao Xiong, Xin Ye, Burhan Yaman +5
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Shuzheng Wang, Haoxuan Li, Shengzhen Li +5
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.
Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io
Guoqing Wang, Pin Tang, Xiangxuan Ren +3
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei