Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.
Figures & tables
Figure 1: Comparison between existing works and ours. Prior works either leverage LLMs/VLMs for multimodal understanding and reasoning, or adopt world models for video-based scene imagination, but treat these capabilities in isolation. In contrast, our proposed LMGenDrive unifies both within a closed-loop end-to-end framework: the LLM interprets and reasons over multimodal inputs, while the world model simulates future scene evolution, together enabling instruction-guided planning, spatio-temporal understanding, and robust long-horizon driving.
Figure 2: Overview of our unified understanding and generation architecture. We start by encoding the language instruction, multi-view RGB images, and the current action into the LLM. Two sets of learnable queries, world query and action query, are then fed into the LLM, and ultimately used to generate the future driving video and corresponding actions. The framework supports two operation modes: (1) offline data generation mode, an autoregressive generation process is adopted, where the last frame of the future video and the predicted action are used as inputs for the next timestep; (2) online planning mode, real-world data are provided as inputs for the following timestep.
Figure 3: Architecture of our world generator. We begin by fusing the world embedding obtained from the LLM with multi-view RGB images. The fused multi-view world embedding is then injected into the diffusion model with the cross-attention mechanism to generate multi-view future videos.
Methods
LangAuto
LangAuto-Short
LangAuto-Tiny
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
LMDrive [ 41 ]
10.7 ± 3.8
16.2 ± 4.9
0.63 ± 0.04
14.2 ± 4.4
20.1 ± 4.4
0.72 ± 0.04
20.1 ± 4.1
24.7 ± 5.1
0.75 ± 0.03
AD-H † [ 63 ]
44.0
53.2
0.83
56.1
68.0
0.78
77.5
85.1
0.91
BEVDriver [ 56 ]
48.9
59.7
0.82
66.7
77.8
0.87
70.2
81.3
0.87
Ours
62.2 ± 3.3
74.5 ± 4.1
0.85 ± 0.04
77.1 ± 4.1
87.9 ± 3.5
0.88 ± 0.03
84.1 ± 3.6
92.5 ± 4.0
0.92 ± 0.04
Table 1: Performance comparison on the LangAuto benchmark. We report the metrics for 3 evaluation runs. AD-H † leverages an extra model OPT-350M [ 62 ] for low-level control.
Table 5
Figure 4: Visualization of multi-view future scenes generated by LMGenDrive, showing consistent left-front-right views aligned with driving instructions.
Setting
LangAuto-Tiny
LangAuto-Short
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
baseline (Stage-2: 2w iter, Stage-3: 1w iter)
84.1
92.5
0.92
77.1
87.9
0.88
w/o Stage-2 (Stage-3: 3w iter)
78.0
87.6
0.90
67.9
81.0
0.85
w/o Stage-3 (Stage-2: 3w iter)
80.1
88.5
0.91
72.3
83.7
0.86
Table 4: Training-stage ablation under controlled iterations. We report DS/RC/IS on LangAuto-Tiny and LangAuto-Short.
Horizon (frames)
8
16
24
32
64
128
FID ↓
6.3
7.4
7.8
9.2
12.8
15.1
FVD ↓
286
340
371
435
610
981
Table 5: Scaling of long-horizon video generation. FID/FVD degrade with longer prediction horizons due to accumulated errors.