Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.
Figures & tables
Figure 2: Overview of our model. Compared to traditional imitation driving models, which can only predict the single expert trajectory, Vega can follow natural language instructions to generate diverse planning trajectories and future image predictions.
Figure 3: Framework of our Unified Vision-Language-World-Action Model. We jointly model action planning and image generation using multi-modal inputs and a MoT architecture [ 38 ] .
Method
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Ego Status
93.1
77.9
92.7
99.6
86.0
91.5
89.4
98.3
85.4
64.0
TransFuser [ 4 ]
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
HydraMDP++ [ 36 ]
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4
DriveSuprim [ 76 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
ARTEMIS [ 11 ]
98.3
95.1
98.6
99.8
81.5
97.4
96.5
98.3
-
83.1
DiffusionDrive [ 39 ]
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
Table 1: Comparison with state-of-the-art methods on the NAVSIM v2 with extended metrics. NC: No at-fault Collision. DAC: Drivable Area Compliance. DDC: Driving Direction Compliance. TLC: Traffic Light Compliance. EP: Ego Progress. TTC: Time to Collision. LK: Lane Keeping. HC: History Comfort. EC: Extended Comfort. EPDMS: Extended Predictive Driver Model Score. †: Using the best-of-N (N=6) strategy following [ 85 ] .
Method
Ref
Sensors
NC ↑
DAC ↑
TTC ↑
C. ↑
EP ↑
PDMS ↑
Human
-
-
100
100
100
99.9
87.5
94.8
BEV-based Methods
UniAD [ 19 ]
CVPR’23
6x Cam
97.8
91.9
92.9
100.0
78.8
83.4
TransFuser [ 4 ]
TPAMI’23
3x Cam + L
97.7
92.8
92.8
100.0
79.2
84.0
PARA-Drive [ 68 ]
CVPR’24
6x Cam
97.9
92.4
93.0
99.8
79.3
84.0
LAW [ 32 ]
ICLR’25
1x Cam
96.4
95.4
88.7
99.9
81.7
84.6
Table 2: Comparison with state-of-the-art methods on the NAVSIM v1. NC: No at-fault Collision. DAC: Drivable Area Compliance. TTC: Time-To-Collision. C.: Comfort. EP: Ego Progress. PDMS: Predictive Driver Model Score. Abbreviations: 1x Cam (single front-view camera), Nx Cam (surround-view cameras), L (LiDAR). †: Using the best-of-N (N=6) strategy following [ 85 ] .
Figure 4: Ablation of interleaving image-action sequences . We compare the training losses of models trained on non-interleaving sequences (original) and interleaving sequences of different lengths.
Figure 5: Instruction-based planning examples . We visualize the effects of language instructions on action planning with front-view camera images and BEV maps.
Figure 6: Future image generation conditioned on instructions and actions. In the same scenario, given three sets of instructions, the model plans three action sequences and generates their respective future images. All action sequences follow their instructions and all images are consistent with their actions.
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5