Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.
Figures & tables
Figure 2: Overview of our model. Compared to traditional imitation driving models, which can only predict the single expert trajectory, Vega can follow natural language instructions to generate diverse planning trajectories and future image predictions.
Figure 3: Framework of our Unified Vision-Language-World-Action Model. We jointly model action planning and image generation using multi-modal inputs and a MoT architecture [ 38 ] .
Method
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Ego Status
93.1
77.9
92.7
99.6
86.0
91.5
89.4
98.3
85.4
64.0
TransFuser [ 4 ]
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
HydraMDP++ [ 36 ]
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4
DriveSuprim [ 76 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
ARTEMIS [ 11 ]
98.3
95.1
98.6
99.8
81.5
97.4
96.5
98.3
-
83.1
DiffusionDrive [ 39 ]
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
Table 1: Comparison with state-of-the-art methods on the NAVSIM v2 with extended metrics. NC: No at-fault Collision. DAC: Drivable Area Compliance. DDC: Driving Direction Compliance. TLC: Traffic Light Compliance. EP: Ego Progress. TTC: Time to Collision. LK: Lane Keeping. HC: History Comfort. EC: Extended Comfort. EPDMS: Extended Predictive Driver Model Score. †: Using the best-of-N (N=6) strategy following [ 85 ] .
Method
Ref
Sensors
NC ↑
DAC ↑
TTC ↑
C. ↑
EP ↑
PDMS ↑
Human
-
-
100
100
100
99.9
87.5
94.8
BEV-based Methods
UniAD [ 19 ]
CVPR’23
6x Cam
97.8
91.9
92.9
100.0
78.8
83.4
TransFuser [ 4 ]
TPAMI’23
3x Cam + L
97.7
92.8
92.8
100.0
79.2
84.0
PARA-Drive [ 68 ]
CVPR’24
6x Cam
97.9
92.4
93.0
99.8
79.3
84.0
LAW [ 32 ]
ICLR’25
1x Cam
96.4
95.4
88.7
99.9
81.7
84.6
Table 2: Comparison with state-of-the-art methods on the NAVSIM v1. NC: No at-fault Collision. DAC: Drivable Area Compliance. TTC: Time-To-Collision. C.: Comfort. EP: Ego Progress. PDMS: Predictive Driver Model Score. Abbreviations: 1x Cam (single front-view camera), Nx Cam (surround-view cameras), L (LiDAR). †: Using the best-of-N (N=6) strategy following [ 85 ] .
Figure 4: Ablation of interleaving image-action sequences . We compare the training losses of models trained on non-interleaving sequences (original) and interleaving sequences of different lengths.
Figure 5: Instruction-based planning examples . We visualize the effects of language instructions on action planning with front-view camera images and BEV maps.
Figure 6: Future image generation conditioned on instructions and actions. In the same scenario, given three sets of instructions, the model plans three action sequences and generates their respective future images. All action sequences follow their instructions and all images are consistent with their actions.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.
Xinyang Wang, Qian Liu, Wenjie Ding +7
State Key Lab of CAD&CG, Zhejiang University · Li Auto
Instruction-conditioned trajectory prediction is an emerging problem in autonomous driving, where a model predicts the future ego trajectory not only from visual scene context and historical motion, but also from a natural-language maneuver instruction. This paper presents our submission to the doScenes Instructed Driving Challenge, built upon OmniDrive, a vision-language-action driving agent with 3D perception, reasoning, and planning capabilities. We adapt OmniDrive to the doScenes setting by training it on instruction-annotated nuScenes scenes and generating a 6-second ego trajectory represented by 12 future waypoints. To improve multi-view visual grounding, we further introduce a DVPE-style divided-view perception module into the OmniDrive perception head. Instead of attending globally to all camera features, the proposed module groups query features and image tokens into divided local view spaces and performs visibility-aware cross-attention within each view. This design reduces irrelevant cross-view interference and helps the model better align language instructions with local driving-relevant visual evidence. The code is publicly available at: https://github.com/feel12348/doscenes-omnidrive.
Zijian Fu, Xiangyang Chu, Mengshi Qi +3
Beijing University of Posts and Telecommunications · Xiaomi EV