Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusion
Authors: Jiaru Zhang, Manav Gagvani, Can Cui, Juntong Peng, Ruqi Zhang, Ziran Wang
Organizations: Institute for Physical Artificial Intelligence (IPAI), Purdue University, West Lafayette, IN 47907, USA · College of Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47907, USA
Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged as promising candidates for end-to-end autonomous driving. However, these models typically face challenges in inference latency, action precision, and explainability. Existing autoregressive approaches struggle with slow token-by-token generation, while prior diffusion-based planners often rely on verbose, general-purpose language tokens that lack explicit geometric structure. In this work, we propose Masked Vision-Language-Action Diffusion for Autonomous Driving (MVLAD-AD), a novel framework designed to bridge the gap between efficient planning and semantic explainability via a masked vision-language-action diffusion model. Unlike methods that force actions into the language space, we introduce a discrete action tokenization strategy that constructs a compact codebook of kinematically feasible waypoints from real-world driving distributions. Moreover, we propose geometry-aware embedding learning to ensure that embeddings in the latent space approximate physical geometric metrics. Finally, an action-priority decoding strategy is introduced to prioritize trajectory generation. Extensive experiments on nuScenes and derived benchmarks demonstrate that MVLAD-AD achieves superior efficiency and outperforms state-of-the-art autoregressive and diffusion baselines in planning precision, while providing high-fidelity and explainable reasoning.
Figures & tables
Fig. 1: Motivation. Comparison of paradigms for LLM/VLM-based autonomous driving. (A) Autoregressive VLMs suffer from high latency due to token-by-token sequential generation. (B) Standard diffusion language models enable parallel generation but operate in a verbose language space. (C) Our proposed Vision-Language-Action Diffusion Model incorporates an expressive codebook to map continuous actions into compact discrete tokens. This design enables simultaneous parallel generation in both action and language spaces, significantly reducing sequence length and achieving the fastest inference speed.
Fig. 2: An overview of MVLAD-AD. (A) Discrete Action Tokenization and Geometry-Aware Embedding Learning: We construct a compact codebook of driving actions from real-world data and learn a geometry-aware embedding space via soft-assignment and geometric consistency objectives. (B) Unified Masked VLA Diffusion: Visual, instruction, action, and reasoning tokens are unified into a single sequence for masked generative modeling. (C) Optimized Training & Inference: During training, we employ a two-stage learning strategy to assist training. During inference, an action-priority decoding strategy is introduced to prioritize trajectory generation for low latency, ensuring the reasoning explanation is highly faithful to the driving actions.
Method
1s ( ↓ )
2s ( ↓ )
3s ( ↓ )
Avg ( ↓ )
FR ( ↓ )
LLaVA-1.6
0.91
2.50
3.44
2.28
55.25%
Llama-3.2
0.80
2.31
3.10
2.07
0.06%
Qwen2-VL
1.32
2.94
3.98
2.74
0.03%
OpenEMMA
1.45
3.21
3.76
2.81
16.11%
LightEMMA
-
-
2.90
1.45
0.00%
UniAD-Single
-
-
-
1.80
-
TABLE I: Planning Errors ( m ) at Different Horizons and Planning Failure Rate (FR). Dashes indicate that the method does not report the corresponding results.
Fig. 3: Planning inference time comparison among autoregressive and diffusion-based methods on a single NVIDIA A100 GPU.
Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.