VLMs for Autonomous Driving

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 45% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
May 22, 2026Kewei Zhang, Jin Wang, Sensen Gao +9VLMs for Autonomous Driving End-to-End Autonomous Driving
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
May 21, 2026Xiaodong Mei, Diankun Zhang, Hongwei Xie +3VLMs for Autonomous Driving World Models
The Hong Kong University of Science and Technology · Xiaomi EV
Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
May 20, 2026Steven Chen, Shivesh Khaitan, Nemanja DjuricVLMs for Autonomous Driving Autonomous Driving
Aurora Innovation, Inc. Mountain View, CA · Aurora Innovation, Inc. San Francisco, CA · Aurora Innovation, Inc. Pittsburgh, PA
Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAs
May 20, 2026Abhinaw Priyadershi, Jelena FrtunikjVLM Robustness VLMs for Autonomous Driving
NVIDIA Corporation, USA · NVIDIA GmbH, Germany
DriveMA: Rethinking Language Interfaces in Driving VLAs with One-Step Meta-Actions
May 20, 2026Weicheng Zheng, Yixin Huang, Qiao Sun +2VLMs for Autonomous Driving Efficient VLA Models
Shanghai Qi Zhi Institute · Tongji University · Tsinghua University
Grounding Driving VLA via Inverse Kinematics
May 20, 2026Junsung Park, Hyunjung ShimVLMs for Autonomous Driving Vision-Language-Action Models
Korea Advanced Institute of Science and Technology KimJaeChul AI Graduate School 18, Taebong-ro, Seocho-gu, Seoul, South Korea
CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving
May 17, 2026Ruiyang Zhu, Yuehan He, Boyuan Zheng +4VLMs for Autonomous Driving End-to-End Autonomous Driving
University of Michigan · University of Arizona
Is VLA Reasoning Faithful? Probing Safety of Chain-of-Causation in Autonomous Driving Models
May 17, 2026Nicanor Mayumu, Xiaoheng Deng, Patrick MukalaCoT Faithfulness VLMs for Autonomous Driving
School of Computer Science and Engineering Central South University 932 Lushan Road, Changsha, Hunan P. R. China · School of Computer Science University of Wollongong in Dubai UOWD Building, Dubai Knowledge Park Dubai, United Arab Emirates
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
May 16, 2026Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
IIIT-Hyderabad, India
Revealing Interpretable Failure Modes of VLMs
May 12, 2026Isha Chaudhary, Vedaant V Jain, Kavya Sachdeva +2VLM Robustness Vision-Language Models
UIUC · Kumo AI · IIT Delhi
MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
May 12, 2026Yuzhou Huang, Benjin Zhu, Hengtong Lu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
CUHK MMLab · Li Auto · Tsinghua University
OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models
May 12, 2026Seungwoo Roh, Huiyeong Kim, Jong-Chan KimVLMs for Autonomous Driving Efficient VLA Model Inference
Graduate School of Automobile and Mobility, Kookmin University, Korea
CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving
May 11, 2026Minqing Huang, Yujiao Xiang, Zihan Liang +7VLMs for Autonomous Driving Diffusion-Based Planning
Afari Intelligent Drive · University of Electronic Science and Technology of China · Shanghai Jiao Tong University +2
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology
VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving
May 1, 2026Rui Zhao, Haofeng Hu, Zhenhai Gao +2Retrieval-Augmented Generation VLMs for Autonomous Driving
College of Automotive Engineering, Jilin University · The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · ReeFocus AI Technology
Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
Apr 30, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +1Adversarial Attacks on VLMs Vision-Language Models
School of Computing, Clemson University, USA
Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
Apr 30, 2026Lijin Yang, Jianing Huang, Zhongzhan Huang +2VLMs for Autonomous Driving Vision-Language-Action Models
Bosch Research
Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectors
Apr 24, 2026Gautam Kumar Jain, Carsten Markgraf, Julian StählerVisual Question Answering VLMs for Autonomous Driving
Technische Hochschule Augsburg TTZ Landsberg, Landsberg am Lech, Germany
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving
Apr 22, 2026Finn Rasmus Schäfer, Yuan Gao, Dingrui Wang +5VLM Evaluation Autonomous Driving Benchmarks
Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany · Bayerische Motoren Werke AG, Munich, Germany · Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany
SpanVLA: Learning from Negative-Recovery Samples with Fast Action Bridging for Vision-Language-Action Model
Apr 21, 2026Zewei Zhou, Ruining Yang, Xuewei Qi +7VLMs for Autonomous Driving Autonomous Driving
Tony · University of California, Los Angeles, USA · Motional, USA +1
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
Apr 21, 2026Lin Sha, Haiyun Guo, Tao Wang +4VLMs for Autonomous Driving Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Carizon, Beijing 100094, China
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Apr 20, 2026Jinghui Lu, Jiayi Guan, Zhijian Huang +47Trajectory Generation VLMs for Autonomous Driving
OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models
Apr 20, 2026Yiwei Zhang, Xuesong Chen, Jin Gao +5VLMs for Autonomous Driving End-to-End Autonomous Driving
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
Apr 18, 2026Lili Gao, Yanbo Xu, William Koch +8VLMs for Autonomous Driving 3D Scene Generation
1Torc Robotics · 2Princeton University · 3Mila
ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving
Apr 3, 2026Zihao Sheng, Xin Ye, Jingru Luo +2World Model Learning VLMs for Autonomous Driving
Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
Mar 6, 2026Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA
HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving
Feb 1, 2026Weizhe Tang, Junwei You, Jiaxi Liu +7VLMs for Autonomous Driving End-to-End Autonomous Driving
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53711 USA
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
Jan 7, 2026Zhexiao Xiong, Xin Ye, Burhan Yaman +5Vision-Language Models World Model Learning
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4

May 22, 2026, cs.CLJ/K move · Enter open · S save

Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

Kewei Zhang, Jin Wang, Sensen Gao, Chengyue Wu+8

1Peking University · 3The University of Hong Kong · 2NVIDIA+1

VLMs for Autonomous Driving End-to-End Autonomous Driving Efficient VLM Inference Blockwise Diffusion Language Models Speculative Decoding Vision-Language-Action Models+2

Open paper AskPDF arXiv

End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking

N

stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to

0.32

m (a

22\%

improvement). When integrated with SGLang, our framework delivers

12\times

throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.

Momentum

Related topics

Latest papers 122