VLMs for Autonomous Driving

VLM: Vision-Language Model

This field studies how models that combine visual and language understanding can interpret traffic scenes, reason about motion and hazards, and help vehicles choose safe actions. Approaches include adapting pretrained models with driving data, grounding them in 3D or bird’s-eye-view geometry, fusing camera, LiDAR, and radar inputs, and connecting reasoning to trajectory planners or predictive world models. Researchers evaluate scene understanding and prediction on datasets such as nuScenes, and test interactive driving in closed-loop simulators such as Bench2Drive and CARLA; reinforcement learning and structured feedback are increasingly used to improve planning. A central challenge is that fluent explanations and strong static-scene recognition do not guarantee grounded, temporally reliable, safe actions under real-time and changing conditions.

Latest papers

192 papers23 papers in the last four weeks, up 10% on the four weeks before. 0.1% of all new papers.

Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures
Mar 1, 2026Yuechen Luo, Qimao Chen, Fang Li +5VLMs for Autonomous Driving RL for Autonomous Driving
Tsinghua University · University of Macau · Beijing Jiaotong University
MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving
Feb 25, 2026Lingjun Zhang, Yujian Yuan, Changjie Wu +7Reasoning for VLMs VLMs for Autonomous Driving
Amap, Alibaba Group · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1
VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving
Feb 24, 2026Jie Wang, Guang Li, Zhijian Huang +4Multi-View Feature Fusion VLMs for Autonomous Driving
College of Intelligence and Computing, Tianjin University · Xiaomi EV
An interactive enhanced driving dataset for autonomous driving
Feb 24, 2026Haojie Feng, Xinrui Zhang, Mengjie Tian +8Autonomous Driving Datasets VLMs for Autonomous Driving
School of Automotive Studies, Tongji University, Shanghai, 201804, China · University of Chicago, Chicago, IL 60637, USA · Faculty of Computer Science, University of New Brunswick, Fredericton, NB E3B 5A3, Canada +1
CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving
Feb 17, 2026Lucas Elbert Suryana, Farah Bierenga, Sanne van Buuren +6Reasoning for VLMs VLM Reasoning
Department of Transport & Planning, Faculty of Civil Engineering and Geosciences, Delft University of Technology · Centre for Meaningful Human Control, Delft University of Technology · Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands +1
DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving
Feb 16, 2026Chenxu Dang, Sining Ang, Yongkang Li +7VLMs for Autonomous Driving Diffusion-Based Planning
Huazhong University of Science and Technology · Xiaomi EV · Institute for AI Industry Research (AIR), Tsinghua University
HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving
Feb 1, 2026Weizhe Tang, Junwei You, Jiaxi Liu +7Long-Tail Autonomous Driving VLMs for Autonomous Driving
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53711 USA
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
Jan 7, 2026Zhexiao Xiong, Xin Ye, Burhan Yaman +5Vision-Language Models World Model Learning
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4
GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving
Nov 14, 2025Fabian Schmidt, Markus Enzweiler, Abhinav ValadaVision-Language Models VLMs for Autonomous Driving
Esslingen University of Applied Sciences · University of Freiburg
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
Oct 6, 2025Xurui Song, Shuo Huai, JingJing Jiang +2VLMs for Autonomous Driving Reasoning for VLA Models
S-Lab, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
May 1, 2025Zhijie Qiao, Haowei Li, Zhong Cao +1VLM Evaluation Autonomous Driving Benchmarks
Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA

Mar 1, 2026J/K move · Enter open · S save

Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures

Yuechen Luo, Qimao Chen, Fang Li, Shaoqing Xu+4

Tsinghua University · University of Macau · Beijing Jiaotong University

VLMs for Autonomous Driving RL for Autonomous Driving Autonomous Driving Planning RL for VLMs

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models for autonomous driving often hit a performance plateau during Reinforcement Learning (RL) optimization. This stagnation arises from exploration capabilities constrained by previous Supervised Fine-Tuning (SFT), leading to persistent failures in long-tail scenarios. In these critical situations, all explored actions yield a zero-value driving score. This information-sparse reward signals a failure, yet fails to identify its root cause -- whether it is due to incorrect planning, flawed reasoning, or poor trajectory execution. To address this limitation, we propose VLA with Explicit Learning from Failures (ELF-VLA), a framework that augments RL with structured diagnostic feedback. Instead of relying on a vague scalar reward, our method produces detailed, interpretable reports that identify the specific failure mode. The VLA policy then leverages this explicit feedback to generate a Feedback-Guided Refinement. By injecting these corrected, high-reward samples back into the RL training batch, our approach provides a targeted gradient, which enables the policy to solve critical scenarios that unguided exploration cannot. Extensive experiments demonstrate that our method unlocks the latent capabilities of VLA models, achieving state-of-the-art (SOTA) performance on the public NAVSIM benchmark for overall PDMS, EPDMS score and high-level planning accuracy.