VLMs for Autonomous Driving

VLM: Vision-Language Model

Ask about this topic

Overview

This field studies how models that combine visual and language understanding can interpret traffic scenes, reason about motion and hazards, and help vehicles choose safe actions. Approaches include adapting pretrained models with driving data, grounding them in 3D or bird’s-eye-view geometry, fusing camera, LiDAR, and radar inputs, and connecting reasoning to trajectory planners or predictive world models. Researchers evaluate scene understanding and prediction on datasets such as nuScenes, and test interactive driving in closed-loop simulators such as Bench2Drive and CARLA; reinforcement learning and structured feedback are increasingly used to improve planning. A central challenge is that fluent explanations and strong static-scene recognition do not guarantee grounded, temporally reliable, safe actions under real-time and changing conditions.

Latest papers

CardsList

192 papers23 papers in the last four weeks, up 10% on the four weeks before. 0.1% of all new papers.

  1. ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

    Apr 3, 2026Zihao Sheng, Xin Ye, Jingru Luo +2World Model LearningVLMs for Autonomous Driving

    Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison

  2. V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views

    Apr 3, 2026Junwei You, Pei Li, Zhuoyu Jiang +7Autonomous Driving BenchmarksVLMs for Autonomous Driving

    Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China +1

  3. Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

    Apr 1, 2026Xiaosong Jia, Yuqian Shao, Zhenjie Yang +3Autonomous Driving BenchmarksVLMs for Autonomous Driving

    Institute of Trustworthy Embodied AI, Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, 200441, China. · School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200240, China.

  4. A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

    Mar 30, 2026Kunal Runwal, Swaraj Gajare, Daniel Adejumo +3Anomaly DetectionVLMs for Autonomous Driving

    Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.

  5. AutoDrive-P3AutoDrive\text{-}P^3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

    Mar 30, 2026Yuqi Ye, Zijian Zhang, Junhong Lin +3Autonomous Driving PlanningVLMs for Autonomous Driving

    School of Electronic and Computer Engineering, Peking University

  6. Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving

    Mar 28, 2026Qiqi Liu, Huan Xu, Jingyu Li +5VLMs for Autonomous DrivingWorld Model-Based Planning

    Fudan University · Shanghai Innovation Institute · Li Auto Inc. +1

  7. Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets

    Mar 26, 2026Alex Koran, Dimitrios Sinodinos, Hadi Hojjati +3End-to-End Autonomous DrivingTraffic Accident Anticipation

    McGill University, Montréal, Canada · Mila - Québec AI Institute, Montréal, Canada · Nissan Motor Corporation, Yokohama, Japan

  8. Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Driving

    Mar 26, 2026Zehao Wang, Huaide Jiang, Shuaiwu Dong +3Autonomous DrivingAutonomous Driving Planning

    University of California, Riverside · University of Michigan

  9. Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment

    Mar 24, 2026Guoyang Zhao, Weiqing Qi, Kai Zhang +7Autonomous Driving DatasetsAutonomous Driving Perception

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Lingnan University, Hong Kong SAR, China · Shenzhen Unity Drive Innovation Technology Co., Ltd., Shenzhen 518063, China +1

  10. DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

    Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous DrivingLLM-Guided RL

    Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada

  11. VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events

    Mar 18, 2026Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7VLM Post-TrainingVLMs for Autonomous Driving

    NVIDIA, California, USA

  12. Learning from Mistakes: Post-Training for Driving VLA with Takeover Data

    Mar 16, 2026Yinfeng Gao, Deqing Liu, Qichao Zhang +7Autonomous DrivingVLMs for Autonomous Driving

    University of Science and Technology Beijing · Xiaomi EV · Institute of Automation, Chinese Academy of Sciences

  13. AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving

    Mar 16, 2026Wenhui Huang, Songyan Zhang, Qihang Huang +6Mixture-of-TransformersEnd-to-End Autonomous Driving

    Nanyang Technological University, Singapore · Harvard University, US · Xiaomi EV, China

  14. WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning

    Mar 15, 2026Stefan Englmeier, Katharina Winter, Fabian B. FlohrVLMs for Autonomous DrivingWorld Models for Autonomous Driving

    Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany

  15. DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

    Mar 11, 2026Mingzhe Tao, Ruiping Liu, Junwei Zheng +6VLMs for Autonomous DrivingMultimodal Large Language Models

    Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation · Hunan University

  16. Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning

    Mar 11, 2026Yuehao Song, Shaoyu Chen, Hao Gao +8End-to-End Autonomous DrivingAutonomous Driving Planning

    Huazhong University of Science & Technology · Horizon Robotics

  17. A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms

    Mar 11, 2026Kejin Yu, Yuhan Sun, Taiqiang Wu +5Autonomous DrivingVLMs for Autonomous Driving

    1Tsinghua University · East China Normal University · 3The University of Hong Kong

  18. Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning

    Mar 10, 2026Chun-Peng Chang, Chen-Yu Wang, Holger Caesar +1PEFT of VLMsTemporal Reasoning in VLMs

    DFKI Augmented Vision · TU Delft

  19. Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

    Mar 9, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +3Adversarial Attacks on VLMsVLM Robustness

    School of Computing, Clemson University, Clemson, SC, USA

  20. SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving

    Mar 9, 2026Zihan You, Hongwei Liu, Chenxu Dang +4VLMs for Autonomous DrivingVision-Language-Action Models

    Institute for AI Industry Research (AIR), Tsinghua University · School of Instrument Science and Engineering, Southeast University · Zhili College, Tsinghua University +3

  21. NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving

    Mar 9, 2026Ximeng Tao, Pardis Taghavi, Dimitar Filev +2VLMs for Autonomous DrivingAutonomous Driving Planning

    J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA

  22. Perception-Aware Multimodal Spatial Reasoning from Monocular Images

    Mar 7, 2026Yanchun Cheng, Rundong Wang, Xulei Yang +4Visual Spatial ReasoningVLMs for Autonomous Driving

    NUS, Singapore · A*STAR, Singapore · NTU, Singapore +1

  23. BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

    Mar 6, 2026Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1VLMs for Autonomous DrivingEnd-to-End Autonomous Driving

    Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA

  24. Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving

    Mar 6, 2026Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu +3Vision-Language ModelsVLMs for Autonomous Driving

    Department of Electronic and Computer Engineering University of Limerick · Valeo Vision Systems

  25. Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

    Mar 6, 2026Canyu Chen, Yuguang Yang, Zhewen Tan +10Autonomous DrivingVLMs for Autonomous Driving

    National Superior College for Engineers, Beihang University · Institute for AI Industry Research, Tsinghua University · School of Electronic Information Engineering, Beihang University +5

  26. K-Gen: A Multimodal Language-Conditioned Approach for Interpretable Keypoint-Guided Trajectory Generation

    Mar 5, 2026Mingxuan Mu, Guo Yang, Lei Chen +2Trajectory GenerationAutonomous Driving Planning

    Harbin Institute of Technology Chongqing Research Inst. of HIT · Changan Automobile Co., Ltd Chongqing, China

  27. VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

    Mar 3, 2026A. Enes Doruk, Hasan F. Ates3D Occupancy PredictionAutonomous Driving Perception

  28. LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers

    Mar 2, 2026Fabian Schmidt, Karol Fedurko, Markus Enzweiler +1Language-Conditioned Robot PlanningAutonomous Driving Planning

    Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.

  29. LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving

    Mar 2, 2026Yuechen Luo, Fang Li, Shaoqing Xu +10VLMs for Autonomous DrivingVision-Language-Action Models

    1Tsinghua University · 2Xiaomi EV · University of Macau

  30. DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving

    Mar 2, 2026Enhui Ma, Jiahuan Zhang, Guantian Zheng +10Autonomous Driving BenchmarksVLMs for Autonomous Driving

    Autolab, Westlake University · Li Auto Inc · Tsinghua University +1