VLMs for Autonomous Driving

VLM: Vision-Language Model

Overview

This field studies how models that combine visual and language understanding can interpret traffic scenes, reason about motion and hazards, and help vehicles choose safe actions. Approaches include adapting pretrained models with driving data, grounding them in 3D or bird’s-eye-view geometry, fusing camera, LiDAR, and radar inputs, and connecting reasoning to trajectory planners or predictive world models. Researchers evaluate scene understanding and prediction on datasets such as nuScenes, and test interactive driving in closed-loop simulators such as Bench2Drive and CARLA; reinforcement learning and structured feedback are increasingly used to improve planning. A central challenge is that fluent explanations and strong static-scene recognition do not guarantee grounded, temporally reliable, safe actions under real-time and changing conditions.

Latest papers

CardsList

221 papers29 papers in the last four weeks, up 26% on the four weeks before. 0.2% of all new papers.

  1. CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving

    Feb 17, 2026Lucas Elbert Suryana, Farah Bierenga, Sanne van Buuren +6Reasoning for VLMsVLM Reasoning

  2. DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving

    Feb 16, 2026Chenxu Dang, Sining Ang, Yongkang Li +7VLMs for Autonomous DrivingDiffusion-Based Planning

  3. HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

    Feb 1, 2026Weizhe Tang, Junwei You, Jiaxi Liu +7Long-Tail Autonomous DrivingVLMs for Autonomous Driving

  4. UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving

    Jan 7, 2026Zhexiao Xiong, Xin Ye, Burhan Yaman +5Vision-Language ModelsWorld Model Learning

  5. SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

    Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language ModelsVLMs for Autonomous Driving

  6. Predicting Risks Before They Happen: Vision-Language Reasoning for Autonomous Driving

    Nov 28, 2025Jiaxin Liu, Xiangyu Yan, Liang Peng +12VLMs for Autonomous DrivingTraffic Accident Anticipation

  7. FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling

    Nov 16, 2025Kaiser Hamid, Can Cui, Khandakar Ashrafi Akbar +2Gaze ForecastingAutonomous Driving Perception

  8. GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

    Nov 14, 2025Fabian Schmidt, Markus Enzweiler, Abhinav ValadaVision-Language ModelsVLMs for Autonomous Driving

  9. More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models

    Oct 6, 2025Xurui Song, Shuo Huai, JingJing Jiang +2VLMs for Autonomous DrivingReasoning for VLA Models

  10. PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systems

    Aug 7, 2025Qi Guo, Xiaojun Jia, Shanmin Pang +5Adversarial Patch AttacksPhysical Adversarial Attacks

  11. LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

    May 1, 2025Zhijie Qiao, Haowei Li, Zhong Cao +1VLM EvaluationAutonomous Driving Benchmarks