VLMs for Autonomous Driving

VLM: Vision-Language Model

Overview

This field studies how models that combine visual and language understanding can interpret traffic scenes, reason about motion and hazards, and help vehicles choose safe actions. Approaches include adapting pretrained models with driving data, grounding them in 3D or bird’s-eye-view geometry, fusing camera, LiDAR, and radar inputs, and connecting reasoning to trajectory planners or predictive world models. Researchers evaluate scene understanding and prediction on datasets such as nuScenes, and test interactive driving in closed-loop simulators such as Bench2Drive and CARLA; reinforcement learning and structured feedback are increasingly used to improve planning. A central challenge is that fluent explanations and strong static-scene recognition do not guarantee grounded, temporally reliable, safe actions under real-time and changing conditions.

Latest papers

CardsList

221 papers29 papers in the last four weeks, up 26% on the four weeks before. 0.2% of all new papers.

  1. GridDriver: On-Board BEV-Based Language Planning and Reasoning

    Nov 16, 2026Katharina Winter, William Engel, Zain A. Alishammat +1VLMs for Autonomous DrivingAutonomous Driving Planning

  2. Competent Autonomous Driving Through Fine Tuning Vision Language Action Models with Symbolic Traffic Rules

    Nov 16, 2026Barkin Dagda, Vinal Asodia, Iman Sharifi +2Autonomous DrivingVLMs for Autonomous Driving

  3. VISTA: A Framework for Vision-Informed Semantic Traffic Awareness

    Nov 16, 2026Murat Arda Onsu, Poonam Lohan, Burak Kantarci +3VLMs for Autonomous DrivingTraffic Accident Anticipation

  4. From Cloud to Edge: A Fully Local Pipeline for Pothole Risk Assessment and Driver Warning Generation

    Nov 15, 2026Abdalmalek Aburaddaha, Samir A. RawashdehVLMs for Autonomous DrivingEfficient VLM Inference

  5. Do VLMs Really Look at the Road? A Diagnostic Evaluation for Autonomous Driving

    Nov 14, 2026Shankar Gangisetty, Muskan Singh, C V JawaharVLMs for Autonomous DrivingAutonomous Driving Safety Evaluation

  6. SuperGlance: Driving at a Glance with a Vision-Language Model

    Nov 14, 2026Jieyuan Pei, Xin Yan, Mingwei Xu +9VLMs for Autonomous DrivingEfficient VLA Model Inference

  7. CARE-Drive: Coupled Alternating Refinement for Ego-trajectories

    Nov 11, 2026Xiao Ma, Yijun Chen, Zhiquan Hu +4Vehicle Trajectory PredictionEnd-to-End Autonomous Driving

  8. Towards Robust Vision-Language Reasoning: A Multi-Stage Optimization Pipeline with Masked Diffusion Model

    Nov 11, 2026Shunming Liu, Pei Huang, Yajing Sun +1VLMs for Autonomous DrivingAutonomous Driving Planning

  9. Can MLLMs Reason over Surround-View Driving Videos? A Multi-View Autonomous Driving QA Benchmark

    Nov 11, 2026Shunming Liu, Jianyi Peng, Cheng Li +4VLMs for Autonomous DrivingAutonomous Driving Benchmarks

  10. AutoVLA-D: Learn a Trajectory-Geometric Condition from Reasoning for Autonomous Driving VLA

    Nov 9, 2026Yuguang Yang, Zhewen Tan, Canyu Chen +9Autonomous Driving PlanningVLMs for Autonomous Driving

  11. DeltaForge: A Post-hoc Mixture-of-Experts VLA for Multi-Domain Autonomous Driving

    Nov 9, 2026Bingzhang Wang, Mo Poorsartep, Yasaman EsfandiariVLMs for Autonomous DrivingMixture-of-Experts Models

  12. Selective Cooperative Perception via Blind Spot Reasoning and Agentic Dialogue with Vision-Language Models

    Nov 7, 2026Chengyuan Ma, Shixiao Liang, Hangyu Li +6Cooperative PerceptionCommunication-Efficient Collaborative Perception

  13. RoadState: A Domain-Prior-Enhanced Vision-Language System for Industrial-Grade Lane-Level Map Updating

    Nov 7, 2026Deguo Xia, Zihan Li, Jiaqi Wang +11VLMs for Autonomous DrivingHD Map Construction

  14. Locating and Repairing Domain Shift in VLM Trajectory Planning

    Nov 7, 2026Zhihong Cui, Hengyu Liu, Michael Alexander Riegler +3VLM AdaptationVLMs for Autonomous Driving

  15. Action-Conditioned VLM Monitoring for Autonomous Vehicle Semantic Anomalies

    Nov 5, 2026Daniel Adejumo, Omkar Ankalkope, Kunal Runwal +1Anomaly DetectionVLMs for Autonomous Driving

  16. DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes

    Nov 4, 2026Sicen Guo, Zhengfei Song, Kuanhsun Chen +5VLMs for Autonomous DrivingAutonomous Driving Perception

  17. Should the Action Expert Train the VLM? Action-Token Supervision and Gradient Routing in Mixture-of-Transformer VLAs

    Nov 3, 2026Asher James Hancock, Boyi Li, Yuxiao Chen +6VLA Model TrainingFlow-Based VLA Models

  18. Scene-Adaptive Readout and Decoding in Diffusion Driving VLAs

    Nov 1, 2026Seungjun Yu, Jiho Choi, Inha Kang +3Diffusion-Based VLA ModelsVision-Language-Action Models

  19. Safety4DM: A Vision-Language Framework for Risk-Aware Dynamic Maps in Vehicle-Road-Cloud Collaborative Driving

    Nov 1, 2026Zhengshu Zhou, Shaoyu Wang, Lu Tao +3Autonomous DrivingVLMs for Autonomous Driving

  20. Fewer Tokens, More Answers: Verifiable-Reward Tuning of an Ornith-Style VLM for Multi-View Driving QA

    Oct 31, 2026T N Rickesh, Pranav Vinodh, Sowmya Kamath SVLM ReasoningVisual Question Answering

  21. Answer Correction with Verifiable Rewards: A Gemma-Family Study of Multi-View Driving QA

    Oct 31, 2026T N Rickesh, Pranav Vinodh, Sowmya Kamath SVLM EvaluationVLMs for Autonomous Driving

  22. RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

    Oct 26, 2026Chuchu Xie, Yang Zhong, Kangan Qian +3VLM DistillationKnowledge Distillation

  23. DriveMind: Mind-Evolving Belief Tracking for Closed-Loop Autonomous Driving

    Oct 23, 2026Yan Shi, Kehua Chen, Jiarui Qi +3Autonomous DrivingVLMs for Autonomous Driving

  24. StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA

    Oct 22, 2026Blessing Agyei Kyem, Richard Dzinyela, Joshua Kofi Asamoah +5Video CaptioningVisual Question Answering

  25. SAGE: A hybrid vision-language and geometry framework for work-zone pathway prediction under temporary traffic control

    Oct 22, 2026Jiaxi Liu, Keshu Wu, Weizhe Tang +5Autonomous Driving PlanningMotion Planning