VLMs for Autonomous Driving

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 45% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs
Jun 22, 2026Xiangbo Gao, Xiukun Huang, Boyu Lu +5VLMs for Autonomous Driving Neuro-Symbolic Reasoning
1Texas A&M University · 2Carnegie Mellon University · University of Maryland +2
Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Driving
Jun 22, 2026Zisheng Chen, Yuping Qiu, Jianhua Han +6VLMs for Autonomous Driving End-to-End Autonomous Driving
Sun Yat-sen University · HKUST(GZ) · Yinwang Intelligent Technology Co. Ltd.
OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs
Jun 21, 2026Hao Vo, Phu Loc Nguyen, Khoa Vo +7Multi-View Geometry Vision-Language Models
University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1
A DVDrive Approach for doScenes Instructed Driving Challenge
Jun 19, 2026Zijian Fu, Xiangyang Chu, Mengshi Qi +3VLMs for Autonomous Driving Vision-Language-Action Models
Beijing University of Posts and Telecommunications · Xiaomi EV
FleetAgent: Teleoperation Assistant for Autonomous Fleets via Vectorized V2N Messages
Jun 19, 2026Juntong Peng, Qi Chen, Deyuan Qu +3Vision-Language Models VLMs for Autonomous Driving
College of Engineering, Purdue University, West Lafayette, IN, USA · Toyota InfoTech Labs, Mountain View, CA, USA
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
Jun 18, 2026Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma +3VLM Evaluation Autonomous Driving Datasets
Artificio Lima, Peru
Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving
Jun 18, 2026Shihao Ji, HongXi Li, Zihui Song +1VLMs for Autonomous Driving End-to-End Autonomous Driving
EventDrive: Event Cameras for Vision-Language Driving Intelligence
Jun 16, 2026Dongyue Lu, Rong Li, Ao Liang +6Autonomous Driving Datasets Event-Based Vision
NUS · IPAL, CNRS IRL 2955, Singapore · HKUST(GZ) +4
DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models
Jun 15, 2026Xinglong Sun, Kevin Xie, Jenny Schmalfuss +5VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
NVIDIA · KE:SAI
When and How Severely: Scenario-Specific Safety Envelopes for Driving VLAs
Jun 12, 2026Abhinaw Priyadershi, Jelena FrtunikjVLMs for Autonomous Driving Language Model Safety Evaluation
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA GmbH, Munich, Germany
VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
Jun 10, 2026Thach Nguyen, Danhua Guo, Tom Lampo +2Autonomous Driving Benchmarks CoT Faithfulness
Uber AV Labs
PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation
Jun 10, 2026Mahmoud Srewa, Praneetsai Iddamsetty, Mohammad Abdullah Al Faruque +1VLMs for Autonomous Driving Autonomous Driving
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA 92697, USA
VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
Jun 10, 2026Jin Yao, Dhruva Dixith Kurra, Tom Lampo +3VLMs for Autonomous Driving End-to-End Autonomous Driving
Uber AV Labs · University of Virginia
Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection
Jun 10, 2026Everett RichardsVLM Evaluation VLM Robustness
Department of Computer Science, San Diego State University.
AutoMine Solution for AV2 2026 Scenario Mining Challenge
Jun 10, 2026Songliang Cao, Jiele Zhao, Yuru Wang +10VLMs for Autonomous Driving Execution-Guided Code Generation
Xiaomi EV · Huazhong University of Science and Technology
Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving
Jun 8, 2026Yimu Wang, Yee Man Choi, Barry Zhang +3VLM Evaluation Autonomous Driving Benchmarks
University of Waterloo
BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving
Jun 7, 2026George Ling, Lijin Yang, Hao Yang +1VLMs for Autonomous Driving Autonomous Driving
Bosch Research
DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving
Jun 7, 2026Qimao Chen, Fang Li, Yuechen Luo +11Autonomous Driving Datasets Autonomous Driving Benchmarks
1Tsinghua University · 2Xiaomi EV
VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planning
Jun 5, 2026Zikai Zhang, Hubert P. H. Shum, Toby P. BreckonAutonomous Driving Datasets VLMs for Autonomous Driving
Department of Computer Science Durham University Stockton Road, Durham, DH1 3LE, UK
MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planning
Jun 2, 2026Letian Chen, Yiren Lu, Justin Fu +5VLMs for Autonomous Driving Multimodal Large Language Models
Waymo LLC.
GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving
Jun 1, 2026Yingzi Ma, Chaowei Xiao, Ming JiangVLM Evaluation Autonomous Driving Benchmarks
University of Wisconsin-Madison · Johns Hopkins University
What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs
Jun 1, 2026Abhishek Aich, Sparsh Garg, Vijay Kumar BG +2VLM Evaluation VLMs for Autonomous Driving
NEC Laboratories, America, USA · University of California, San Diego, USA
DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions
May 29, 2026Weicheng Zheng, Yixin Huang, Qiao Sun +2VLMs for Autonomous Driving Autonomous Driving
1Shanghai Qi Zhi Institute · 3Tongji University · 2IIIS Tsinghua University
Multi-Stage VLM Pipeline for Zero-Shot Traffic Accident Understanding
May 28, 2026Fumiya Tatematsu, Fumihiko TakahashiVLMs for Autonomous Driving Temporal Video Understanding
GO Drive Inc.
ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving
May 27, 2026Mohammadreza Teymoorianfard, Jean-Philippe Monteuuis, Jonathan Petit +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
University of Massachusetts Amherst · Qualcomm
TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Driving
May 26, 2026Jiaxiang Li, Yumao Liu, Ke MaVision-Language Models VLMs for Autonomous Driving
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction
May 23, 2026Naman Mishra, Shankar Gangisetty, C. V. JawaharVLM Evaluation Pedestrian Intention Prediction
CVIT, IIIT-Hyderabad, India.
D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving
May 22, 2026Kevin Richard, Alphin Varghese, Colin Pham +2VLMs for Autonomous Driving Cross-Modal Alignment
University of North Carolina at Charlotte
ChainFlow-VLA: Causal Flow Planning with Vision-Language Models
May 22, 2026Xiyang Wang, Xinlin Wang, Tingguang Zhou +7Trajectory Generation VLMs for Autonomous Driving
1Afari Intelligent Drive · 2Tianjin University · University of Macau
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
May 22, 2026Hao Vo, Khoa Vo, Phu Loc Nguyen +10VLM Evaluation Autonomous Driving Datasets
University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
May 22, 2026Kewei Zhang, Jin Wang, Sensen Gao +9VLMs for Autonomous Driving End-to-End Autonomous Driving
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
May 21, 2026Xiaodong Mei, Diankun Zhang, Hongwei Xie +3VLMs for Autonomous Driving World Models
The Hong Kong University of Science and Technology · Xiaomi EV
Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
May 20, 2026Steven Chen, Shivesh Khaitan, Nemanja DjuricVLMs for Autonomous Driving Autonomous Driving
Aurora Innovation, Inc. Mountain View, CA · Aurora Innovation, Inc. San Francisco, CA · Aurora Innovation, Inc. Pittsburgh, PA
Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAs
May 20, 2026Abhinaw Priyadershi, Jelena FrtunikjVLM Robustness VLMs for Autonomous Driving
NVIDIA Corporation, USA · NVIDIA GmbH, Germany
DriveMA: Rethinking Language Interfaces in Driving VLAs with One-Step Meta-Actions
May 20, 2026Weicheng Zheng, Yixin Huang, Qiao Sun +2VLMs for Autonomous Driving Efficient VLA Models
Shanghai Qi Zhi Institute · Tongji University · Tsinghua University
Grounding Driving VLA via Inverse Kinematics
May 20, 2026Junsung Park, Hyunjung ShimVLMs for Autonomous Driving Vision-Language-Action Models
Korea Advanced Institute of Science and Technology KimJaeChul AI Graduate School 18, Taebong-ro, Seocho-gu, Seoul, South Korea
CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving
May 17, 2026Ruiyang Zhu, Yuehan He, Boyuan Zheng +4VLMs for Autonomous Driving End-to-End Autonomous Driving
University of Michigan · University of Arizona
Is VLA Reasoning Faithful? Probing Safety of Chain-of-Causation in Autonomous Driving Models
May 17, 2026Nicanor Mayumu, Xiaoheng Deng, Patrick MukalaCoT Faithfulness VLMs for Autonomous Driving
School of Computer Science and Engineering Central South University 932 Lushan Road, Changsha, Hunan P. R. China · School of Computer Science University of Wollongong in Dubai UOWD Building, Dubai Knowledge Park Dubai, United Arab Emirates
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
May 16, 2026Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
IIIT-Hyderabad, India
Revealing Interpretable Failure Modes of VLMs
May 12, 2026Isha Chaudhary, Vedaant V Jain, Kavya Sachdeva +2VLM Robustness Vision-Language Models
UIUC · Kumo AI · IIT Delhi
MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
May 12, 2026Yuzhou Huang, Benjin Zhu, Hengtong Lu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
CUHK MMLab · Li Auto · Tsinghua University
OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models
May 12, 2026Seungwoo Roh, Huiyeong Kim, Jong-Chan KimVLMs for Autonomous Driving Efficient VLA Model Inference
Graduate School of Automobile and Mobility, Kookmin University, Korea
CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving
May 11, 2026Minqing Huang, Yujiao Xiang, Zihan Liang +7VLMs for Autonomous Driving Diffusion-Based Planning
Afari Intelligent Drive · University of Electronic Science and Technology of China · Shanghai Jiao Tong University +2
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology
VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving
May 1, 2026Rui Zhao, Haofeng Hu, Zhenhai Gao +2Retrieval-Augmented Generation VLMs for Autonomous Driving
College of Automotive Engineering, Jilin University · The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · ReeFocus AI Technology
Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
Apr 30, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +1Adversarial Attacks on VLMs Vision-Language Models
School of Computing, Clemson University, USA
Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
Apr 30, 2026Lijin Yang, Jianing Huang, Zhongzhan Huang +2VLMs for Autonomous Driving Vision-Language-Action Models
Bosch Research
Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectors
Apr 24, 2026Gautam Kumar Jain, Carsten Markgraf, Julian StählerVisual Question Answering VLMs for Autonomous Driving
Technische Hochschule Augsburg TTZ Landsberg, Landsberg am Lech, Germany
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving
Apr 22, 2026Finn Rasmus Schäfer, Yuan Gao, Dingrui Wang +5VLM Evaluation Autonomous Driving Benchmarks
Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany · Bayerische Motoren Werke AG, Munich, Germany · Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany
SpanVLA: Learning from Negative-Recovery Samples with Fast Action Bridging for Vision-Language-Action Model
Apr 21, 2026Zewei Zhou, Ruining Yang, Xuewei Qi +7VLMs for Autonomous Driving Autonomous Driving
Tony · University of California, Los Angeles, USA · Motional, USA +1
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
Apr 21, 2026Lin Sha, Haiyun Guo, Tao Wang +4VLMs for Autonomous Driving Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Carizon, Beijing 100094, China
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Apr 20, 2026Jinghui Lu, Jiayi Guan, Zhijian Huang +47Trajectory Generation VLMs for Autonomous Driving
OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models
Apr 20, 2026Yiwei Zhang, Xuesong Chen, Jin Gao +5VLMs for Autonomous Driving End-to-End Autonomous Driving
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
Apr 18, 2026Lili Gao, Yanbo Xu, William Koch +8VLMs for Autonomous Driving 3D Scene Generation
1Torc Robotics · 2Princeton University · 3Mila
ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving
Apr 3, 2026Zihao Sheng, Xin Ye, Jingru Luo +2World Model Learning VLMs for Autonomous Driving
Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
Mar 6, 2026Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA
HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving
Feb 1, 2026Weizhe Tang, Junwei You, Jiaxi Liu +7VLMs for Autonomous Driving End-to-End Autonomous Driving
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53711 USA
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
Jan 7, 2026Zhexiao Xiong, Xin Ye, Burhan Yaman +5Vision-Language Models World Model Learning
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4

Jun 22, 2026, cs.AIJ/K move · Enter open · S save

Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs

Xiangbo Gao, Xiukun Huang, Boyu Lu, Junge Zhang+4

1Texas A&M University · 2Carnegie Mellon University · University of Maryland+2

VLMs for Autonomous Driving Neuro-Symbolic Reasoning Autonomous Driving Planning

Open paper AskPDF arXiv

Driving VLA models incorporating Chain-of-Thought (CoT) reasoning are attractive because they leverage pretrained VLM representations and expose intermediate decisions in natural language, yet current rationales often lack the step-by-step decision semantics needed to keep the rationale causally connected to the planned motion. We introduce Neuro-Symbolic Drive, a neuro-symbolic driving framework that supervises a driving VLA with rule-grounded reasoning traces extracted directly from classical rule-based planners. Our key observation is that rule-based planners are symbolic AI systems that already function as executable reasoning engines: they reason about active safety constraints, search over candidate maneuvers, and select a final trajectory. We instrument these planners in simulation to capture both the executed trajectory and the internal decision trace at each rule-evaluation step. Each trace is serialized into structured rule-grounded reasoning and paired with the trajectory to fine-tune Qwen3.5-4B as a driving VLA. Because these traces are derived directly from the planner states that determine the action, they ensure reasoning is structurally coupled to motion generation by construction, rather than by post-hoc alignment. On our simulator-generated benchmark, detailed rule-grounded reasoning reduces ADE@3s from 0.47 to 0.26 and miss rate from 8.30% to 6.40% under three-camera perception, and from 0.54 to 0.26 and 10.13% to 5.99% under eight-camera perception. Neuro-Symbolic Drive thus converts neuro-symbolic planning logic into structured supervision. Code base: https://github.com/XiangboGaoBarry/Neural-Symbolic-Drive.

Momentum

Related topics

Latest papers 122