cs.ROJan 13, 2026

Large Multimodal Models for Embodied Intelligent Driving: Enabling the Next Generation of Autonomous Mobility

Authors: Long Zhang, Yuchen Xia, Bingqing Wei, Zhen Liu, Shiwen Mao, Zhu Han, Mohsen Guizani

Organizations: School of Information and Electrical Engineering, Hebei University of Engineering, Handan 056038, China · School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Department of Electrical and Computer Engineering, Auburn University, Auburn, AL 36849, USA · Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA · 

Abstract

The advent of Large Multimodal Models (LMMs) offers a promising technology to tackle the limitations of modular design in autonomous driving, which often falters in open-world scenarios requiring sustained environmental understanding and logical reasoning. Besides, embodied artificial intelligence facilitates policy optimization through closed-loop interactions to achieve the continuous learning capability, thereby advancing autonomous driving toward embodied intelligent (EI) driving. However, such capability will be constrained by relying solely on LMMs to enhance EI driving without joint decision-making. This article proposes a novel semantics and policy dual-driven hybrid decision framework to tackle this challenge, ensuring continuous learning and joint decision. The framework merges LMMs for semantic understanding and cognitive representation, and deep reinforcement learning for real-time policy optimization. This article particularly details the foundational principles of EI driving and LMMs, and also introduces the emerging opportunities this framework enables, encompassing potential benefits and representative use cases. The proposed framework, evaluated through a lane-change planning case study, achieves at least a 19% performance improvement over the baselines, as verified by simulations. Finally, key open challenges and future research directions are identified to guide subsequent work.

Figures & tables

Explore similar work

CardsList
  1. LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

    Apr 9, 2026Hao Shao, Letian Wang, Yang Zhou +5Autonomous DrivingEnd-to-End Autonomous Driving

  2. DriveMind: Mind-Evolving Belief Tracking for Closed-Loop Autonomous Driving

    Oct 23, 2026Yan Shi, Kehua Chen, Jiarui Qi +3Autonomous DrivingVLMs for Autonomous Driving

  3. LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving

    Jun 7, 2026Ruoyu Yao, Pei Liu, Ruiguo Zhong +3Continual Learning for LLMsLanguage Model-Based Control