Large Multimodal Models for Embodied Intelligent Driving: Enabling the Next Generation of Autonomous Mobility
Organizations: School of Information and Electrical Engineering, Hebei University of Engineering, Handan 056038, China · School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Department of Electrical and Computer Engineering, Auburn University, Auburn, AL 36849, USA · Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA ·
Abstract
The advent of Large Multimodal Models (LMMs) offers a promising technology to tackle the limitations of modular design in autonomous driving, which often falters in open-world scenarios requiring sustained environmental understanding and logical reasoning. Besides, embodied artificial intelligence facilitates policy optimization through closed-loop interactions to achieve the continuous learning capability, thereby advancing autonomous driving toward embodied intelligent (EI) driving. However, such capability will be constrained by relying solely on LMMs to enhance EI driving without joint decision-making. This article proposes a novel semantics and policy dual-driven hybrid decision framework to tackle this challenge, ensuring continuous learning and joint decision. The framework merges LMMs for semantic understanding and cognitive representation, and deep reinforcement learning for real-time policy optimization. This article particularly details the foundational principles of EI driving and LMMs, and also introduces the emerging opportunities this framework enables, encompassing potential benefits and representative use cases. The proposed framework, evaluated through a lane-change planning case study, achieves at least a 19% performance improvement over the baselines, as verified by simulations. Finally, key open challenges and future research directions are identified to guide subsequent work.
Figures & tables
| Reference | Year | LMMs | DRL | Joint Decision | Main Contributions |
|---|---|---|---|---|---|
| Liao et al. [ 5 ] | They proposed an LMM-based traffic accident anticipation framework through the CoT prompting. By hierarchically fusing video, optical flow, and linguistic descriptions, the early perception capability can be effectively enhanced for high-risk scenarios. | ||||
| Xu et al. [ 6 ] | They developed an interpretable end-to-end autonomous driving system based on the visually instruction-tuned and mixed fine-tuned LMMs. This system achieves the unified modeling from forward-view videos to low-level control commands and natural language explanations. | ||||
| Hu et al. [ 7 ] | They introduced a training-free multimodal LLM method driven by human-centric context and self-uncertainty. This method extracts the scene graphs and integrates the multiple reasoning responses through the evidence theory to recognize the driver states accurately without any training. | ||||
| Zhang et al. [ 9 ] | They designed an embodied AI-enhanced vehicular network framework that integrates the vision-LLMs for semantic extraction and the DRL to stabilize decision under uncertainty. This framework significantly enhances the efficiency of semantic communication via the vision-LLMs and decision through the DRL. | ||||
| Chen et al. [ 10 ] | They presented an embodied AI‑enabled perception and decision framework for the scenarios of connected vehicles. By employing LMMs, this framework allows vehicle-to-vehicle (V2V) data to achieve greater situational awareness and reliable decision in uncertain scenarios. | ||||
| Zhou et al. [ 11 ] | They proposed a framework for an autonomous operation of rail transportation system enabled by embodied AI through integrated perception, execution, and learning. This framework allows the system to improve its adaptability to the environment and enhance autonomous decision capabilities. |
Autonomous Driving Planning
Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.
See all 416 papers