Multi-Agent Orchestration with the Common-Sense Reasoning Capabilities of LLMs for Autonomous Driving

Mehdi Azarafza; Faezeh Pasandideh; Ali Ehteshami Bejnordi; Stefan Henkler; Achim Rettberg

cs.MAAug 20, 2026

Multi-Agent Orchestration with the Common-Sense Reasoning Capabilities of LLMs for Autonomous Driving

Authors: Mehdi Azarafza, Faezeh Pasandideh, Ali Ehteshami Bejnordi, Stefan Henkler, Achim Rettberg

Organizations: Department of Computer Science Hamm-Lippstadt University of Applied Sciences, Germany

Abstract

Autonomous vehicles require robust perception and decision-making capabilities to operate in diverse and unseen scenarios. While reinforcement learning and rule-based methods can provide effective control and safety mechanisms, their performance may degrade in situations requiring contextual reasoning. Large Language Models (LLMs) have demonstrated strong capabilities in understanding multimodal information and generating contextual reasoning, however, their use for direct vehicle control can introduce latency and hallucination risks. To address these limitations, a hybrid framework is proposed. This system uses an orchestrator to coordinate PPO-trained reinforcement learning and PID control, with LLM common-sense reasoning applied throughout the framework. LLM reasoning is further employed iteratively to refine the RL reward function for dynamic driving environments. The proposed framework is evaluated in highly randomized CARLA scenarios under diverse environmental and traffic conditions. The results demonstrate the potential of integrating LLM-based reasoning with conventional autonomous driving methods while retaining structured control and safety mechanism.

Figures & tables

Criteria	LLM a	RL b	Rule-based	Hybrid
Decision-making	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Flexibility	$\checkmark$	$\times$	$\times$	$\checkmark$
Generalization	$\checkmark$	$\times$	$\times$	$\checkmark$
Real-time Response	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Common-Sense Reasoning	$\checkmark$	$\times$	$\times$	$\checkmark$

Table 1: Comparison of LLM, RL, rule-based, and hybrid decision-making methods in autonomous driving

Figure 1: High-level system architecture

Figure 2: Hierarchical system architecture of the proposed multi-agent framework

Strategy	Steer	Curve	Speed	Reasoning
Base	PID	PID	N/A	Common sense
Curve	PID	RL	N/A	Common sense
Speed	PID	N/A	RL	Common sense
Hybrid	PID+RL	RL	RL	Common sense

Table 2: Comparison of control strategies used in autonomous driving.

Controller	State Variable	Value/Scale
RL Speed	Speed	$v/50$
	Target Speed	$v_{\mathrm{target}}/50$
	Front Distance	$d/50$
	Front-Left / Front-Right Distance	$d/30$
RL Curve	Center Offset	$o/2$
	Curvature	$10\kappa$

Table 3: Key state features used by the RL speed and curve controllers.

Panel of Figure 3 — Figure 3: Examples of common-sense reasoning in CARLA. (a) Rule 53: speed reduction under heavy rain. (b) Rule 75: speed reduction when cross traffic cannot be observed

Figure 4: Training behavior of the four curve-steering reward variants

Figure 5: End-of-training assessment of tracking accuracy, steering stability, and curve progression across the four reward variants

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: High-level overview of the proposed multi-agent orchestration with agents

Panel of Figure 7 — Figure 7: Additional examples of common-sense reasoning in CARLA. (a) Short following distance with an approaching storm. (b) Red traffic light under heavy rain.

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving
May 27, 2026Kangyu Wu, Peng Cui, Guoxi Chen +1Autonomous Driving Traffic Accident Anticipation
School of Automation, Southeast University, Nanjing, China · Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Mar 18, 2026Zilin Huang, Zihao Sheng, Zhengyang Wan +4VLMs for Autonomous Driving LLM-Guided RL
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada
Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving
Jul 31, 2026Zeyu Dong, Yimin Zhu, Yu Wu +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Stony Brook University, USA · Rutgers University, USA · Sunrise Technology Inc., USA

May 27, 2026J/K move · Enter open · S save

SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving

Kangyu Wu, Peng Cui, Guoxi Chen, Ya Zhang

School of Automation, Southeast University, Nanjing, China · Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China

Autonomous Driving Traffic Accident Anticipation LLM-Guided RL RL for Autonomous Driving Safe RL

Open paper AskPDF arXiv

Ensuring both safety and efficiency in decision-making for autonomous driving systems remains a fundamental challenge. Traditional Deep Reinforcement Learning (DRL) suffers from unsafe random exploration and slow convergence, while Large Language Models (LLMs) demonstrate inherent latency in real-time inference operations. To address these limitations, this paper proposes SARAD, a novel safety-aware hybrid framework that synergizes LLMs and DRL for autonomous driving. SARAD substitutes the random exploration of DRL with Retrieval-Augmented Generation (RAG)-enhanced, LLM-guided decisions sourced from a dynamic expert knowledge repository. An attention discriminator is proposed to integrate the prior knowledge of LLMs into DRL policy optimization. A collision predictor module, fine-tuned with historical collision data, is further designed to improve vehicle safety. Extensive experiments show that SARAD achieves significant performance improvements in the Highway-Env simulator, validating the effectiveness of the proposed model in autonomous driving.

Abstract

LLM-Guided RL

Explore similar work

SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving