Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Driving

Dechen Xie; Mushuang Liu

cs.LGOct 14, 2026

Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Driving

Authors: Dechen Xie, Mushuang Liu

Organizations: Department of Mechanical Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA 24060, USA

Abstract

This paper presents a comprehensive comparison of four representative reinforcement learning (RL) methods in autonomous vehicle (AV) decision-making, including: Fitted Value Iteration (FVI), Rollout Classification Policy Iteration (RCPI), Deep Q-Network (DQN), and a discrete-action adaptation of Deep Deterministic Policy Gradient (D-DDPG). The four methods are applied to the same AV decision-making problem at an unsignalized intersection and are compared in terms of optimality and safety, training stability, and training and decision cost. RCPI is the closest to a reference solution and has the most stable training process. However, it is the least data-efficient. D-DDPG reaches the second-smallest gap to a reference solution with the highest data efficiency, but has the least training stability. DQN is the most stable model-free method but has the highest collision rate. FVI achieves a lower collision rate than DQN, but is less data-efficient and less stable. Based on these results, we provide suggestions for selecting a method according to whether a model is available, the budget for training, and whether stable training is required.

Figures & tables

	Value-based	Policy-based
Model-based	FVI [ 11 ]	RCPI [ 12 ]
Model-free	DQN [ 13 ]	D-DDPG [ 14 , 15 ]

Table 1: Classification of the four RL methods compared in this paper along the two dimensions.

Figure 1: Illustration of the unsignalized intersection.

Figure 2: Intersection layouts C1–C4. Vehicle 1 (red) is the controlled vehicle. A surrounding vehicle is conflicting (dark blue) or non-conflicting (light blue).

	C1		C2		C3		C4		All
Method	Gap	Col.%	Gap	Col.%	Gap	Col.%	Gap	Col.%	Gap	Col.%
RCPI	-0.15	0.00	0.16	0.00	0.63	0.00	1.56	0.02	0.55	0.01
FVI	11.25	0.00	18.06	0.00	23.30	0.02	31.29	0.38	20.98	0.10
DQN	7.24	0.00	11.11	0.16	19.98	0.36	35.26	1.06	18.40	0.40
D-DDPG	6.03	0.00	6.60	0.00	9.96	0.12	17.09	0.40	9.92	0.13

Table 2: Optimality gap and collision rate over four layouts. The smallest gap in each column is in bold. Col.% is the collision rate in percent.

Figure 4: Discounted reward of the four methods versus the number of network updates.

Method	Drop
FVI	$0.356\pm 0.169$
RCPI	$0.004\pm 0.001$
DQN	$0.105\pm 0.011$
D-DDPG	$0.946\pm 0.496$

Table 3: Drop from the peak discounted reward to the discounted reward at the end of training. Mean

\pm

standard deviation over five seeds.

Figure 5: Discounted reward of the four methods versus the number of experiences used (log scale). The dashed lines mark

2\bar{V}^{\mathrm{DE}}

and

1.5\bar{V}^{\mathrm{DE}}

	Experiences		Decision
Method	Peak	Total	time (ms)
RCPI	$(3.62\pm 0.11)\times 10^{10}$	$3.79\times 10^{10}$	$0.051\pm 0.003$
FVI	$(4.46\pm 1.53)\times 10^{8}$	$1.44\times 10^{9}$	$0.227\pm 0.007$
DQN	$(1.54\pm 0.04)\times 10^{8}$	$1.60\times 10^{8}$	$0.048\pm 0.003$
D-DDPG	$(5.47\pm 1.68)\times 10^{7}$	$1.60\times 10^{8}$	$0.050\pm 0.003$
DE	—	—	$370{,}416\pm 9{,}326$

Table 4: Experience usage and decision time of each method. Mean

\pm

SD over five seeds.

RL for Autonomous Driving

Reinforcement learning for autonomous driving trains vehicle policies through interaction to plan, control, and coordinate safely in traffic.

See all 120 papers

Safe Reinforcement Learning of Autonomous Highway Driving: A Unified Framework for Safety and Efficiency
Jun 12, 2026Chufei Yan, Zhihao Cui, Yiyan Lv +3Reinforcement Learning Mixture of Experts
School of Physics, Northeast Normal University, Changchun, 130024, China. · 2Clean Energy Automotive Engineering Center, School of Automotive Studies, Tongji University, Shanghai, 201804, China. · 3Mengshi Automobile Technology Company, Dongfeng Motor Corporation, Wuhan, 430010, China.
DriveMLP: When Training Strategy Outperforms Model Complexity in Autonomous Driving
Oct 21, 2026Alessandro Canevaro, Julian Schmidt, Hang Yu +4Autonomous Driving Planning Imitation Learning
Mercedes-Benz AG, Research & Development, Sindelfingen, Germany · University of Tübingen, Tübingen, Germany · Karlsruhe Institute of Technology, ITIV, Karlsruhe, Germany +2
Autonomous Vehicle Collision Avoidance With Racing Parameterized Deep Reinforcement Learning
Apr 17, 2026Shathushan Sivashangaran, Vihaan Dutta, Apoorva Khairnar +2OOD Generalization Autonomous Driving
Virginia Commonwealth University, Richmond, VA 23284, USA · University of Michigan, Ann Arbor, MI 48109, USA · Virginia Tech, Blacksburg, VA 24061, USA

Jun 12, 2026J/K move · Enter open · S save

Safe Reinforcement Learning of Autonomous Highway Driving: A Unified Framework for Safety and Efficiency

Chufei Yan, Zhihao Cui, Yiyan Lv, Taojie Chen+2

School of Physics, Northeast Normal University, Changchun, 130024, China. · 2Clean Energy Automotive Engineering Center, School of Automotive Studies, Tongji University, Shanghai, 201804, China. · 3Mengshi Automobile Technology Company, Dongfeng Motor Corporation, Wuhan, 430010, China.

Reinforcement Learning Mixture of Experts Autonomous Driving RL for Autonomous Driving Safe RL

Open paper AskPDF arXiv

Deep reinforcement learning (DRL) offers a compelling route to decision-making for advanced autonomous vehicles (AVs), yet its trial-and-error nature makes it difficult to guarantee safety during training and to achieve both safety and efficiency at deployment. We propose a unified safe reinforcement learning (SRL) framework that integrates safe distance (SD), reward machines (RM), and mixture-of-experts (MoE), termed MoE-RM-SRL. For deployment, SD and RM jointly shape a rule-aware reward that encodes highway traffic regulations and stage-wise objectives, enabling safe and reliable behavior without sacrificing efficiency. For training, we introduce a sparsely gated MoE layer comprising up to 11 deep Q-networks (DQNs); an SD-based gating rule activates a minimal set of experts for lane-keeping and lane-changing, mitigating the instability, discontinuities, and impulsive transients commonly induced by switching between heterogeneous controllers (e.g., MPC/rule-based modules and learned policies). We implement the proposed architecture in CARLA and integrate it with a 6-DoF driver-in-the-loop virtual-reality (DiL-VR) platform. Experiments in stochastic two-lane traffic show that MoE-RM-SRL substantially improves safety and efficiency over state-of-the-art baselines, and the framework naturally extends to multi-lane driving as well as on-ramp merging and exiting scenarios.

Abstract

RL for Autonomous Driving

Explore similar work

Safe Reinforcement Learning of Autonomous Highway Driving: A Unified Framework for Safety and Efficiency

DriveMLP: When Training Strategy Outperforms Model Complexity in Autonomous Driving

Autonomous Vehicle Collision Avoidance With Racing Parameterized Deep Reinforcement Learning