PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning

Yinfeng Gao; Qichao Zhang; Deqing Liu; Zhongpu Xia; Guang Li; Kun Ma; Guang Chen; Hangjun Ye

cs.ROMar 16, 2026

PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning

Authors: Yinfeng Gao, Qichao Zhang, Deqing Liu, Zhongpu Xia, Guang Li, Kun Ma, Guang Chen, Hangjun Ye, +3 more

Organizations: School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China. · Xiaomi EV. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.

Abstract

End-to-end autonomous driving policies based on Imitation Learning (IL) often struggle in closed-loop execution due to the misalignment between inadequate open-loop training objectives and real driving requirements. While Reinforcement Learning (RL) offers a solution by directly optimizing driving goals via reward signals, the rendering-based training environments introduce the rendering gap and are inefficient due to high computational costs. To overcome these challenges, we present a novel Pseudo-simulation-based RL method for closed-loop end-to-end autonomous driving, PerlAD. Based on offline datasets, PerlAD constructs a pseudo-simulation that operates in vector space, enabling efficient, rendering-free trial-and-error training. To bridge the gap between static datasets and dynamic closed-loop environments, PerlAD introduces a prediction world model that generates reactive agent trajectories conditioned on the ego vehicle's plan. Furthermore, to facilitate efficient planning, PerlAD utilizes a hierarchical decoupled planner that combines IL for lateral path generation and RL for longitudinal speed optimization. Comprehensive experimental results demonstrate that PerlAD achieves state-of-the-art performance on the Bench2Drive benchmark, surpassing the previous E2E RL method by 10.29% in Driving Score without requiring expensive online interactions. Additional evaluations on the DOS benchmark further confirm its reliability in handling safety-critical occlusion scenarios.

Figures & tables

Fig. 1: Different training paradigms of E2E autonomous driving.

Fig. 2: The framework of PerlAD. (a) The RL training loop. An offline dataset initializes motion states for the pseudo-simulation and provides sensor observations for the E2E model. The E2E model generates reactive agent predictions and planning actions, which are then provided to the simulation to compute rewards. (b) The pseudo-simulation environment. It is responsible for simulating future scenarios and calculating rewards. (c) The structure of the E2E model. The sparse perception extracts structured representations, which are then processed by unified transformer blocks for feature interaction. This is followed by a decoupled planner that outputs lateral and longitudinal actions, and a prediction world model that generates reactive agent trajectories.

Fig. 3: PerlAD adopts a hierarchical decoupled planning scheme: the lateral planner outputs multi-modal future paths, while the longitudinal planner generates target speeds conditioned on the selected path.

Fig. 4: The Prediction World Model autoregressively generates multi-modal trajectory predictions for surrounding agents, explicitly conditioned on the ego’s future trajectory. The highest-probability modality is selected for subsequent simulation and reward computation.

Methods	Reference	Closed-loop Metric $\uparrow$				Multi-ability (%) $\uparrow$
Methods	Reference	DS	SR (%)	Efficiency	Comfortness	Merging	Overtaking	Emergency Brake	Give Way	Traffic Sign	Mean
UniAD [ 1 ]	CVPR 23	45.81	16.36	129.21	43.58	12.16	20.00	23.64	10.00	13.89	15.89
VAD [ 2 ]	ICCV 23	42.35	15.00	157.94	46.01	7.14	20.00	16.36	20.00	20.22	16.75
SparseDrive [ 13 ]	ICRA 25	44.54	16.71	170.21	48.63	12.18	23.19	17.91	20.00	20.98	17.45
DriveTrans.-L. [ 3 ]	ICLR 25	63.46	35.01	100.64	20.78	17.57	35.00	48.36	40.00	52.10	38.60
DiffAD [ 15 ]	arXiv 25	67.92	38.64	-	-	30.00	35.55	46.66	40.00	46.32	38.79

Table I: Closed-loop and Multi-ability Results of E2E-AD Methods on Bench2Drive Leaderboard.

Methods	Driving Score (DS) in DOS $\uparrow$
Methods	DOS_01	DOS_02	DOS_03	DOS_04	Average
UniAD [ 1 ]	66.00	69.32	73.18	75.87	71.09
VAD [ 2 ]	58.46	57.84	61.16	65.88	60.84
LMDrive [ 17 ]	64.00	62.00	73.65	58.54	64.53
ReasonPlan [ 18 ]	67.77	91.65	80.57	77.85	78.02
PerlAD ( Ours )	87.71	87.00	87.40	85.21	86.83

Table II: Closed-loop Results on DOS. We report the Driving Score across four occlusion scenarios: Parked Cars (DOS_01), Sudden Brake (DOS_02), Left Turn (DOS_03), and Red Light Infraction (DOS_04).

Methods	Best-of-K (m) $\downarrow$		Top-1 (m) $\downarrow$
Methods	ADE	FDE	ADE	FDE
Vanilla, w/o. action	0.69	0.97	1.44	2.56
PWM, w. predicted action	0.64	0.87	1.31	2.31
PWM, w. ground truth action	0.62	0.85	1.29	2.24

Table III: Improved prediction accuracy from the Prediction World Model . ADE / FDE = Average / Final Displacement Error.

	Counterfactual Collision Numbers $\downarrow$
Methods	High-speed $\rightarrow$	Stationary $\rightarrow$	Total
	Sudden Brake	Abrupt Lane-change
Logged Replay	144 / 160	22 / 40	166 / 200
Vanilla, w/o. action	124 / 160	23 / 40	147 / 200
PWM	57 / 160	13 / 40	70 / 200

Table IV: Improved reactivity from the Prediction World Model .

Training Strategies				Closed-loop Metric
IL-lon	RL-lon	LLA	RTS	DS $\uparrow$	SR (%) $\uparrow$	CR $\downarrow$
✓	$\times$	$\times$	$\times$	32.81	0	0.74
$\times$	✓	$\times$	$\times$	65.01	20	0.39
$\times$	✓	✓	$\times$	70.28	40	0.20
$\times$	✓	✓	✓	74.00	40	0.09

Table V: Ablation on proposed training strategies. LLA and RTS denote lateral-longitudinal alignment and reactive training simulation, respectively.

ID	Reward Terms				Closed-loop Metric
ID	$r^{col}$	$r^{lk}$	$r^{prog}$	$r^{dist}$	DS $\uparrow$	SR (%) $\uparrow$	CR $\downarrow$
1	$\times$	$\times$	$\times$	✓	53.58	20	0.83
2	✓	$\times$	$\times$	✓	63.54	20	0.41
3	✓	✓	$\times$	✓	67.26	40	0.21
4	✓	✓	✓	$\times$	65.66	30	0.31
5	✓	✓	✓	✓	74.00	40	0.09

Table VI: Ablation on reward function design. Total reward includes collision

r^{col}

, lane-keeping

r^{lk}

, progress

r^{prog}

, and distance

r^{dist}

Fig. 5: Ablation on the influence of reactive scenarios portions and cumulative reactive scenarios number. Driving Score on Dev10 is reported.

Explore similar work

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving
Jul 3, 2026Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Qualcomm AI Research
Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving
Mar 14, 2026Zhexi Lian, Haoran Wang, Xuerun Yan +4End-to-End Autonomous Driving RL for Autonomous Driving
Tongji University · Nanyang Technological University · Chery Automobile
DriveZero: End-to-End Driving Beyond Human Demonstrations
Sep 5, 2026Hao He, Chengcheng Hu, Zirun Su +17End-to-End Autonomous Driving RL for Autonomous Driving
Xiaomi EV AD & Robotics, L3 Team

Jul 3, 2026J/K move · Enter open · S save

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh, Fatih Porikli

Qualcomm AI Research

VLMs for Autonomous Driving End-to-End Autonomous Driving RL for Autonomous Driving

Open paper AskPDF arXiv

End-to-end autonomous driving (E2E-AD) aims to directly map raw sensor information to driving actions. Recently, with the rapid advancement of multi-modal large language models (MLLMs), researchers have proposed the paradigm of Vision-Language-Action (VLA) models for E2E-AD, where it seeks to integrate visual perception, language understanding and action prediction within a single policy. However, existing VLA-based policies largely adopts imitation learning, where it only learns to drive by optimizing distance-based metrics w.r.t. logged expert trajectories. Such distribution shift between open-loop training and closed-loop inference leads to suboptimal performance in closed-loop planning. To close this gap, we present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD. We propose to learn a novel residual waypoint policy around the waypoint prior from pretrained VLA policies, effectively harnessing the knowledge within. On another front, one of the key challenges to scale up RL for vision-based policies is the number of parallel simulation environments since RL is data hungry. To that end, we design a heterogeneous pipeline that places the simulator and the VLA learner on distinct compute groups, which allows us to dramatically increase the number of simulation environments running in parallel while avoiding resource contention and maintaining training stability. We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance on the challenging benchmarks of CARLA longest6 v2 and Bench2Drive.

Abstract

Explore similar work

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving

DriveZero: End-to-End Driving Beyond Human Demonstrations