Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving

Zhexi Lian; Haoran Wang; Xuerun Yan; Weimeng Lin; Xianhong Zhang; Yongyu Chen; Jia Hu

cs.ROMar 14, 2026

Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving

Authors: Zhexi Lian, Haoran Wang, Xuerun Yan, Weimeng Lin, Xianhong Zhang, Yongyu Chen, Jia Hu

Organizations: Tongji University · Nanyang Technological University · Chery Automobile

Abstract

End-to-end autonomous driving is typically built upon imitation learning (IL), yet its performance is constrained by the quality of human demonstrations. To overcome this limitation, recent methods incorporate reinforcement learning (RL) through sequential fine-tuning. However, such a paradigm remains suboptimal: sequential RL fine-tuning can introduce policy drift and often leads to a performance ceiling due to its dependence on the pretrained IL policy. To address these issues, we propose PaIR-Drive, a general Parallel framework for collaborative Imitation and Reinforcement learning in end-to-end autonomous driving. During training, PaIR-Drive separates IL and RL into two parallel branches with conflict-free training objectives, enabling fully collaborative optimization. This design eliminates the need to retrain RL when applying a new IL policy. During inference, RL leverages the IL policy to further optimize the final plan, allowing performance beyond prior knowledge of IL. Furthermore, we introduce a tree-structured trajectory neural sampler to group relative policy optimization (GRPO) in the RL branch, which enhances exploration capability. Extensive analysis on NAVSIMv1 and v2 benchmark demonstrates that PaIR-Drive achieves Competitive performance of 91.2 PDMS and 87.9 EPDMS, building upon Transfuser and DiffusionDrive IL baselines. PaIR-Drive consistently outperforms existing RL fine-tuning methods, and could even correct human experts' suboptimal behaviors. Qualitative results further confirm that PaIR-Drive can effectively explore and generate high-quality trajectories.

Figures & tables

Figure 1: Examples of human’s bad behaviors in the real-world dataset NAVSIM. (a) Singapore: The human drives in the wrong direction on the opposite lane; (b) Las Vegas: The human violates traffic light and turns left.

Figure 2: Comparisons of existing training schemes and ours for end-to-end autonomous driving. (a) One-shot IL $\to$ RL: IL-based training with subsequent RL fine-tuning; (b) Iterative IL $\leftrightarrow$ RL: alternately conducting IL training and RL fine-tuning; (c) Ours parallel framework for collaborative IL and RL.

Figure 3: Training process illustration of the parallel scheme of PaIR-Drive. IL branch follows a typical end-to-end planning fashion and is supervised by the human trajectory. Simultaneously, the RL branch builds upon human trajectories and aims to further explore better trajectories. In the RL branch, a tree-structured trajectory neural sampler is designed to recurrently predict the trajectory point offsets of driving intentions unseen in human demonstrations. Finally, we use trajectories and their simulated rewards for GRPO to update the policy.

Figure 4: Inferring process illustration of the parallel scheme of PaIR-Drive. Compared with Fig. 3 , we replace the human trajectory in the RL branch with the trajectory generated by the IL branch, while employing an additional trained reward world model to evaluate and select the final plan.

Figure 5: Illustration of the tree-structured trajectory neural sampler with the capability of generating trajectories under different driving intentions unseen in human demonstrations.

Data split	Agent	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Human bad v2	human	100.0	100.0	97.0	70.4	83.2	99.6	46.2	82.7	56.6	50.0
Human bad v2	human w/ PaIR-Drive	100.0	100.0	98.3	77.5	84.2	99.4	66.5	82.4	49.3	60.8( +10.8 )
Navtest	human	100.0	100.0	99.7	97.4	87.4	100.0	87.4	98.1	90.1	90.3
Navtest	human w/ PaIR-Drive	100.0	100.0	99.9	98.0	89.6	100.0	91.7	98.1	86.4	91.9( +1.6 )

Table 2: The human suboptimal behavior improvement capability evaluation on NAVSIMv2 benchmark. The green score means the improvement compared with the human driver.

Method type	Agent	source	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
IL	VADv2 [ 6 ]	ICCV’23	97.3	91.7	98.2	99.9	77.6	92.7	98.2	100.0	97.4	76.6
	Transfuser w/o RL [ 8 ]	TPAMI’23	97.2	91.8	99.2	99.8	87.6	95.7	95.7	98.4	87.7	79.7
	ARTEMIS [ 13 ]	arxiv’25	98.3	95.1	98.6	99.8	81.5	97.4	96.5	100.0	98.3	83.1
	WOTE [ 33 ]	ICCV’25	98.5	96.8	98.8	99.8	86.1	97.9	95.5	98.3	82.9	84.2
	DiffusionDrive [ 39 ]	CVPR’25	98.0	96.0	99.5	99.8	87.7	97.1	97.2	98.3	87.6	84.3
Sequential IL+RL	RecogDrive w/ GRPO [ 34 ]	arxiv’25	98.3	95.2	99.5	99.8	87.1	97.5	96.6	98.3	86.5	83.6

Table 4: Comparison with SOTA methods on NAVSIMv2 benchmark. The best and second-best scores are highlighted in bold and underlined , respectively. The green score means the improvement compared with the origin IL policy. †: using the best-of-N (N=6) strategy following [ 57 ] .

Figure 6: Comparison between our parallel scheme and conventional sequential scheme. Left is the PDMS improvement results of our PaIR-Drive, and right is the results of conventional sequential-RL-based methods.

Table 5: Ablation studies. The baseline IL policy is Transfuser. offset pred. - predict the trajectory offsets. Traj pred. - generate the whole trajectory directly. All results follow the best-of-N (N=6) strategy [ 57 ] .

Figure 7: Visualization analysis. We compare PaIR-Drive with DiffusionDrive and Transfuser w/ GRPO. Our PaIR-Drive shows (a) better collision avoidance, (b) more compliant with drivable area.

Explore similar work

Mar 16, 2026

PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning

End-to-end autonomous driving policies based on Imitation Learning (IL) often struggle in closed-loop execution due to the misalignment between inadequate open-loop training objectives and real driving requirements. While Reinforcement Learning (RL) offers a solution by directly optimizing driving goals via reward signals, the rendering-based training environments introduce the rendering gap and are inefficient due to high computational costs. To overcome these challenges, we present a novel Pseudo-simulation-based RL method for closed-loop end-to-end autonomous driving, PerlAD. Based on offline datasets, PerlAD constructs a pseudo-simulation that operates in vector space, enabling efficient, rendering-free trial-and-error training. To bridge the gap between static datasets and dynamic closed-loop environments, PerlAD introduces a prediction world model that generates reactive agent trajectories conditioned on the ego vehicle's plan. Furthermore, to facilitate efficient planning, PerlAD utilizes a hierarchical decoupled planner that combines IL for lateral path generation and RL for longitudinal speed optimization. Comprehensive experimental results demonstrate that PerlAD achieves state-of-the-art performance on the Bench2Drive benchmark, surpassing the previous E2E RL method by 10.29% in Driving Score without requiring expensive online interactions. Additional evaluations on the DOS benchmark further confirm its reliability in handling safety-critical occlusion scenarios.

Yinfeng Gao, Qichao Zhang, Deqing Liu +8

School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China. · Xiaomi EV. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.

End-to-End Autonomous Driving RL for Autonomous Driving Autonomous Driving Planning Autonomous Driving Simulation World Models for Autonomous Driving

Ask PDF

Jul 5, 2025

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving

Most end-to-end autonomous driving methods rely on imitation learning from single expert demonstrations, often leading to conservative and homogeneous behaviors that limit generalization in complex real-world scenarios. In this work, we propose DIVER, an end-to-end driving framework that integrates reinforcement learning with diffusion-based generation to produce diverse and feasible trajectories. At the core of DIVER lies a reinforced diffusion-based generation mechanism. First, the model conditions on map elements and surrounding agents to generate multiple reference trajectories from a single ground-truth trajectory, alleviating the limitations of imitation learning that arise from relying solely on single expert demonstrations. Second, reinforcement learning is employed to guide the diffusion process, where reward-based supervision enforces safety and diversity constraints on the generated trajectories, thereby enhancing their practicality and generalization capability. Furthermore, to address the limitations of L2-based open-loop metrics in capturing trajectory diversity, we propose a novel Diversity metric to evaluate the diversity of multi-mode predictions.Extensive experiments on the closed-loop NAVSIM and Bench2Drive benchmarks, as well as the open-loop nuScenes dataset, demonstrate that DIVER significantly improves trajectory diversity, effectively addressing the mode collapse problem inherent in imitation learning.

Ziying Song, Lin Liu, Hongyu Pan +7

Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · School of Artificial Intelligence (School of Software), Yanshan University, Qinhuangdao, China · Horizon Robotics +3

End-to-End Autonomous Driving Diffusion-Based Planning Imitation Learning RL for Diffusion Models RL for Autonomous Driving Trajectory Prediction

Ask PDF

Sep 17, 2026

OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher

As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6

\times

and 9.5

\times

, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: https://01dami23.github.io/opted/

Damiano Da Col, Maximilian Igl, Peter Karkus +5

KE:SAI · ETH Zürich · NVIDIA Research +2

End-to-End Autonomous Driving RL Post-Training RL Fine-Tuning

Ask PDF