Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

Ding Fan; Xuewen Luo; Hwa Hui Tew; Susilawati; Vishnu Monn Baskaran; Fucai Ke

cs.ROAug 23, 2025

Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

Authors: Ding Fan, Xuewen Luo, Hwa Hui Tew, Susilawati, Vishnu Monn Baskaran, Fucai Ke

Organizations: Monash University, Malaysia · University of Utah, USA · Monash University, Australia

Abstract

Personalized autonomous driving requires understanding the driving needs implicit in user requests. This is challenging when driving demonstrations lack style annotations and behavioral adaptation risks degrading planning quality. We propose M-Diffusion Planner with a dual-decoupling design: separating intent understanding from trajectory planning, and general planning capability learning from strategy alignment. A large language model infers driving needs and selects a strategy, which is executed by the corresponding head of a shared diffusion planner. Following imitation pretraining, strategy-specific rewards specialize individual heads without style annotations. We innovatively adapt Group Relative Policy Optimization using surrogate policy-ratio estimation and constrained head-specific updates to mitigate degradation of pretrained planning capabilities. Open-loop and closed-loop evaluations on nuPlan demonstrate distinct strategy-aligned behaviors and competitive planning performance. Further experiments assess intent-to-strategy mapping and collision risk, and analyze the effects of reward design and optimization on strategy alignment and planning quality.

Figures & tables

Figure 1: Comparison of autonomous driving planning paradigms. (A) Imitation learning directly maps inputs to a single trajectory, lacking interactivity and controllability. (B) Label-guided methods introduce external instructions for short-term control, but remain limited in understanding and long-term alignment. (C) Our hybrid framework combines LLM-based intent understanding with RL-guided planning, enabling multi-strategy trajectory generation with improved interactivity and alignment.

Figure 2: Model architecture of M-Diffusion Planner.

Method	Velocity (m/s)	Acceleration (m/s 2 )	Jerk (m/s 3 )	Low Speed (%)	Mid Speed (%)	High Speed (%)
M-Diffusion Planner (Base)	10.59	1.97	2.66	49.61	33.85	16.54
M-Diffusion Planner (Aggressive)	12.50	2.31	2.43	43.49	29.95	26.56
M-Diffusion Planner (Conservative)	9.57	1.80	2.58	56.51	35.81	7.68
M-Diffusion Planner (Comfortable)	11.03	1.72	1.85	47.1	35.2	17.7

Table 1: Open-loop evaluation on 2,000 scenarios comparing different M-Diffusion planners in terms of motion and velocity distributions. We analyze velocity, acceleration, and jerk under different fine-tuning strategies, and group trajectories into low (0–10 m/s), medium (10–20 m/s), and high (

>20

m/s) speed ranges for fair behavioral comparison.

Figure 3: Comparison between the aggressive and base strategies in two representative driving scenarios. The upper plots show velocity profiles, and the lower plots show acceleration magnitudes, with time steps on the horizontal axis. Additional cases are provided in the Appendix.

Method	Val14 (NR)	Val14 (R)	Test14-hard (NR)	Test14-hard (R)	Test14 (NR)	Test14 (R)
Learning-based Methods
PDM-Open* [ 38 ]	53.53	54.24	33.51	35.83	52.81	57.23
UrbanDriver [ 39 ]	68.57	64.11	50.40	49.95	51.83	67.15
GameFormer [ 24 ] w/o refine.	13.32	8.69	7.08	6.69	11.36	9.31
PlanTF [ 40 ]	84.72	76.95	69.70	61.61	85.62	79.58
PLUTO [ 41 ] w/o refine.*	88.89	78.11	70.03	59.74	89.90	78.62

Table 2: Closed-loop evaluation on nuPlan. Bold: column best; “/” denotes results unavailable in the published baseline comparisons; *: pre-searched reference lines or additional proposals used as inputs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

x_{T}\leftarrow\text{Concat}(x_{0}^{\text{obs}},\epsilon),\quad\epsilon\sim\mathcal{N}(0,I)

Appendix

Algorithm 1 Deterministic Trajectory Sampling

Figure 4: Overview of the proposed M-Diffusion framework with RL-based strategy alignment. Given a driving scene and natural language instruction, an LLM first performs semantic understanding and selects the corresponding driving strategy. The selected strategy activates the corresponding output head of the diffusion planner, which generates trajectories from the scene context using that head. To further align generated behaviors with different driving styles, a GRPO-based reinforcement learning optimization module is introduced, where sampled trajectories are evaluated using multiple reward functions and optimized through relative advantage estimation and policy updates. The final planner produces trajectories that simultaneously satisfy scene constraints and high-level driving intentions.

Figure 5: LLM-based intent understanding and strategy selection. The LLM interprets the user’s natural-language input and selects a driving strategy, which activates the corresponding planner output head.

Reward Design	Epoch	Avg. Speed $\downarrow$	nuPlan Score $\uparrow$
Explicit Reward	0	5.0160	93.31
	3	4.6827	91.37
	5	4.4628	86.21
	10	4.2101	78.11
	15	4.1023	72.21
	20	4.0622	62.13

Appendix

Table 3: Effect of explicit and implicit reward designs on conservative-strategy alignment across training epochs. Average speed is evaluated on 500 open-loop scenarios, and nuPlan scores on 100 closed-loop scenarios.

Figure 6: Representative closed-loop simulation cases in the nuPlan environment under different driving strategies.

Figure 7: Comparison between the base model and the conservative driving strategy under closed-loop simulation scenarios in NuPlan. Representative examples illustrate the behavioral differences in trajectory planning and motion patterns across different driving situations.

Model	Accuracy (%)
Gemma 3 12B	97.1
GPT-4o	96.9
Laguna S 2.1	97.6

Appendix

Table 4: Intent classification accuracy on 1,000 human-intent pairs.

Strategy	Collision Rate (%)
Base	0.85
Aggressive	1.20
Conservative	0.90
Comfortable	1.10

Appendix

Table 5: Open-loop collision rates on 2,000 eight-second scenarios.

Method	Epochs	nuPlan Score
REINFORCE	3	61.27
REINFORCE	10	$<20$

Appendix

Table 6: Planning performance during REINFORCE-based alignment.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving
Apr 4, 2026Wenhao Yao, Xinglong Sun, Zhenxin Li +4End-to-End Autonomous Driving Autonomous Driving Planning
Fudan University · NVIDIA
D $^3$ -MoE:Dual Disentangled Diffusion Mixture-of-Experts for Style-Controllable End-to-End Autonomous Driving
Jun 3, 2026Renju Feng, Rukang Wang, Ning Xi +4End-to-End Autonomous Driving Diffusion-Based Planning
Intelligent Transportation Systems Research Center, Wuhan University of Technology, Wuhan, Hubei, China · School of Mechanical and Electronic Engineering, Wuhan University of Technology, Wuhan, Hubei, China · School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei, China +1
Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Driving
Mar 26, 2026Zehao Wang, Huaide Jiang, Shuaiwu Dong +3Autonomous Driving Autonomous Driving Planning
University of California, Riverside · University of Michigan

Apr 4, 2026J/K move · Enter open · S save

HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving

Wenhao Yao, Xinglong Sun, Zhenxin Li, Shiyi Lan+3

Fudan University · NVIDIA

End-to-End Autonomous Driving Autonomous Driving Planning Diffusion-Based Planning RL for Autonomous Driving Multi-Objective Reinforcement Learning

Open paper AskPDF arXiv

End-to-end planning has emerged as a dominant paradigm for autonomous driving, where recent models often adopt a scoring-selection framework to choose trajectories from a large set of candidates, with diffusion-based decoding showing strong promise. However, directly selecting from the entire candidate space remains difficult to optimize, and Gaussian perturbations used in diffusion often introduce unrealistic trajectories that complicate the denoising process. In addition, for training these models, reinforcement learning (RL) has shown promise, but existing end-to-end RL approaches typically rely on a single coupled reward without structured signals, limiting optimization effectiveness. To address these challenges, we propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy that decomposes planning into a coarse-to-fine process. To improve trajectory generation, we introduce Structure-Preserved Trajectory Expansion, which produces realistic candidates while maintaining kinematic structure. For policy learning, we develop Metric-Decoupled Policy Optimization (MDPO) to enable structured RL optimization across multiple driving objectives. Extensive experiments show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM, outperforming prior arts by a huge margin: +2.3 EPDMS on NAVSIM and +4.9 Route Completion on HUGSIM.

Abstract

Autonomous Driving Planning

Explore similar work

HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving

D3^33-MoE:Dual Disentangled Diffusion Mixture-of-Experts for Style-Controllable End-to-End Autonomous Driving

Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Driving

D $^3$ -MoE:Dual Disentangled Diffusion Mixture-of-Experts for Style-Controllable End-to-End Autonomous Driving