$AutoDrive\text{-}P^3$: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

Yuqi Ye; Zijian Zhang; Junhong Lin; Shangkun Sun; Changhao Peng; Wei Gao

cs.ROMar 30, 2026

$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

Authors: Yuqi Ye, Zijian Zhang, Junhong Lin, Shangkun Sun, Changhao Peng, Wei Gao

Organizations: School of Electronic and Computer Engineering, Peking University

Abstract

Vision-language models (VLMs) are increasingly being adopted for end-to-end autonomous driving systems due to their exceptional performance in handling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMs directly output planning results without chain-of-thought (CoT) reasoning, bypassing crucial perception and prediction stages which creates a significant domain gap and compromises decision-making capability; 2) Other VLMs can generate outputs for perception, prediction, and planning tasks but employ a fragmented decision-making approach where these modules operate separately, leading to a significant lack of synergy that undermines true planning performance. To address these limitations, we propose ${AutoDrive\text{-}P^3}$ , a novel framework that seamlessly integrates $\textbf{P}$ erception, $\textbf{P}$ rediction, and $\textbf{P}$ lanning through structured reasoning. We introduce the ${P^3\text{-}CoT}$ dataset to facilitate coherent reasoning and propose ${P^3\text{-}GRPO}$ , a hierarchical reinforcement learning algorithm that provides progressive supervision across all three tasks. Specifically, ${AutoDrive\text{-}P^3}$ progressively generates CoT reasoning and answers for perception, prediction, and planning, where perception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final planning decisions, enabling safer and more interpretable autonomous driving. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-art performance in planning tasks. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3.

Figures & tables

Figure 1: The difference between

{AutoDrive\text{-}P^{3}}

and other paradigms. Our method combines an end-to-end training framework with a three-stage collaborative supervision form with VLM.

Figure 2: Overview of ${AutoDrive\text{-}P^{3}}$ . It processes video and ego vehicle data through structured Perception-Prediction-Planning Chain-of-Thought ( $P^{3}\text{-}CoT$ ) reasoning, generating interpretable step-by-step rationale and structured outputs for perception, prediction, and planning.

Figure 3: The pipeline for constructing ${P^{3}\text{-}CoT}$ dataset. We first sample data and annotations from existing datasets, then construct the labels of samples, focusing on key objects and using rule-based and manual filtering. Finally, with the help of advanced VLM, we construct the CoT, focusing on the connection among perception, prediction and planning three stages.

Figure 4: The pipeline of ${P^{3}\text{-}GRPO}$ . We first cold start the base model using $P^{3}\text{-}CoT$ to make up for the gap between VLM and autonomous driving and learn the CoT answer format. Next we use GRPO to find the best optimization path and update our model.

Figure 5

Non-Autoregressive Methods
Method	L2 (m) $\downarrow$				Collision (%) $\downarrow$				VLM
Method	1s	2s	3s	Avg.	1s	2s	3s	Avg.	VLM
ST-P3 ( ? )	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71	-
VAD ( ? )	0.17	0.34	0.60	0.37	0.07	0.10	0.24	0.14	-
Ego-MLP ( ? )	0.46	0.76	1.12	0.78	0.21	0.35	0.58	0.38	-
UniAD ( ? )	0.44	0.67	0.96	0.69	0.04	0.08	0.23	0.12	-

Table 1: Performance comparison on nuScenes Benchmark.

Method	Image	Lidar	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf $\uparrow$	PDMS $\uparrow$
Human	✗	✗	100.0	100.0	87.5	100.0	99.9	94.8
Constant Velocity	✗	✗	69.9	58.8	49.3	49.3	100.0	21.6
Ego Status MLP	✗	✗	93.0	77.3	62.8	83.6	100.0	65.6
VADv2 ( ? )	✔	✗	97.9	91.7	77.6	92.9	100.0	83.0
UniAD ( ? )	✔	✗	97.8	91.9	78.8	92.9	100.0	83.4
LTF ( ? )	✔	✗	97.4	92.8	79.0	92.4	100.0	83.8

Table 2: Performance comparison on NAVSIMv1 benchmark.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$ False / True
Human	100.0	100.0	99.8	100.0	87.4	100.0	100.0	98.1	90.1	90.3 / 94.5
Ego Status MLP	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0 / -
Transfuser ( ? )	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7 / 84.0
HydraMDP++ ( ? )	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4 / -
DiffusionDrive ( ? )	98.2	96.2	99.5	99.8	87.4	97.3	96.9	98.4	87.7	84.7 / 88.2
WoTE ( ? )	98.5	96.8	98.8	99.8	86.1	97.9	95.5	98.3	82.9	84.2 / 87.7

Table 3: Performance comparison on NAVSIMv2 benchmark.

Method	Perception $\uparrow$	Prediction $\uparrow$	Planning (Collision, %) $\downarrow$
Method	Perception $\uparrow$	Prediction $\uparrow$	1s	2s	3s	Avg.
UniAD ( ? )	0.32	0.31	0.04	0.08	0.23	0.12
OmniDrive ( ? )	0.37	–	0.01	0.04	0.27	0.11
$AutoDrive\text{-}P^{3}$ (Only SFT)	0.33	0.23	0.01	0.08	0.40	0.17
$AutoDrive\text{-}P^{3}$ (SFT + Only Planning GRPO)	–	–	0.03	0.08	0.24	0.12
$AutoDrive\text{-}P^{3}$ (SFT + $P^{3}\text{-}GRPO$ )	0.64	0.54	0.00	0.02	0.15	0.06

Table 4: Ablation study on

{AutoDrive\text{-}P^{3}}

on nuScenes Benchmark.

Method	Group	History	Sensor	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Method	Size	Traj.	Type	1s	2s	3s	Avg.	1s	2s	3s	Avg.
Ablation 1	4	✔	Video	0.17	0.32	0.65	0.38	0.01	0.06	0.30	0.13
Ablation 2	8	✗	Video	0.17	0.33	0.68	0.39	0.02	0.07	0.33	0.14
Ablation 3	8	✔	Image	0.16	0.32	0.61	0.36	0.01	0.05	0.26	0.12
$P^{3}\text{-}GRPO$	8	✔	Video	0.15	0.30	0.54	0.33	0.00	0.02	0.15	0.06

Table 5: Ablation study on different training setting on nuScenes benchmark.

Figure 5: Dual thinking modes and running time on nuScenes Benchmark.

Figure 6: Visualization. Our model, taking into account the scenarios comprehensively, makes efficient plans that are both reasonable and safe.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	0	1	2	3	4	>4	Average
$P^{3}\text{-}CoT$ (nuScenes)	8187	9926	5121	1081	76	12	0.97
$P^{3}\text{-}CoT$ (NAVSIM)	24675	22915	20172	20794	20793	6085	2.08

Appendix

Table 6: The number of key objects in each frame of

{P^{3}\text{-}CoT}

. The numbers on the table head are the number of key objects in each frame. Average is calculated as total number of key objects divided by total number of frames.

$\bm{P^{3}\text{-}CoT}$ (NAVSIM)	Left	Front	Right
Far	7915	46909	4530
Middle	10601	33908	6558
Near	41435	31219	47815
Behind	9137

Appendix

Table 7: The distribution of prediction actions of

{P^{3}\text{-}CoT}

(NAVSIM). The number of position of key objects in longitudinal and horizontal direction.

Planning Commands	Left	Right	Stop	Straight
$P^{3}\text{-}CoT$ (nuScenes)	1299	1627	4919	16558
$P^{3}\text{-}CoT$ (NAVSIM)	27293	13524	1832	72785

Appendix

Table 8: The distribution of planning commands in

{P^{3}\text{-}CoT}

. The number of planning commands of ego vehicle.

Dataset	Source Dataset	Frames	Perception	Prediction	Planning	Type
nuScenes-QA ( ? )	nuScenes	34149	✔	✗	✗	QA
nuInstruct ( ? )	nuScenes	11850	✔	✔	✔	QA
nuPrompt ( ? )	nuScenes	34149	✔	✗	✗	QA
DriveLM-nuScenes ( ? )	nuScenes	4871	✔	✔	✔	QA
LingoQA ( ? )	LingoQA	28000	—	—	—	QA
DRAMA ( ? )	DRAMA	17785	✔	✗	✔	QA

Appendix

Table 9: The comparisons of existing datasets of scale and structure. Source Dataset: Source of data sampled. Frames: the number of frames labeled by methods. Perception, Prediction, Planning: whether the dataset includes the information about perception, prediction and planning. Type: data organization format.

Figure 9: Training and testing rewards for perception, prediction, planning, and all reward. The results shows the consistent improvement in all rewards, which proves the tight inter-connection among three stages and the effectiveness of

P^{3}\text{-}CoT

Reward weight	Perception $\uparrow$	Prediction $\uparrow$	Planning (Avg. L2) $\downarrow$
1:2:2:5	0.64	0.54	0.33
1:1:1:7	0.63	0.53	0.34

Appendix

Table 10: Weight Setting on nuScenes Benchmark.

	nuScenes		NAVSIM
	Cold-Start	$P^{3}\text{-}GRPO$	Cold-Start	$P^{3}\text{-}GRPO$
Data Setting
Video Shape	[6, 3, 252, 448]	[6, 3, 252, 448]	[4, 3, 168, 672]	[4, 3, 168, 672]
History Traj	6	6	4	4
Future Traj	6	6	8	8
Ego Infos	Only $V$	Only $V$	$a_{x},a_{y},v_{x},v_{y}$	$a_{x},a_{y},v_{x},v_{y}$

Appendix

Table 11: Experimental setup.

Figure 10: Ablation Study on KL Divergence.

Figure 11: Visualization examples of specific samples. Every example is equipped with visualization of results on the top and CoT answer on the bottom.

Figure 12: Visualization nuScenes cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully identifies and localizes the key objects, giving the correct actions. Based on these judgments, our method makes the efficiency planning decision in this sample.

Figure 13: Visualization nuScenes cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully recognizes the driving command and provides the best trajectory instead of conservative one compared with other method.

Figure 14: Visualization NAVSIM cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully predicts the future action of the truck and follows the forward vehicle in a safe distance.

Figure 15: Visualization NAVSIM cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully locates the key object on the left and take a appropriate lane change action to move forward, while other methods provide a wrong trajectories to turn right.

Figure 16: Visualization NAVSIM cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully locates key objects on the left and take a appropriate lane change action to move forward, while other methods drive into an illegal driving area.

Figure 17: Visualization NAVSIM cases with completed questions and answers. ${AutoDrive\text{-}P^{3}}$ successfully recognizes the “Turn Right” command and provide the right planning trajectory, while other methods drive towards the building leading to collisions.

Explore similar work

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving
Jun 29, 2026Chen Yang, Yuhao Wei, Ze Xu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
Chongqing University
NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving
Mar 9, 2026Ximeng Tao, Pardis Taghavi, Dimitar Filev +2VLMs for Autonomous Driving Autonomous Driving Planning
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
Aug 31, 2026Xin Zhou, Zongchuang Zhao, Zhibo Yang +13VLMs for Autonomous Driving Autonomous Driving Perception
Qwen Team, Huazhong University of Science and Technology

Jun 29, 2026J/K move · Enter open · S save

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

Chen Yang, Yuhao Wei, Ze Xu, Ziheng Zou+5

Chongqing University

VLMs for Autonomous Driving End-to-End Autonomous Driving World Model-Based Planning Model-Based Planning World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.

Abstract

Explore similar work

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving