MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

Lingjun Zhang; Yujian Yuan; Changjie Wu; Xinyuan Chang; Xin Cai; Shuang Zeng; Linzhe Shi; Sijin Wang

cs.CVFeb 25, 2026

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

Authors: Lingjun Zhang, Yujian Yuan, Changjie Wu, Xinyuan Chang, Xin Cai, Shuang Zeng, Linzhe Shi, Sijin Wang, +2 more

Organizations: Amap, Alibaba Group · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong · Xi’an Jiaotong University

Abstract

Vision-Language Models (VLM) exhibit strong reasoning capabilities, showing promise for end-to-end autonomous driving systems. Chain-of-Thought (CoT), as VLM's widely used reasoning strategy, is facing critical challenges. Existing textual CoT has a large gap between text semantic space and trajectory physical space. Although the recent approach utilizes future image to replace text as CoT process, it lacks clear planning-oriented objective guidance to generate images with accurate scene evolution. To address these, we innovatively propose MindDriver, a progressive multimodal reasoning framework that enables VLM to imitate human-like progressive thinking for autonomous driving. MindDriver presents semantic understanding, semantic-to-physical space imagination, and physical-space trajectory planning. To achieve aligned reasoning processes in MindDriver, we develop a feedback-guided automatic data annotation pipeline to generate aligned multimodal reasoning training data. Furthermore, we develop a progressive reinforcement fine-tuning method to optimize the alignment through progressive high- level reward-based learning. MindDriver demonstrates superior performance in both nuScences open-loop and Bench2Drive closed-loop evaluation. Codes are available at https://github.com/hotdogcheesewhite/MindDriver.

Figures & tables

Figure 1: Comparison of different reasoning methods. Text reasoning struggles with space misalignment, while image reasoning suffers from guideless image prediction. Our proposed progressive multimodal reasoning conducts aligned smooth reasoning.

Figure 2: Overview. (Left) Framework of MindDriver. MindDriver conducts the perception-imagination-action process for accurate trajectory planning. (Right) (Top) Reasoning data annotation pipeline. The progressive multimodal reasoning data is auto-annotated by both rule-based and model-based filtering and feedback-guided regeneration. (Bottom) Progressive reinforcement fine-tuning is applied to enhance the progressive reasoning process.

Figure 3: Auto-annotation pipeline for progressive multimodal reasoning training data. Qwen2.5-VL-72B first annotates raw CoT, which is then filtered based on format, decision, and logic. Failed cases are re-annotated using error feedback to improve generation quality.

Method	ST-P3 metrics								UniAD metrics								LLM
	L2 (m) $\downarrow$				Collision (%) $\downarrow$				L2 (m) $\downarrow$				Collision (%) $\downarrow$
	1s	2s	3s	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
With Ego status methods
ST-P3* [ECCV22] [ 9 ]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71	-	-	-	-	-	-	-	-		-
VAD* [ICCV23] [ 18 ]	0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33	-	-	-	-	-	-	-	-	-
UniAD* [CVPR23] [ 10 ]	-	-	-	-	-	-	-	-	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37	-

Table 1: End-to-end trajectory planning experiments on nuScenes [ 3 ] . We evaluated the L2 and collision metrics based on the distinct computational methodologies of ST-P3 [ 9 ] and UniAD [ 10 ] , respectively. * indicates that ego status is additionally used. VAD [ 18 ] and UniAD [ 10 ] results are derived from BEV-Planner [ 24 ] . † indicates the re-implemented results using official codes for Qwen2.5-VL-3B.

Method	DriveGAN [ 19 ]	DriveDreamer [ 46 ]	Drive-WM [ 48 ]	GenAD [ 53 ]	GEM [ 6 ]	Doe-1 [ 66 ]	FSDrive [ 59 ]	MindDriver (ours)
Method	[CVPR21]	[ECCV24]	[CVPR24]	[CVPR24]	[CVPR25]	[arxiv24]	[NeruIPS25]	MindDriver (ours)
Type	GAN	Diffusion	Diffusion	Diffusion	Diffusion	Autoregressive	Autoregressive	Autoregressive
Resolution	256 $\times$ 256	128 $\times$ 192	192 $\times$ 384	256 $\times$ 448	576 $\times$ 1024	384 $\times$ 672	128 $\times$ 192	128 $\times$ 192
FID $\downarrow$	73.4	52.6	15.8	15.4	10.5	15.9	10.1	9.4

Table 2: Future frames generation results on the nuScenes [ 3 ] dataset.

Method	DS $\uparrow$	SR (%) $\uparrow$	Effi $\uparrow$	Comf $\uparrow$
AD-MLP* [Arxiv23] [ 60 ]	18.05	0.00	48.45	22.63
UniAD-Base* [CVPR23] [ 11 ]	45.81	16.36	129.21	43.58
VAD* [ICCV25] [ 18 ]	42.35	15.00	157.94	46.01
TCP-traj* [NeurIPS22] [ 50 ]	59.90	30.00	76.54	18.08
DriveAdapter* [CVPR23] [ 15 ]	64.22	33.08	70.22	16.01
ReasonPlan* [CoRL25] [ 27 ]	64.01	34.55	180.64	25.63

Table 3: Closed-loop Results on the Bench2Drive (CARLA) Benchmark. *: the ego status is additionally use.

\ddagger

: trained not on Bench2Drive training set

Type	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Type	1s	2s	3s	Avg.	1s	2s	3s	Avg.
None	0.38	0.93	1.67	0.99	0.13	0.35	1.18	0.56
Text CoT	0.37	0.90	1.61	0.96	0.08	0.23	1.05	0.46
Image CoT	0.39	0.98	1.81	1.06	0.15	0.37	1.13	0.55
MultiModal(I2T) CoT	0.38	0.95	1.70	1.01	0.07	0.25	1.08	0.47
MultiModal(T2I) CoT	0.36	0.89	1.60	0.95	0.05	0.22	0.97	0.41

Table 4: Ablation results of different CoT.

Type	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Type	1s	2s	3s	Avg.	1s	2s	3s	Avg.
Next 1.5s image	0.39	0.95	1.68	1.01	0.08	0.30	0.97	0.45
Next 1s image	0.40	0.96	1.75	1.04	0.03	0.29	0.97	0.43
Next 0.5s image	0.36	0.89	1.60	0.95	0.05	0.22	0.97	0.41

Table 5: Ablation results of dreaming different future scene.

Figure 4: Qualitative comparison of MindDriver with baselines. (Left) Three scenarios from the open-loop nuScenes benchmark. The red trajectory is the prediction and the green one is the GT. (Right) The performance variation with timestamps on closed-loop Bench2Drive.

Strategy	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Strategy	1s	2s	3s	Avg.	1s	2s	3s	Avg.
No CoT	0.38	0.93	1.67	0.99	0.13	0.35	1.18	0.56
Raw CoT	1.32	2.50	3.62	2.48	0.17	1.06	2.97	1.40
+ 3 Filters	0.37	0.92	1.64	0.98	0.08	0.32	1.18	0.53
+ Error feedback	0.37	0.90	1.61	0.96	0.08	0.23	1.05	0.46

Table 6: Ablation results of data filtering.

Type	L2 (m) $\downarrow$				Collision (%) $\downarrow$				FID $\downarrow$
Type	1s	2s	3s	Avg.	1s	2s	3s	Avg.	Score
None RL	0.36	0.89	1.60	0.95	0.05	0.22	0.97	0.41	9.8
One-stage RL	0.37	0.91	1.64	0.97	0.07	0.23	0.95	0.42	9.7
Progressive RL	0.35	0.87	1.57	0.93	0.04	0.19	0.91	0.38	9.4

Table 7: Ablation results of Progressive RFT.

Behavior	Meta Action Type
Direction Change	[Maintain Current Lane, Change Lane Left, Change Lane Right, Turn Left, Turn Right]
Speed Change	[Smooth Deceleration, Emergency Brake, Maintain Current Speed, Smooth Acceleration, Stop, Remain Stationary]

Table 8: Meta action type in desion filter.

Figure 5: Prompt for logical verification to Qwen3-235B-A22B-Instruct.

Figure 6: Combined COT with feedback for re-annotation.

Figure 7: Qualitative Analysis of MindDriver. Red represents our predicted values, and green represents the ground truth (GT).

Error Type	Feedback Content Example
Format Error	Missing Scene Analysis / … Missing Action Decision part.
Decision Error	1.Direction decision error(GT: Change Lane Right; Prediction: Turn Right). 2.Speed decision error(GT: Smooth Deceleration; Prediction: Maintain Current Speed).
Logic Error	Reasoning is discontinuous … consideration is incomplete

Table 9: Specific feedback type in data auto-annotation.

Figure 8: An example of a logical quality check.

Figure 9: Prompt for CoT annotation by Qwen2.5-VL-72B

Figure 10: A complete sample of the annotation dataset.

Model	Layers	Hidden size	Num Heads	Patch Size
Qwen2.5-VL	32	1280	16	14

Table 10: Model parameters of image encoder model (Vision Transformer).

Model	Layers	Hidden size	KV Heads	Head Size
Qwen2.5-VL	36	2048	2	128

Table 11: Model parameters of LLM.

Figure 11: nuScenes results. The red trajectory is the prediction and the green one is the GT.

Figure 12: Bench2drive results. From left to right indicates increasing timestamps.

Explore similar work

$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning
Mar 30, 2026Yuqi Ye, Zijian Zhang, Junhong Lin +3Autonomous Driving Planning VLMs for Autonomous Driving
School of Electronic and Computer Engineering, Peking University
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4
OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models
Apr 20, 2026Yiwei Zhang, Xuesong Chen, Jin Gao +5VLMs for Autonomous Driving End-to-End Autonomous Driving
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2

Mar 30, 2026J/K move · Enter open · S save

$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

Yuqi Ye, Zijian Zhang, Junhong Lin, Shangkun Sun+2

School of Electronic and Computer Engineering, Peking University

Autonomous Driving Planning VLMs for Autonomous Driving RL for VLMs Autonomous Driving Perception

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly being adopted for end-to-end autonomous driving systems due to their exceptional performance in handling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMs directly output planning results without chain-of-thought (CoT) reasoning, bypassing crucial perception and prediction stages which creates a significant domain gap and compromises decision-making capability; 2) Other VLMs can generate outputs for perception, prediction, and planning tasks but employ a fragmented decision-making approach where these modules operate separately, leading to a significant lack of synergy that undermines true planning performance. To address these limitations, we propose

{AutoDrive\text{-}P^3}

, a novel framework that seamlessly integrates

\textbf{P}

erception,

\textbf{P}

rediction, and

\textbf{P}

lanning through structured reasoning. We introduce the

{P^3\text{-}CoT}

dataset to facilitate coherent reasoning and propose

{P^3\text{-}GRPO}

, a hierarchical reinforcement learning algorithm that provides progressive supervision across all three tasks. Specifically,

{AutoDrive\text{-}P^3}

progressively generates CoT reasoning and answers for perception, prediction, and planning, where perception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final planning decisions, enabling safer and more interpretable autonomous driving. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-art performance in planning tasks. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3.

Abstract

Explore similar work

AutoDrive-P3AutoDrive\text{-}P^3AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

$AutoDrive\text{-}P^3$ : Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning