FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving

Mingzhe Guo; Yixiang Yang; Chuanrong Han; Rufeng Zhang; Shirui Li; Ji Wan; Zhipeng Zhang

cs.CVMar 11, 2026

FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving

Authors: Mingzhe Guo, Yixiang Yang, Chuanrong Han, Rufeng Zhang, Shirui Li, Ji Wan, Zhipeng Zhang

Organizations: AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu Inc.

Abstract

Effective environment modeling is the foundation for autonomous driving, underpinning tasks from perception to planning. However, current paradigms often inadequately consider the feedback of ego motion to the observation, which leads to an incomplete understanding of the driving process and consequently limits the planning capability. To address this issue, we introduce a novel ego-scene interactive modeling paradigm. Inspired by human recognition, the paradigm represents ego-scene interaction as the scene flow relative to the ego-vehicle. This conceptualization allows for modeling ego-motion feedback within a feature learning pattern, advantageously utilizing existing log-replay datasets rather than relying on scenario simulations. We specifically propose FlowAD, a general flow-based framework for autonomous driving. Within it, an ego-guided scene partition first constructs basic flow units to quantify scene flow. The ego-vehicle's forward direction and steering velocity directly shape the partition, which reflects ego motion. Then, based on flow units, spatial and temporal flow predictions are performed to model dynamics of scene flow, encompassing both spatial displacement and temporal variation. The final task-aware enhancement exploits learned spatio-temporal flow dynamics to benefit diverse tasks through object and region-level strategies. We also propose a novel Frames before Correct Planning (FCP) metric to assess the scene understanding capability. Experiments in both open and closed-loop evaluations demonstrate FlowAD's generality and effectiveness across perception, end-to-end planning, and VLM analysis. Notably, FlowAD reduces 19% collision rate over SparseDrive with FCP improvements of 1.39 frames (60%) on nuScenes, and achieves an impressive driving score of 51.77 on Bench2Drive, proving the superiority. Code, model, and configurations will be released here.

Figures & tables

Figure 1: (a) Vanilla auto-driving system that performs isolated inference for each timestamp. (b) Temporal auto-driving system that integrates historical observations, yet incompletely captures the feedback of previous ego planning. (c) Our ego-scene interactive system that leverages previous planning to inform future observations, which benefits comprehension of dynamic driving process.

Method	Backbone	Detection		Tracking		Online Map		Motion Prediction		Planning
Method	Backbone	mAP $\uparrow$	NDS $\uparrow$	AMOTA $\uparrow$	AMOTP $\downarrow$	loU-Lane $\uparrow$	loU-Road $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	Avg.L2 $\downarrow$	Avg.Col $\downarrow$
UniAD Hu et al. (2023)	ResNet101	0.380	0.498	0.359	1.320	0.302	0.672	0.71	1.02	1.03	0.31
UniAD w/o.Tem Hu et al. (2023)	ResNet101	0.356 -6%	0.473 -5%	0.303 -16%	1.549 -17%	0.288 -5%	0.648 -4%	0.86 -21%	1.22 -20%	1.08 -5%	0.33 -6%

Table 1: Influence of the temporal fusion in UniAD Hu et al. (2023) on the nuScenes validation set.

Figure 2: Illustration of Ego-guided Scene Partition. The feedback of ego motion is reflected in Starting Point of Partition (forward direction) and Dynamic Adjustment of Partition Size (velocity). Then Multi-level Partition and Local Aggregation divides basic flow units and fuses local messages.

Panel of Figure 3 — Figure 3: (a) The pipeline of Spatial Flow Prediction. (b) The pipeline of Temporal Flow Prediction.

Figure 4: The architecture of our FlowAD. For the input stage, the image features of multi-view videos are extracted with the backbone network. Then, the ego-scene interactive modeling introduces the feedback of the ego-motion and builds the spatio-temporal scene flow feature. Finally, the flow feature serves for the downstream tasks with the task-aware enhancement.

Method	Backbone	Input Size	Dataset	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
StreamPETR Wang et al. (2023)	ResNet50	$256\times 704$	nuScenes	0.432	0.540	0.581	0.272	0.413	0.295	0.195
SparseBEV Liu et al. (2023)	ResNet50	$256\times 704$	nuScenes	0.445	0.553	0.605	0.278	0.389	0.253	0.189
FlowAD (Ours)	ResNet50	$256\times 704$	nuScenes	0.475	0.574	0.585	0.265	0.361	0.236	0.184
BEVDepth Li et al. (2023a)	ResNet101	$512\times 1408$	nuScenes	0.412	0.535	0.565	0.266	0.358	0.331	0.190
SparseBEV Liu et al. (2023)	ResNet101	$512\times 1408$	nuScenes	0.501	0.592	0.562	0.265	0.321	0.243	0.195
FlowAD (Ours)	ResNet101	$512\times 1408$	nuScenes	0.528	0.607	0.552	0.253	0.311	0.230	0.192

Table 2: Object detection results on nuScenes Caesar et al. (2020) and Bench2Drive Jia et al. (2024) . The baseline is SparseBEV.

Method	Backbone	RayIoU	RayIoU 1m,2m,4m			FPS
SimpleOcc Gan et al. (2024)	ResNet101	22.5	17.0	22.7	27.9	9.7
BEVFormer Li et al. (2022)	ResNet101	32.4	26.1	32.9	38.0	3.0
BEVDet-Occ Huang and Huang (2022)	ResNet50	32.6	26.6	33.1	38.2	0.8
FB-Occ Li et al. (2023b)	ResNet50	33.5	26.7	34.1	39.7	10.3
SparseOcc Liu et al. (2024)	ResNet50	35.7	29.3	36.5	41.4	17.3
FlowAD (Ours)	ResNet50	38.4	32.2	39.3	43.8	14.6

Table 3: Comparison of 3D occupancy prediction on the Occ3D-nuScenes Tian et al. (2023) dataset. The baseline is SparseOcc Liu et al. (2024) .

Method	Backbone	Detection		Tracking		Online Map	Motion Prediction		Planning			FPS $\uparrow$
Method	Backbone	mAP $\uparrow$	NDS $\uparrow$	AMOTA $\uparrow$	AMOTP $\downarrow$	mAP $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	Avg.L2 $\downarrow$	Avg.Col $\downarrow$	FCP $\downarrow$	FPS $\uparrow$
VAD Jiang et al. (2023)	ResNet50	-	-	-	-	0.476	-	-	0.72	0.21	3.07	5.2
MomAD Song et al. (2025)	ResNet50	0.423	0.531	0.391	1.243	0.559	0.61	0.98	0.60	0.09	-	7.8
DriveWM Wang et al. (2024b)	-	-	-	-	-	-	-	-	0.80	0.26	-	-
LAW Li et al. (2024)	-	-	-	-	-	-	-	-	0.61	0.30	-	-
UncAD Yang et al. (2025)	ResNet50	-	-	-	-	-	-	-	0.60	0.07	-	-

Table 4: Results of perception, motion prediction and planning on the nuScenes Caesar et al. (2020) validation set. The FPS is measured on an RTX3090 GPU. The baselines are SparseDrive Sun et al. (2024) and DiffusionDrive Liao et al. (2025) . ‡ denotes using ego status in the planning module.

Method	Backbone	Open-loop	Closed-loop				Closed-loop Ability
Method	Backbone	Avg. L2 $\downarrow$	DS $\uparrow$	SR $\uparrow$	Effi $\uparrow$	Smooth $\uparrow$	Merging $\uparrow$	Overtaking $\uparrow$	Emergency Brake $\uparrow$	Give Way $\uparrow$	Traffic Sign $\uparrow$	Mean $\uparrow$
UniAD Hu et al. (2023)	ResNet101	0.73	45.81	16.36	129.21	43.58	14.10	17.78	21.67	10.00	14.21	15.55
VAD Jiang et al. (2023)	ResNet50	0.91	42.35	15.00	157.94	46.01	8.11	24.44	18.64	20.00	19.15	18.07
SparseDrive Sun et al. (2024)	ResNet50	0.83	44.54	16.71	170.21	48.63	12.50	17.50	20.00	20.00	23.03	18.60
MomAD Song et al. (2025)	ResNet50	0.82	47.91	18.11	174.91	51.20	-	-	-	-	-	-
FlowAD (Ours)	ResNet50	0.73	51.77	22.02	178.42	53.89	20.00	28.67	25.00	25.00	28.42	25.42

Table 5: Planning results on the Bench2Drive dataset Jia et al. (2024) . The baseline method is SparseDrive Sun et al. (2024) .

Method	Planning Accuracy $\uparrow$	Path (F1 Score)			Speed (F1 Score)				FPS $\uparrow$
Method	Planning Accuracy $\uparrow$	Go Straight $\uparrow$	Turn Left $\uparrow$	Turn Right $\uparrow$	Keep $\uparrow$	Accelerate $\uparrow$	Decelerate $\uparrow$	Stop $\uparrow$	FPS $\uparrow$
Senna Jiang et al. (2024)	18.67	67.81	6.99	0.00	78.11	0.00	2.12	31.71	0.43
Senna ∗ Jiang et al. (2024)	88.54	96.85	30.53	46.94	96.50	0.00	0.00	90.82	0.43
FlowAD (Ours)	90.99	97.76	60.71	68.17	96.97	51.53	40.38	93.55	0.38

Table 6: Results of VLM high-level planning on the nuScenes Caesar et al. (2020) validation set. The FPS is measured on an NVIDIA RTX3090 GPU. The baseline is Senna Jiang et al. (2024) . ∗ denotes finetuned on nuScenes dataset.

#	Ego-guided Scene Partition				Flow Prediction		Detection	Planning		VLM Analysis	FPS $\uparrow$
#	Start	Multi	Aggre.	Adjust	Spatial	Temporal	mAP $\uparrow$	L2@3s $\downarrow$	FCP $\downarrow$	Planning Acc $\uparrow$	FPS $\uparrow$
①							0.445	0.96	2.55	88.54	21.7
②					✓		0.454	0.93	2.23	89.13	20.9
③					✓	✓	0.459	0.91	1.87	89.66	20.3
④	✓				✓	✓	0.463	0.88	1.31	90.22	20.2
⑤	✓	✓			✓	✓	0.466	0.87	1.16	90.30	19.4

Table 7: Ablation on the proposed modules in FlowAD. The FPS is measured on the detection baseline SparseBEV Liu et al. (2023) . “@3s” denotes the planning metric at 3s.

Figure 5: Qualitative results of multi-view object detection on nuScenes Caesar et al. (2020) . The baseline is SparseBEV Liu et al. (2023) .

Figure 6: Qualitative results of end-to-end planning and VLM analysis on nuScenes Caesar et al. (2020) .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of the scene flow caused by the relative motion of the ego vehicle.

#	Multi-level Partition Sizes				mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
#	Stage1	Stage2	Stage3	Stage4	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
①	0	0	0	1	0.462	0.562	0.600	0.272	0.382	0.240	0.198	21.1
②	8	4	2	1	0.475	0.574	0.585	0.265	0.361	0.236	0.184	18.9
③	16	8	4	2	0.470	0.570	0.594	0.268	0.363	0.244	0.197	19.3
④	24	12	6	3	0.459	0.562	0.601	0.271	0.380	0.243	0.199	19.8

Appendix

Table 8: Ablation on the base partition size

P

on the nuScenes Caesar et al. (2020) validation set.

Method	Backbone	Detection		Tracking		Online Map	Motion Prediction		Planning			FPS $\uparrow$
Method	Backbone	mAP $\uparrow$	NDS $\uparrow$	AMOTA $\uparrow$	AMOTP $\downarrow$	mAP $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	Avg.L2 $\downarrow$	Avg.Col $\downarrow$	FCP $\downarrow$	FPS $\uparrow$
SparseDrive Sun et al. (2024)	ResNet50	0.418	0.525	0.386	1.254	0.551	0.62	0.99	0.61	0.08	2.55	9.0
FlowAD (Ours)	ResNet50	0.448	0.555	0.427	1.181	0.582	0.58	0.95	0.56	0.06	1.03	7.6
FlowAD ∗ (Ours)	ResNet50	0.434	0.544	0.416	1.198	0.567	0.59	0.96	0.57	0.06	1.45	8.5
FlowAD △ (Ours)	ResNet50	0.429	0.533	0.401	1.207	0.559	0.60	1.01	0.59	0.07	2.03	8.5

Appendix

Table 9: Results of perception, motion prediction and planning on the nuScenes Caesar et al. (2020) validation set. The baseline method is SparseDrive Sun et al. (2024) . ∗ denotes using the last FPN feature for multi-level partition and spatial-temporal flow prediction. △ indicates fusing the transition state and forecasted in the flow prediction module.

#	Power Coefficient	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
①	Linear	0.469	0.569	0.595	0.271	0.363	0.240	0.190
②	Quadratic	0.475	0.574	0.585	0.265	0.361	0.236	0.184
③	Cubic	0.474	0.572	0.589	0.256	0.354	0.237	0.187

Appendix

Table 10: Ablation on the power coefficient in the dynamic adjustment of partition size on the nuScenes Caesar et al. (2020) validation set.

#	Range of Local Aggregation	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
①	1	0.460	0.563	0.605	0.270	0.365	0.239	0.200	19.2
②	3	0.475	0.574	0.585	0.265	0.361	0.236	0.184	18.9
③	5	0.471	0.575	0.574	0.269	0.350	0.238	0.196	18.8
④	7	0.464	0.567	0.588	0.271	0.366	0.242	0.194	18.6

Appendix

Table 11: Ablation on the range of local aggregation on the nuScenes Caesar et al. (2020) validation set.

Method	Aggregation	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
SparseBEV Liu et al. (2023)		0.445	0.553	0.605	0.278	0.389	0.253	0.189	21.7
FlowAD (Ours)	Local	0.475	0.574	0.585	0.265	0.361	0.236	0.184	18.9
FlowAD (Ours)	Deformable	0.480	0.580	0.582	0.273	0.367	0.241	0.185	16.7

Appendix

Table 12: Ablation of aggregation strategies on the nuScenes Caesar et al. (2020) validation set.

#	Levels for Flow Prediction				mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
#	Stage1	Stage2	Stage3	Stage4	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$	FPS $\uparrow$
①					0.445	0.553	0.605	0.278	0.389	0.253	0.189	21.7
②	✓				0.460	0.560	0.599	0.272	0.381	0.247	0.190	19.8
③				✓	0.466	0.564	0.594	0.269	0.378	0.244	0.192	20.1
④			✓	✓	0.475	0.574	0.585	0.265	0.361	0.236	0.184	18.9
⑤		✓	✓	✓	0.479	0.580	0.581	0.266	0.357	0.232	0.183	17.2

Appendix

Table 13: Ablation on the levels for flow prediction on nuScenes Caesar et al. (2020) validation set.

#	Flow Prediction Loss		mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
#	Spatial	Temporal	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
①			0.444	0.548	0.619	0.280	0.392	0.256	0.192
②	✓		0.459	0.562	0.596	0.273	0.379	0.244	0.187
③		✓	0.456	0.559	0.601	0.276	0.381	0.246	0.189
④	✓	✓	0.475	0.574	0.585	0.265	0.361	0.236	0.184

Appendix

Table 14: Ablation on the flow prediction losses on the nuScenes Caesar et al. (2020) validation set.

#	Method	Train Epoch	Flow Prediction Loss		mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
#	Method	Train Epoch	Spatial	Temporal	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
①	SparseBEV Liu et al. (2023)	18			0.437	0.541	0.625	0.283	0.401	0.255	0.191
②	SparseBEV Liu et al. (2023)	24			0.445	0.553	0.605	0.278	0.389	0.253	0.189
③	SparseBEV Liu et al. (2023)	30			0.448	0.554	0.599	0.273	0.389	0.256	0.193
④	FlowAD (Ours)	18			0.393	0.499	0.671	0.294	0.404	0.275	0.198
⑤	FlowAD (Ours)	24			0.444	0.548	0.619	0.280	0.392	0.256	0.192

Appendix

Table 15: Influence of the flow prediction losses on the nuScenes Caesar et al. (2020) validation set. The default train epoch is 24, following the baseline SparseBEV Liu et al. (2023) .

Method	Backbone	L2 Error (m) $\downarrow$				Collision Rate (%) $\downarrow$				FCP (frame) $\downarrow$				FCP ♢ (frame) $\downarrow$				FPS $\uparrow$
Method	Backbone	1s	2s	3s	Avg.	1s	2s	3s	Avg.	0.25m	0.50m	0.75m	Avg.	1 frame	2 frame	3 frame	Avg.	FPS $\uparrow$
UniAD Hu et al. (2023)	ResNet101	0.44	0.67	0.96	0.69	0.04	0.08	0.23	0.12	5.41	2.69	0.93	3.01	0.73	2.02	3.43	2.06	1.8
VAD Jiang et al. (2023)	ResNet50	0.41	0.70	1.05	0.72	0.07	0.17	0.41	0.22	6.44	3.07	1.42	3.64	0.87	2.39	4.12	2.46	5.2
MomAD Song et al. (2025)	ResNet50	0.31	0.57	0.91	0.60	0.01	0.05	0.22	0.09	-	-	-	-	-	-	-	-	7.8
SparseDrive Sun et al. (2024)	ResNet50	0.29	0.58	0.96	0.61	0.01	0.05	0.18	0.08	4.87	2.55	1.09	2.84	0.54	1.88	3.02	1.48	9.0
FlowAD (Ours)	ResNet50	0.28	0.55	0.84	0.56	0.01	0.03	0.15	0.06	2.12	1.03	0.48	1.21	0.37	0.90	1.79	1.02	7.6

Appendix

Table 16: Planning results in terms of L2 error, collision rate, and the proposed FCP metrics with different thresholds on the nuScenes Caesar et al. (2020) validation set. The baseline method is SparseDrive Sun et al. (2024) . ♢ denotes the extended FCP metric.

#	Scale of Training Data	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	mAAE $\downarrow$
①	25%	0.391	0.497	0.691	0.275	0.519	0.297	0.200
②	50%	0.448	0.554	0.593	0.269	0.389	0.257	0.194
③	75%	0.466	0.569	0.587	0.266	0.367	0.244	0.188
④	100%	0.475	0.574	0.585	0.265	0.361	0.236	0.184

Appendix

Table 17: Ablation on the scale of training data on the nuScenes Caesar et al. (2020) validation set.

Method	Backbone	Perf. go straight $\downarrow$ (5309 samples)		Perf. turn left $\downarrow$ (301 samples)		Perf. turn right $\downarrow$ (409 samples)		Perf. Overall $\downarrow$ (6019 samples)
Method	Backbone	Avg. L2 (m)	Avg. Col. (%)	Avg. L2 (m)	Avg. Col. (%)	Avg. L2 (m)	Avg. Col. (%)	Avg. L2 (m)	Avg. Col. (%)
SparseDrive Sun et al. (2024)	ResNet50	0.58	0.08	0.83	0.29	0.86	0.08	0.61	0.09
FlowAD SparseDrive	ResNet50	0.53	0.06	0.77	0.08	0.69	0.09	0.56	0.06
SparseDrive Sun et al. (2024)	ResNet101	0.56	0.06	0.89	0.13	0.81	0.10	0.58	0.06
FlowAD SparseDrive	ResNet101	0.50	0.05	0.74	0.08	0.65	0.08	0.52	0.05
DiffusionDrive Liao et al. (2025)	ResNet50	0.55	0.07	0.64	0.12	0.59	0.10	0.57	0.08

Appendix

Table 18: Performance comparison under different driving scenarios on the nuScenes Caesar et al. (2020) validation set.

Method	Planning Accuracy (5119 samples)	Keep Right Turn (187 samples)	Keep Left Turn (139 samples)	Keep Straight (3704 samples)	Accelerate Right Turn (3 samples)	Accelerate Left Turn (4 samples)	Accelerate Straight (62 samples)	Decelerate Straight (45 samples)	Stop Straight (975 samples)	FPS
Senna Jiang et al. (2024)	18.67	0.00	18.03	37.53	0.00	0.00	0.00	2.22	26.56	0.43
Senna ∗ Jiang et al. (2024)	88.54	35.98	28.78	92.28	0.00	0.00	0.00	6.67	88.13	0.43
FlowAD (Ours)	90.99	69.52	69.06	94.71	66.67	25.00	43.55	17.78	90.87	0.39

Appendix

Table 19: Accuracy of high-level planning on the nuScenes Caesar et al. (2020) validation set. The baseline is Senna Jiang et al. (2024) . ∗ denotes finetuned on the nuScenes dataset.

Method	GaussianBlur Noise	Calibration Error	Timestamp Drift	Detection		Tracking		Online Map	Motion Prediction		Planning
Method	GaussianBlur Noise	Calibration Error	Timestamp Drift	mAP $\uparrow$	NDS $\uparrow$	AMOTA $\uparrow$	AMOTP $\downarrow$	mAP $\uparrow$	minADE $\downarrow$	minFDE $\downarrow$	Avg.L2 $\downarrow$	Avg.Col $\downarrow$	FCP $\downarrow$
SparseDrive Sun et al. (2024)				0.496	0.588	0.501	1.085	0.562	0.60	0.96	0.58	0.06	2.30
FlowAD (Ours)				0.523	0.605	0.518	1.040	0.595	0.56	0.93	0.52	0.05	0.91
SparseDrive Sun et al. (2024)	✓			0.472	0.555	0.487	1.159	0.533	0.63	1.03	0.60	0.07	2.34
FlowAD (Ours)	✓			0.517	0.596	0.514	1.053	0.583	0.57	0.96	0.53	0.05	0.96
SparseDrive Sun et al. (2024)		✓		0.438	0.540	0.466	1.211	0.509	0.64	1.07	0.65	0.08	3.22

Appendix

Table 20: Robustness analysis on the nuScenes Caesar et al. (2020) validation set. The baseline method is SparseDrive Sun et al. (2024) with ResNet101.

Figure 8: Qualitative results of multi-view object detection on nuScenes Caesar et al. (2020) . Results show the strong capability of our method to handle occluded objects with scene flow dynamics.

Panel of Figure 9 — Figure 9: Qualitative results of end-to-end planning and VLM analysis on the nuScenes Caesar et al. (2020) dataset. Our method performs more robust planning under different driving scenarios with enhanced comprehension by scene flow dynamics.

Figure 10: Visualization of the planning results on the Bench2Drive dataset Jia et al. (2024) with CARLA simulator.

Figure 11: Visualization of a challenging cutting-in case. Observing the front vehicle with an abrupt lane change (marked by the yellow arrow), our planner captures the spatio-temporal dynamics and performs the brake, proving the capability to handle complex driving interactions.

Figure 12: Visualization of planning stability under challenging longitudinal dynamics. Under the complex scenarios including hard braking (Left, $-2.54m/s^{2}$ ) and rapid acceleration (Right, $+2.65m/s^{2}$ ), our method is capable of making robust planning compared with other methods.

Figure 13: Visualization of planning robustness in a steering case. Our method could plan a more reasonable ego trajectory under the safety-critical turning scenario, proving the effectiveness.

Figure 14: Visualization of activation maps. The proposed flow prediction helps to concentrate on the critical dynamic objects and suppress the background clutter.

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

Mar 20, 2026

DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving

Recently, world models have been incorporated into the autonomous driving systems to improve the planning reliability. Existing approaches typically predict future states through appearance generation or deterministic regression, which limits their ability to capture trajectory-conditioned scene evolution and leads to unreliable action planning. To address this, we propose DynFlowDrive, a latent world model that leverages flow-based dynamics to model the transition of world states under different driving actions. By adopting the rectifiedflow formulation, the model learns a velocity field that describes how the scene state changes under different driving actions, enabling progressive prediction of future latent states. Building upon this, we further introduce a stability-aware multi-mode trajectory selection strategy that evaluates candidate trajectories according to the stability of the induced scene transitions. Extensive experiments on the nuScenes and NavSim benchmarks demonstrate consistent improvements across diverse driving frameworks without introducing additional inference overhead. Source code will be abaliable at https://github.com/xiaolul2/DynFlowDrive.

Xiaolu Liu, Yicong Li, Song Wang +3

Zhejiang University · National University of Singapore

World Models for Autonomous Driving Latent World Models Autonomous Driving Planning Flow-Based Generative Modeling

Ask PDF

Apr 28, 2026

ProDrive: Proactive Planning for Autonomous Driving via Ego-Environment Co-Evolution

End-to-end autonomous driving planners typically generate trajectories from current observations alone. However, real-world driving is highly dynamic, and such reactive planning cannot anticipate future scene evolution, often leading to myopic decisions and safety-critical failures. We propose ProDrive, a world-model-based proactive planning framework that enables ego-environment co-evolution for autonomous driving. ProDrive jointly trains a query-centric trajectory planner and a bird's-eye-view (BEV) world model end-to-end: the planner generates diverse candidate trajectories and planning-aware ego tokens, while the world model predicts future scene evolution conditioned on them. By injecting planner features into the world model and evaluating all candidates in parallel, ProDrive preserves end-to-end gradient flow and allows future outcome assessment to directly shape planning. This bidirectional coupling enables proactive planning beyond current-observation-driven decision-making. Experiments on NAVSIM v1 show that ProDrive outperforms strong baselines in both safety and planning efficiency, while ablations validate the effectiveness of the proposed ego-environment coupling design.

Chuyao Fu, Shengzhe Gan, Zhuoli Ouyang +5

1Southern University of Science and Technology · 2Hong Kong University of Science and Technology

End-to-End Autonomous Driving World Models World Models for Autonomous Driving Autonomous Driving Planning

Ask PDF

Jun 29, 2026

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.

Chen Yang, Yuhao Wei, Ze Xu +6

Chongqing University

VLMs for Autonomous Driving End-to-End Autonomous Driving World Model-Based Planning Model-Based Planning World Models for Autonomous Driving Autonomous Driving Planning

Ask PDF

Abstract

Autonomous Driving

Explore similar work

DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving

ProDrive: Proactive Planning for Autonomous Driving via Ego-Environment Co-Evolution

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving