Planning-Aligned Pretraining of BEV Representations with Sparse Action-Conditioned Targets for End-to-End Autonomous Driving

Jaeha Song; Soonmin Hwang

Figure 1: PAVER improves planning L2 on VAD-Tiny and VAD-Base with about 36% less estimated total training time, including pretraining (Table 9 ). The lower panels compare planning performance and detection NDS across pretraining methods and auxiliary-head sizes.

Method	PT	FT	Pretraining	Fine-tuning	L2 (m) $\downarrow$	Col. (%) $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	MR $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$
	Epochs		Auxiliary Tasks		Planning		Motion			Detection		Map
VAD-Tiny †	0	60	None	Det., Map, Mot., Plan.	0.66	0.51	0.91	1.25	0.13	0.23	0.34	0.42
+ Ours	20	30	Actions	Det., Map, Mot., Plan.	0.60	0.19	0.80	1.09	0.12	0.28	0.40	0.44
VAD-Base †	0	60	None	Det., Map, Mot., Plan.	0.74	0.31	0.76	1.03	0.11	0.29	0.42	0.50
+ Ours	20	30	Actions	Det., Map, Mot., Plan.	0.56	0.40	0.69	0.90	0.09	0.33	0.45	0.50
GenAD †	0	60	None	Det., Map, Mot., Plan.	0.59	0.37	0.87	1.18	0.14	0.19	0.26	0.46

Table 1: Multi-task performance on the nuScenes validation set. Actions denotes PAVER’s sparse action targets. PT and FT denote pretraining and fine-tuning epochs; Det., Mot., and Plan. denote detection, motion prediction, and planning. L2 is the trajectory displacement error and Col. the box collision rate in percent, both averaged over 1, 2, and 3 seconds; ADE, FDE, and MR are the motion average and final displacement errors and miss rate; mAP and NDS are mean average precision and the nuScenes detection score.

Figure 2: Driving pretraining paradigms. Existing work supervises pretraining with (a) perception labels or (b) dense reconstruction; (c) PAVER introduces sparse action targets.

Figure 3: Two-stage training protocol. Only the pretrained BEV encoder is transferred, and the existing end-to-end model is fine-tuned unchanged.

Figure 4: PAVER overview. PAVER constructs sparse action targets for risk and unknown evidence, masks their associated BEV regions, and predicts the targets from the remaining context. The PAVER head is removed before downstream training.

Figure 7: VAD-Tiny qualitative comparisons on nuScenes. More results are in the Technical Supplement.

Pretraining					Plan.		Mot.	Det.	Map
Det.	Map	Occ. †	Act.	#Params $\downarrow$	L2 (m) $\downarrow$	Col. (%) $\downarrow$	ADE $\downarrow$	NDS $\uparrow$	mAP $\uparrow$
				0.00M	0.662	0.513	0.905	0.338	0.419
$\checkmark$				3.01M	0.515	0.240	0.798	0.365	0.406
	$\checkmark$			2.91M	0.694	0.333	0.857	0.331	0.391
		$\checkmark$		3.56M	0.646	0.320	0.905	0.336	0.393
$\checkmark$	$\checkmark$			5.92M	0.504	0.190	0.805	0.361	0.397

Table 7: Pretraining strategy ablation. All pretraining runs use 20 epochs. L2 and collision rate are averaged over 1, 2, and 3 seconds. #Params denotes the number of auxiliary parameters. Occ. and Act. denote occupancy and PAVER’s action conditioning. Bold and underline denote the best and second-best results, respectively, among the pretraining runs.

		Planning		Motion			Detection	Map
Mask	Action	L2 (m) $\downarrow$	Col. (%) $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	MR $\downarrow$	NDS $\uparrow$	mAP $\uparrow$
		0.662	0.513	0.905	1.250	0.135	0.338	0.419
$\checkmark$		0.655	0.523	0.808	1.084	0.115	0.396	0.443
	$\checkmark$	0.705	0.550	0.797	1.080	0.123	0.388	0.445
$\checkmark$	$\checkmark$	0.603	0.187	0.803	1.086	0.121	0.397	0.439

Table 8: Component ablation. Mask denotes action-corridor feature masking with a shared learnable token, and Action denotes conditioning the prediction head on the ego-frame action state

[x,y,\psi,v]

, where

\psi

is the yaw heading. The variant using both is PAVER; the empty row is the VAD-Tiny baseline.

Figure 8: Grad-CAM for the BEV target on nuScenes. With PAVER initialization, activations highlight scene objects, including vehicles and pedestrians. More results are provided in the Technical Supplement.

Method	PT	FT	Time	L2 (m) $\downarrow$	Col. (%) $\downarrow$	ADE $\downarrow$	NDS $\uparrow$	mAP $\uparrow$
	Training Cost			Planning		Motion	Detection	Map
VAD-Tiny †	48 ‡	12	20.5h	0.701	0.730	0.840	0.363	0.413
VAD-Tiny †	0	60	21.3h	0.662	0.513	0.905	0.338	0.419
+ Ours	20	10	6.5h	0.637	0.300	0.851	0.357	0.361
+ Ours	20	20	10.1h	0.647	0.220	0.781	0.387	0.425
+ Ours	20	30	13.6h	0.603	0.187	0.803	0.397	0.439

Table 9: Training schedule ablation. Total training time, including pretraining, is estimated from mean iteration times on 4 NVIDIA RTX 5090 GPUs. Bold marks the best result and underline marks the second-best result.

Item	Configuration
Training split	28,130 nuScenes keyframes
Validation split	6,019 nuScenes keyframes
Camera input	6 synchronized surround views
Image backbone and neck	ResNet-50 and FPN
BEV representation	$[B,256,100,100]$
BEV range	$x\in[-15,15)$ m, $y\in[-30,30)$ m

Table 10: PAVER data and model configuration.

Learned component	10K	30K	90K
Mask token	256	256	256
Projector	4,112	8,224	73,760
First readout layer	5,376	18,944	18,944
Output layer	514	1,026	1,026
Total	10,258	28,450	93,986

Table 11: Auxiliary parameter counts for the PAVER capacity variants. The 10K and 30K projectors are pointwise; the 90K projector uses a

3\times 3

kernel.

Setting	Value
Pretraining duration	20 epochs
Pretraining batch	4 GPUs, 4 samples per GPU
Optimizer	AdamW ( Loshchilov and Hutter 2019 )
Learning rate	$5\times 10^{-5}$
Image-backbone multiplier	0.2
Weight decay	0.01

Table 12: Optimization settings for the reported PAVER experiments.

Input	Risk BCE	$\Delta$	Unknown BCE	$\Delta$
Correct input	0.3366	0.0000	0.2333	0.0000
Cross-scene BEV	0.3571	0.0204	0.2723	0.0390
Zero context	0.3511	0.0145	0.2784	0.0451
Spatial permutation	0.3691	0.0325	0.3254	0.0920
Action permutation	0.3879	0.0513	0.2688	0.0355

Table 13: Frozen-input intervention results. Lower is better;

\Delta

is measured from the correct-input result and computed before rounding the displayed losses.

	Cross-scene	Zero ctx.	Spatial perm.	Action perm.
$\Delta$ Risk [95% CI]	$0.0204\,[0.0150,\,0.0241]$	$0.0145\,[0.0099,\,0.0176]$	$0.0325\,[0.0292,\,0.0358]$	$0.0513\,[0.0500,\,0.0535]$
$\Delta$ Unknown [95% CI]	$0.0390\,[0.0288,\,0.0464]$	$0.0451\,[0.0390,\,0.0512]$	$0.0920\,[0.0835,\,0.1002]$	$0.0355\,[0.0329,\,0.0378]$

Table 14: Scene-bootstrap uncertainty of the frozen-input interventions.

Figure 9: Action–time shortcut audit. The frozen BEV predictor is compared with a prior conditioned on control family, horizon, and perturbed action state.

Region	Init.	PAVER	$\Delta$	95% CI
Rear	0.6532	0.7399	+0.0868	[0.0598, 0.1131]
Lateral	0.6455	0.7072	+0.0617	[0.0250, 0.0956]
Front off-path	0.6716	0.8205	+0.1488	[0.1032, 0.1916]

Table 15: Frozen object–background probe AUROC outside direct action support. Differences are computed before rounding the displayed AUROC values.

Figure 10: Frozen evidence probes by BEV region. PAVER pretraining increases linear accessibility across regions; downstream differences are smaller and mixed.

Figure 11: Native per-checkpoint evaluator trajectories for 13 VAD-Tiny experiments. Missing epochs are not interpolated; PAVER (10K) is emphasized.

Figure 12: Parameter dynamics during PAVER pretraining and downstream training. The panels show the learned mask-token trajectory, changes from the state before the first pretraining batch to pretraining epoch 20, and subsequent changes at downstream epoch 30. Registered buffers are excluded.

Figure 13: Parameter-only changes from PAVER pretraining to downstream epoch 30 for six selected parameter groups. Left: L2 distance divided by the pretrained parameter norm. Right: cosine similarity to the pretrained parameters. Both panels share the centered legend; epoch 0 is the pretrained reference. Image-neck changes are reported in the text.

	L2 (m) $\downarrow$
	All	Worst 10%	Worst 5%	Worst 1%
VAD-Tiny †	0.66	2.22	3.09	6.98
+ Ours	0.60	1.71	2.15	3.51
VAD-Base	0.74	2.85	4.28	10.56
+ Ours	0.56	2.08	3.05	7.01
GenAD †	0.59	2.10	2.98	6.16

Table 16: Planning results by error percentile on nuScenes. Samples are ranked by their own average L2; each column reports the mean over that fraction of the 5,119 valid planning samples.

Pretraining target	Det. mAP	NDS	Map mAP	Occ. IoU	Occ. mIoU
Detection	0.1208	0.1907	–	–	–
Map	–	–	0.0040	–	–
Occupancy	–	–	–	0.6011	0.2045
Det. + Map + Occ.	0.1314	0.1771	0.0120	0.5579	0.1907

Table 17: Auxiliary-head quality after 20 pretraining epochs. Det., Map, and Occ. denote pretraining on detection, on the vector map, and on occupancy, so Det. + Map + Occ. is the joint objective. Each column uses its task-native evaluator.

AbsRel $\downarrow$	RMSE $\downarrow$	SILog $\downarrow$	$\delta_{1}$ $\uparrow$	$\delta_{2}$ $\uparrow$	$\delta_{3}$ $\uparrow$
0.1286	5.5592	21.7424	0.9238	0.9593	0.9739

Table 18: Pseudo-LiDAR depth quality on a 1,000-frame training audit. RMSE is in meters. The 95% intervals are

[0.1268,0.1305]

for AbsRel and

[0.9225,0.9250]

for

\delta_{1}

.

Model	Method / Pretraining	Divider	Ped. Crossing	Boundary	mAP
VAD-Tiny	Scratch	0.476	0.317	0.463	0.419
VAD-Tiny	Staged	0.445	0.352	0.443	0.413
VAD-Tiny	PAVER (10K)	0.483	0.339	0.494	0.439
VAD-Tiny	PAVER (30K)	0.480	0.383	0.479	0.447
VAD-Tiny	PAVER (90K)	0.469	0.329	0.501	0.433
VAD-Tiny	w/o Mask	0.479	0.372	0.484	0.445

Table 20: Vector map prediction results, aggregate and class-wise.

		Planning		Motion				Detection		Map
Model	Method / Pretraining	L2 $\downarrow$	Col. (%) $\downarrow$	EPA $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	MR $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$
VAD-Tiny	Scratch	0.662	0.513	0.458	0.905	1.250	0.135	0.231	0.338	0.419
VAD-Tiny	Staged	0.701	0.730	0.455	0.840	1.151	0.122	0.232	0.363	0.413
VAD-Tiny	PAVER (10K)	0.603	0.187	0.498	0.803	1.086	0.121	0.278	0.397	0.439
VAD-Tiny	PAVER (30K)	0.514	0.320	0.492	0.800	1.090	0.125	0.256	0.383	0.447
VAD-Tiny	PAVER (90K)	0.596	0.220	0.509	0.782	1.056	0.120	0.280	0.397	0.433

Table 19: Planning, motion, detection, and map results on nuScenes. Planning values are 3-horizon averages, collision is in percent, and motion uses the 2.0 m matching threshold.

Figure 14: Auxiliary capacity against every native metric. PAVER, the task-supervised controls, and the reported UniPAD and MIM4D baselines are shown against the VAD-Tiny scratch reference.

Model	Method / Pretraining	mAP	NDS	Car	Truck	C.Veh.	Bus	Trail.	Barr.	Motor.	Bicy.	Ped.	Cone
VAD-Tiny	Scratch	0.231	0.338	0.492	0.189	0.060	0.283	0.005	0.338	0.126	0.171	0.275	0.367
VAD-Tiny	Staged	0.232	0.363	0.485	0.168	0.059	0.338	0.004	0.336	0.126	0.162	0.299	0.345
VAD-Tiny	PAVER (10K)	0.278	0.397	0.509	0.237	0.109	0.414	0.010	0.323	0.217	0.227	0.346	0.391
VAD-Tiny	PAVER (30K)	0.256	0.383	0.514	0.245	0.060	0.335	0.009	0.298	0.213	0.215	0.334	0.335
VAD-Tiny	PAVER (90K)	0.280	0.397	0.527	0.220	0.099	0.369	0.013	0.318	0.258	0.237	0.335	0.419
VAD-Tiny	w/o Mask	0.276	0.388	0.506	0.217	0.081	0.394	0.012	0.316	0.236	0.268	0.344	0.388

Table 21: 3D object detection results, aggregate and class-wise. C.Veh., Trail., Barr., Motor., Bicy., Ped., and Cone denote construction vehicle, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone. A dash marks a class the evaluator did not report.

Route	Scenario	Policy	Outcome	RC $\uparrow$	DS $\uparrow$	Collisions $\downarrow$
24759	Parking cut in	UniAD-Tiny	Blocked	25.42	25.42	0
		+ PAVER	Blocked	66.91	52.43	0
		GenAD + PAVER	Completed †	100.00	100.00	0
25318	Parked obstacle	UniAD-Tiny	Timeout	35.69	21.41	1
		+ PAVER	Completed	100.00	36.00	2
		GenAD + PAVER	Timeout †	31.86	31.86	0

Table 22: Bench2Drive Town05 Long, per-route outcome audit. One repetition per route; RC is route completion, DS the composed driving score. Timeout marks an exhausted 200 s simulated-time budget. † scored without a completion trace, so it cannot be matched on route completion.

Figure 30

Figure 16: Bench2Drive Town05 Long, route 25857: accident two ways. Rows are matched on route completion, so both policies are shown at the same physical point; each camera image is paired with the simulator’s top-down view. UniAD-Tiny exhausts the 200 s budget at 32.30 route completion, while the same architecture initialized with PAVER completes the route (Table 22 ).

Figure 17: Bench2Drive Town05 Long, route 25318: parked obstacle at night. UniAD-Tiny exhausts the time budget behind the obstacle; with PAVER initialization the same architecture passes it and reaches the end of the route.

Figure 18: Bench2Drive Town05 Long, route 24759: parking cut in. Both policies are blocked by the cut-in vehicle, but the pretrained model travels more than twice as far before it stops (Table 22 ).

Figure 19: Bench2Drive Town05 Long, route 24759: parking cut in, for GenAD initialized with PAVER. The camera image is paired with the simulator’s top-down view, on which the planned path is drawn as a band running from its near end to its far end. This run has no completion trace, so the columns are spaced by elapsed time rather than by route completion, and the campaign contains no scratch GenAD counterpart, so the figure reports single-policy behaviour. The policy passes the cut-in vehicle and completes the route.

Figure 20: Bench2Drive Town05 Long, route 25955: hazard at side lane two ways, for GenAD initialized with PAVER. The policy passes a stopped vehicle and a cyclist in fog and reaches the end of the route; the driving score of 60.00 reflects one recorded collision on an otherwise completed route (Table 22 ).

Figure 21: Pseudo-LiDAR quality, set 1. Measured LiDAR against the UniK3D pseudo-LiDAR used as the pretraining target, in top-down and oblique views.

Figure 22: Pseudo-LiDAR quality, set 2.

Figure 23: Detection pretraining quality. Each panel concatenates the front-left, front, and front-right camera views. Ground-truth and predicted 3D boxes are shown for detection-only, detection–map, and detection–map–occupancy pretraining.

Figure 24: Detection pretraining quality, sample 4.

Figure 25: Detection pretraining quality, sample 5.

Figure 26: Detection pretraining quality, sample 6.

Figure 27: Detection pretraining quality, sample 7.

Figure 28: Map pretraining quality after the shared 20-epoch budget. Columns compare map-only, detection–map, and detection–map–occupancy pretraining against ground truth. The fixed 20-epoch budget is not sufficient for the map task to converge or reach its best attainable quality, so these outputs should be interpreted as intermediate pretraining states.

Figure 44

Figure 30: Downstream detection quality on nuScenes. All detection classes are projected onto the front camera at a common score threshold.

Figure 31: Additional downstream detection examples on nuScenes.

Figure 47

Figure 33: Joint map–planning and curved-driving cases. The first three rows improve both planning L2 and vector-map F1; the remaining rows show curved trajectories with lower planning L2.

Figure 34: Action-state ablation. After ground truth, the columns are VAD-Tiny trained from scratch, PAVER pretrained without action-state conditioning, and full PAVER.

Figure 35: Action-corridor masking ablation. After ground truth, the columns are VAD-Tiny trained from scratch, PAVER pretrained without action-corridor masking, and full PAVER.

Figure 36: Auxiliary capacity quality, PAVER (30K). After ground truth, the columns are VAD-Tiny trained from scratch and PAVER pretrained with 10K and with 30K auxiliary parameters.

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 42: Grad-CAM results on the nuScenes validation set.

Figure 44: Grad-CAM results on the nuScenes validation set.

Figure 46: Grad-CAM results on the nuScenes validation set.

Figure 48: Grad-CAM response to a road-surface stop marking on the nuScenes validation set. PAVER concentrates attribution on the marking.

Figure 50: Grad-CAM results on the nuScenes validation set.

Figure 52: Grad-CAM results on the nuScenes validation set.

Figure 54: Grad-CAM results on the nuScenes validation set.

Figure 56: Grad-CAM results on the nuScenes validation set.

Figure 58: Grad-CAM results on the nuScenes validation set.

Figure 60: Grad-CAM results on the nuScenes validation set.

Figure 62: Grad-CAM results on the nuScenes validation set.

Figure 64: Grad-CAM spatial localization for VAD-Base, scene 1. Grad-CAM maps for the BEV and planning targets with and without PAVER pretraining, shown for the three front cameras.

Figure 65: Grad-CAM spatial localization for VAD-Base, scene 1. Grad-CAM maps for the detection, motion and map targets with and without PAVER pretraining, shown for the three front cameras.

Figure 66: Grad-CAM spatial localization for VAD-Base, scene 3. Grad-CAM maps for the BEV and planning targets with and without PAVER pretraining, shown for the three front cameras.

Figure 67: Grad-CAM spatial localization for VAD-Base, scene 3. Grad-CAM maps for the detection, motion and map targets with and without PAVER pretraining, shown for the three front cameras.

Figure 68: Grad-CAM spatial localization for VAD-Base, scene 4. Grad-CAM maps for the BEV and planning targets with and without PAVER pretraining, shown for the three front cameras.

Figure 69: Grad-CAM spatial localization for VAD-Base, scene 4. Grad-CAM maps for the detection, motion and map targets with and without PAVER pretraining, shown for the three front cameras.

Figure 70: Grad-CAM spatial localization for VAD-Base, scene 5. Grad-CAM maps for the BEV and planning targets with and without PAVER pretraining, shown for the three front cameras.

Figure 71: Grad-CAM spatial localization for VAD-Base, scene 5. Grad-CAM maps for the detection, motion and map targets with and without PAVER pretraining, shown for the three front cameras.

Planning-Aligned Pretraining of BEV Representations with Sparse Action-Conditioned Targets for End-to-End Autonomous Driving

Abstract

Figures & tables

End-to-End Autonomous Driving

Faster Training, Fewer Labels: Self-Supervised Pretraining for Fine-Grained BEV Segmentation

ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

Abstract

End-to-End Autonomous Driving

Explore similar work

Faster Training, Fewer Labels: Self-Supervised Pretraining for Fine-Grained BEV Segmentation

ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving