Dense Bird's Eye View (BEV) semantic maps are central to autonomous driving, yet current multi-camera methods depend on costly, inconsistently annotated BEV ground truth. We address this limitation with a two-phase training strategy for fine-grained road marking segmentation that removes full supervision during pretraining and halves the amount of training data during fine-tuning while still outperforming the comparable supervised baseline model. During the self-supervised pretraining, BEVFormer predictions are differentiably reprojected into the image plane and trained against multi-view semantic pseudo-labels generated by the widely used semantic segmentation model Mask2Former. A temporal loss encourages consistency across frames. The subsequent supervised fine-tuning phase requires only 50% of the dataset and significantly less training time. With our method, the fine-tuning benefits from rich priors learned during pretraining boosting the performance and BEV segmentation quality (up to +2.5pp mIoU over the fully supervised baseline) on nuScenes. It simultaneously halves the usage of annotation data and reduces total training time by up to two thirds. The results demonstrate that differentiable reprojection plus camera perspective pseudo labels yields transferable BEV features and a scalable path toward reduced-label autonomous perception.
Figures & tables
Fig. 1 : BEVFormer architecture [ 1 ] with the self-supervised pretraining. Left: Camera input. Center: the BEVFormer with its transformer module containing Temporal Self-Attention into History Bird’s Eye View and spatial Cross-Attention. Right: Output of the BEVFormer. Bottom right the differentiable reprojection and rendering. Bottom center and left 2D Reconstruction loss comparing the six reconstructed predctions Predcp with the camera perspective pseudo ground truth GTcp .
Fig. 2 : Ground truth comparison for pseudo ground truth (a) and NuScenes ground truth (b) in front view (left) and Bird’s Eye View (right). For comparison the pseudo ground truth is projected to Bird’s Eye View using nuScenes lidar data. ( road boundary ; lanes ; crosswalk ).
Figure 3
Method
boundary IoU ↑
lane IoU ↑
crosswalk IoU ↑
mIoU20 ↑
mIoU40 ↑
mIoU60 ↑
supervised baseline
25.1
26.4
11.5
31.5
25.6
21.0
self-supervised pretraining
7.1
6.0
3.2
6.3
6.0
5.4
self-superv. pretrain + temp
6.8
6.0
5.4
7.4
7.0
6.1
ours(3ep pretrain + temp)
25.2
27.2
14.9
32.7
26.7
22.4
ours(3ep pretrain)
24.8
27.0
14.9
32.7
26.7
22.3
ours(5ep pretrain + temp)
24.5
26.4
14.8
32.3
26.3
21.9
TABLE I : The different Intersection over Unions for our methods: Self-superv. pretrain: supervision with camera perspective pseudo ground truth, temp: Temporal loss mechanism included, ours( x ep pretrain): two phase training with x epochs pretraining and fine-tuning on the halved dataset. mIoU20: 20m consideration range ( 1/3 ), mIoU40: 40m consideration range ( 2/3 ), mIoU60: full range of 60m . ↑ : higher values are better. best results , second best results .
Fig. 4 : Relation between road distance in m and image portion in % . The different colors represent equally long distances each covering one quarter of the total predicted distance.
Fig. 5 : Results of our self-supervised pretraining (pretrain), the temporal loss mechanism (temp) and our two phase strategy including temporal loss pretraining and supervised fine-tuning (ours( x epochs pretraining)). road boundary ; lanes ; crosswalk .
Fig. 6 : mean Intersection over Union progress evaluated on nuScenes ground truth for the supervised baseline, self-supervised pretraining, and our two phase training with different pretraining lengths including the temporal loss.
Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.
Transforming image features from perspective view (PV) space to bird's-eye-view (BEV) space remains challenging in autonomous driving due to depth ambiguity and occlusion. Although several view transformation (VT) paradigms have been proposed, the challenge still remains. In this paper, we propose a new regularization framework, dubbed CycleBEV, that enhances existing VT models for BEV semantic segmentation. Inspired by cycle consistency, widely used in image distribution modeling, we devise an inverse view transformation (IVT) network that maps BEV segmentation maps back to PV segmentation maps and use it to regularize VT networks during training through cycle consistency losses, enabling them to capture richer semantic and geometric information from input PV images. To further exploit the capacity of the IVT network, we introduce two novel ideas that extend cycle consistency into geometric and representation spaces. We evaluate CycleBEV on four representative VT models covering three major paradigms using the large-scale nuScenes dataset. Experimental results show consistent improvements -- with gains of up to 0.74, 4.86, and 3.74 mIoU for drivable area, vehicle, and pedestrian classes, respectively -- without increasing inference complexity, since the IVT network is used only during training. The implementation code is available at https://github.com/JeongbinHong/CycleBEV.
Jeongbin Hong, Dooseop Choi, Taeg-Hyun An +2
Electronics and Telecommunications Research Institute (ETRI) · University of Science and Technology (UST)
End-to-end driving requires planning-relevant bird's-eye-view (BEV) representations, but existing pretraining approaches often rely on task annotations or dense scene reconstruction. We introduce PAVER, Planning-Aligned BEV Encoder Pretraining. From a single LiDAR sweep, PAVER constructs sparse risk and unknown targets describing occupied and unobserved evidence along rule-based ego motions. A 10K-parameter head predicts these targets from masked BEV features conditioned on the action state, directing supervision toward geometric constraints on candidate motions. Pretraining requires no driving-task annotations or dense reconstruction. Only the BEV encoder is transferred, preserving the downstream architecture and camera-only inference. On nuScenes, PAVER reduces VAD-Tiny's average collision rate from 0.51% to 0.19%, while improving planning L2, motion prediction, detection, and mapping. The selected VAD-Tiny and VAD-Base schedules use about 36% less estimated total training time than scratch training, including pretraining. On Bench2Drive Town05 Long, PAVER improves UniAD-Tiny's closed-loop Driving Score from 48.45 to 58.79. The project page is available at https://archiiive99.github.io/PAVER.