Reconstruction Matters: Learning Geometry-Aligned BEV Representation through 3D Gaussian Splatting
Authors: Yiren Lu, Xin Ye, Burhaneddin Yaman, Jingru Luo, Zhexiao Xiong, Liu Ren, Yu Yin
Organizations: Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · Case Western Reserve University · Washington University in St. Louis
Bird's-Eye-View (BEV) perception serves as a cornerstone for autonomous driving, offering a unified spatial representation that fuses surrounding-view images to enable reasoning for various downstream tasks, such as semantic segmentation, 3D object detection, and motion prediction. However, most existing BEV perception frameworks adopt an end-to-end training paradigm, where image features are directly transformed into the BEV space and optimized solely through downstream task supervision. This formulation treats the entire perception process as a black box, often lacking explicit 3D geometric understanding and interpretability, leading to suboptimal performance. In this paper, we claim that an explicit 3D representation matters for accurate BEV perception, and we propose Splat2BEV, a Gaussian Splatting-assisted framework for BEV tasks. Splat2BEV aims to learn BEV feature representations that are both semantically rich and geometrically precise. We first pre-train a Gaussian generator that explicitly reconstructs 3D scenes from multi-view inputs, enabling the generation of geometry-aligned feature representations. These representations are then projected into the BEV space to serve as inputs for downstream tasks. Extensive experiments on nuScenes and argoverse dataset demonstrate that Splat2BEV achieves state-of-the-art performance and validate the effectiveness of incorporating explicit 3D reconstruction into BEV perception.
Figures & tables
Figure 1: An overview of the proposed Splat2BEV framework.Compared to traditional implicit end-to-end approaches, Splat2BEV first reconstructs the scene using 3D Gaussian Splatting, and then projects the reconstructed scene into the Bird’s-Eye-View to obtain geometry-aligned representations for downstream tasks, leading to substantial performance improvements.
Figure 2: An overview of our training process. Given multi-view perspective images as input, Splat2BEV first trains a feed-forward Gaussian generator to reconstruct 3D scene using 3D Gaussian Splatting. In stage 2, the Gaussian generator is frozen, and the reconstructed geometry along with its associated features are projected onto the BEV plane. A BEV encoder and segmentation head are then trained on top of this BEV representation to perform downstream tasks. Finally, in the third stage, the Gaussian generator, BEV encoder, and segmentation head are jointly fine-tuned, allowing geometry, semantics, and task-specific cues to be harmonized for optimal BEV perception.
Figure 3: Our Gaussian generator consists of two branches: a multi-view branch based on UniMatch [ 42 ] , and a per-view branch built upon ViT-S [ 9 ] to augment the multi-view representation. The multi-view branch outputs multi-view features and cost volumes, which are concatenated with monocular features from the per-view branch to serve as input for a U-Net [ 39 ] to predict depth maps and per-Gaussian parameters. The predicted depths are then unprojected and combined with the Gaussian parameters to form 3D Gaussians.
Vehicle IoU ( ↑ )
No visibility filtering
Visibility filtering
Method
224 × 480
448 × 800
224 × 480
448 × 800
BEVFormer [ 27 ]
35.8
39.0
42.0
45.5
Simple-BEV [ 14 ]
36.9
40.9
43.0
44.9
PointBeV [ 4 ]
38.7
42.1
44.0
47.6
CVT [ 50 ]
31.4
32.5
36.0
37.7
BAEFormer [ 36 ]
36.0
37.8
38.9
41.0
Table 1: Comparison of BEV segmentation results for Vehicle on the nuScenes dataset.
Table 5
Method
Driv.
Walk.
Cross.
Carpark
Car
Truck
Bus
Trailer
C.V.
Ped.
Motor.
Mean
VPN [ 11 ]
58.0
29.4
27.3
12.9
25.5
17.3
20.0
16.6
4.9
7.1
5.6
20.4
PON [ 12 ]
60.4
31.0
28.0
18.4
24.7
16.8
20.8
16.6
12.3
8.2
7.0
22.2
LSS [ 38 ]
55.9
34.4
31.3
23.7
27.3
16.8
27.3
17.0
9.2
6.8
6.6
23.3
PYVA [ 13 ]
56.2
32.2
26.4
21.3
19.3
13.2
21.4
12.5
7.4
4.2
3.5
19.8
DiffBEV [ 53 ]
65.4
41.1
41.3
28.4
38.9
23.1
33.7
21.1
8.4
9.6
14.4
29.6
Ours
74.7
46.2
39.1
39.9
39.8
26.1
37.8
30.9
12.3
16.2
14.7
34.4
Table 4: Multi-class segmentation results on nuScenes.
Method
Driv.
Vehicle
Ped.
L. Veh.
Bicycle
Bus
Trailer
Motor.
Mean
VED [ 32 ]
62.9
14.0
1.0
3.9
0.0
12.3
1.3
0.0
11.9
VPN [ 11 ]
64.9
23.9
6.2
9.7
0.9
3.0
0.4
1.0
13.9
PON [ 12 ]
65.4
31.4
7.4
11.1
3.6
11.0
0.7
5.7
17.0
TaDe [ 49 ]
68.3
34.5
9.3
14.7
4.4
37.8
3.1
6.4
22.3
Ours
77.2
36.4
12.3
15.3
3.9
38.6
4.5
6.6
24.4
Table 5: Multi-class segmentation results on Argoverse.
Figure 4: Visualization of reconstruction quality. The left side shows the 3D reconstruction, its feature field, and the corresponding BEV map and projected BEV feature. The right side provides zoomed-in views that highlight fine-grained details, such as the zebra crossing in View 1, the dustbin in View 2, the person in View 3, and the no-parking line in View 4.
Figure 5: Visual comparison of features learned with and without explicit reconstruction. The BEV feature refers to the feature map produced by the BEV encoder, while the projected feature denotes the feature directly projected from the 3D representation.
Method
Vehicle ( ↑ )
Pedestrian ( ↑ )
Lane ( ↑ )
LSS [ 38 ]
32.1
15.0
20.0
BEVFormer [ 26 ]
35.8
16.4
25.7
w/o joint (Ours)
34.5
15.9
49.2
Ours
39.6
20.1
60.2
Table 6: Quantitative assessment (IoU) of our geometry-aligned feature on downstream tasks.
Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.
The bird's-eye view (BEV) representation enables multi-sensor features to be fused within a unified space, serving as the primary approach for achieving comprehensive 3D perception. However, the discrete grid representation of BEV leads to significant detail loss and limits feature alignment and cross-modal information interaction in multimodal fusion perception. In this work, we break from the conventional BEV paradigm and propose a new universal framework for multi-modal fusion based on 3D Gaussian representation. This approach naturally unifies multi-modal features within a shared and continuous 3D Gaussian space, effectively preserving edge and fine texture details. To achieve this, we design a novel forward-projection-based multi-modal Gaussian initialization module and a shared cross-modal Gaussian encoder that iteratively updates Gaussian properties based on an attention mechanism. GaussianFusion is inherently a task-agnostic model, with its unified Gaussian representation naturally supporting various 3D perception tasks. Extensive experiments demonstrate the generality and robustness of GaussianFusion. On the nuScenes dataset, it outperforms the 3D object detection baseline BEVFusion by 2.6 NDS. Its variant surpasses GaussFormer on 3D semantic occupancy with 1.55 mIoU improvement while using only 30% of the Gaussians and achieving a 450% speedup.
Bird's Eye View (BEV)-based multi-view 3D object detection suffers from challenges of computational complexity, multi-scale feature extraction, and efficiency of dense 2D-to-BEV view transformation. To address these problems, this paper proposes an improved BEV 3D object detection algorithm Sparse-BEVNet. Firstly, a Bi-Level Routing Attention (BRA) mechanism is introduced into the image feature extraction network to reduce the computational burden of the backbone. Second, Cascaded Group Attention (CGA) is employed in the feature fusion module, which enhances deep interaction across features of different hierarchical levels without introducing additional computational overhead. Furthermore, a Sparse Spatial Cross-Attention mechanism is adopted to replace the conventional dense view projection pipeline. Experimental results on the public nuScenes dataset demonstrate that the proposed method achieves a mean Average Precision (mAP) of 45.2% and a nuScenes Detection Score (NDS) of 54.5%, corresponding to 3.6% and 2.8% improvements relative to the baseline model, respectively.
Jing Zhang, Jiaqi Liu, Zibo Wang
Stuart Weitzman School of Design University of Pennsylvania Philadelphia, USA · Courant Institute of Mathematical Science New York University New York, USA