VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Prediction

Xiaoyang Yan; Muleilan Pei; Shaojie Shen

cs.CVMar 6, 2026

VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Prediction

Authors: Xiaoyang Yan, Muleilan Pei, Shaojie Shen

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China

Abstract

3D semantic occupancy prediction has become a crucial perception task for comprehensive scene understanding in autonomous driving. While recent advances have explored 3D Gaussian splatting for occupancy modeling to substantially reduce computational overhead, the generation of high-quality 3D Gaussians relies heavily on accurate geometric cues, which are often insufficient in purely vision-centric paradigms. To bridge this gap, we advocate for injecting the strong geometric grounding capability from Vision Foundation Models (VFMs) into occupancy prediction. In this regard, we introduce Visual Geometry Grounded Gaussian Splatting (VG3S), a novel framework that empowers Gaussian-based occupancy prediction with cross-view 3D geometric grounding. Specifically, to fully exploit the rich 3D geometric priors from a frozen VFM, we propose a plug-and-play hierarchical geometric feature adapter, which can effectively transform generic VFM tokens via feature aggregation, task-specific alignment, and multi-scale restructuring. Extensive experiments on the nuScenes occupancy benchmark demonstrate that VG3S achieves remarkable improvements of 12.6% in IoU and 7.5% in mIoU over the baseline. Furthermore, we show that VG3S generalizes seamlessly across diverse VFMs, consistently enhancing occupancy prediction accuracy and firmly underscoring the immense value of integrating priors derived from powerful, pre-trained geometry-grounded VFMs.

Figures & tables

Fig. 2: Framework overview of VG3S . Our approach leverages a powerful, pre-trained frozen VFM to provide rich 3D geometric priors, empowering the downstream Gaussian-based decoder with cross-view 3D geometric grounding and thereby significantly improving 3D semantic occupancy prediction.

Method	Venue	SC IoU	SSC mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ const. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. suf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation
MonoScene [ 31 ]	CVPR 2022	23.96	7.31	4.03	0.35	8.00	8.04	2.90	0.28	1.16	0.67	4.01	4.35	27.72	5.20	15.13	11.29	9.03	14.86
Atlas [ 32 ]	ECCV 2020	28.66	15.00	10.64	5.68	19.66	24.94	8.90	8.84	6.47	3.28	10.42	16.21	34.86	15.46	21.89	20.95	11.21	20.54
BEVFormer [ 8 ]	ECCV 2022	30.50	16.75	14.22	6.58	23.46	28.28	8.66	10.77	6.64	4.05	11.20	17.78	37.28	18.00	22.88	22.17	13.80	22.21
TPVFormer [ 16 ]	CVPR 2023	11.51	11.66	16.14	7.17	22.63	17.13	8.83	11.39	10.46	8.23	9.43	17.02	8.07	13.64	13.85	10.34	4.90	7.37
TPVFormer † [ 16 ]	CVPR 2023	30.86	17.10	15.96	5.31	23.86	27.32	9.79	8.74	7.09	5.20	10.97	19.22	38.87	21.25	24.26	23.15	11.73	20.81
OccFormer [ 14 ]	ICCV 2023	31.39	19.03	18.65	10.41	23.92	30.29	10.31	14.19	13.59	10.13	12.49	20.77	38.78	19.79	24.19	22.21	13.48	21.35

TABLE I: 3D semantic occupancy prediction results on the nuScenes benchmark.

Fig. 3: Qualitative comparison between the baseline GaussianFormer-2 [ 11 ] and our proposed VG3S . Our approach produces more geometrically accurate and consistent object structures across four challenging scenes compared to the baseline, demonstrating that leveraging strong 3D geometric priors embedded within VFMs significantly improves 3D semantic occupancy predictions.

Method	IoU	mIoU
GaussianFormer-2 [ 11 ]	30.56	20.02
VG3S-DINOv2 [ 19 ]	32.34	20.42
VG3S-VGGT [ 13 ]	33.29	21.10
VG3S-DGGT [ 22 ]	33.37	20.81
VG3S-DINOv3 [ 20 ]	33.20	21.36
VG3S-DVGT [ 21 ]	34.41	21.52

TABLE II: Evaluation of generalization across diverse VFMs.

Method	HGFA	IoU	mIoU
VG3S-base	✓	33.29	21.10
w/o HGFA	✗	30.59	19.31

TABLE III: Effect of the HGFA.

Method	GATF	TATR	LSFP	IoU	mIoU
VG3S-base	✓	✓	✓	33.29	21.10
w/o GATF	✗	✓	✓	32.52	20.43
w/o TATR	✓	✗	✓	32.70	20.38
w/o LSFP	✓	✓	✗	32.57	20.33

TABLE IV: Ablation study on the components of the HGFA.

# of groups ( $K$ )	IoU	mIoU
1	31.45	19.52
3	32.66	20.25
4	33.29	21.10
6	32.57	20.55

TABLE V: Effect of the grouping strategy in the GATF.

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction
Jul 20, 2026Junhong Lin, Xianda Guo, Kangli Wang +4Autonomous Driving 3D Occupancy Prediction
1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor
Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction
Feb 25, 2026Changqing Zhou, Yueru Luo, Changhao Chen3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction
May 16, 2026Xun Chen, Tianchen Deng, Rui Wang +53D Semantic Segmentation 3D Occupancy Prediction
Nanyang Technological University · Shanghai Jiao Tong University · ETH Zurich

Jul 20, 2026J/K move · Enter open · S save

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Junhong Lin, Xianda Guo, Kangli Wang, Yuqi Ye+3

1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor

Autonomous Driving 3D Occupancy Prediction 3D Scene Representation

Open paper AskPDF arXiv

Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.

Abstract

3D Occupancy Prediction

Explore similar work

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction

Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction

VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction