Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Prediction

Chengxin Lv; Yihui Li; Hongyu Yang; YunHong Wang

cs.CVMar 24, 2026

Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Prediction

Authors: Chengxin Lv, Yihui Li, Hongyu Yang, YunHong Wang

Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China

Abstract

3D semantic occupancy prediction is crucial for autonomous driving. While multi-modal fusion improves accuracy over vision-only methods, it typically relies on computationally expensive dense voxel or BEV tensors. We present Gau-Occ, a multi-modal framework that bypasses dense volumetric processing by modeling the scene as a compact collection of semantic 3D Gaussians. To ensure geometric completeness, we propose a LiDAR Completion Diffuser (LCD) that recovers missing structures from sparse LiDAR to initialize robust Gaussian anchors. Furthermore, we introduce Gaussian Anchor Fusion (GAF), which efficiently integrates multi-view image semantics via geometry-aligned 2D sampling and cross-modal alignment. By refining these compact Gaussian descriptors, Gau-Occ captures both spatial consistency and semantic discriminability. Extensive experiments across challenging benchmarks demonstrate that Gau-Occ achieves state-of-the-art performance with significant computational efficiency.

Figures & tables

Figure 1: Overview of Gau-Occ . Sparse LiDAR scans are first completed by a pretrained LiDAR Completion Diffuser (LCD) to recover occluded geometry. The completed points are encoded into geometric features to initialize density-aware semantic 3D Gaussians. Each Gaussian then anchors multi-view image features via our Gaussian Anchor Fusion (GAF) , producing geometry-aligned multi-modal representations. The refined Gaussians are finally splatted into voxel space for semantic occupancy prediction.

Figure 2: Hybrid Gaussian initialization. Left: raw sparse LiDAR input. Middle: completed point cloud $\mathcal{P}^{\prime}$ from LCD. Right: initialized Gaussian centers derived from $\mathcal{P}^{\prime}$ , with density-based subset $\mathcal{P}_{d}$ (red) and random subset $\mathcal{P}_{r}$ (green).

Figure 3: Schematic of geometry-aware image token resampling and modulation.

Method	Modality	IoU $\uparrow$	mIoU $\uparrow$	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive. surf.	other flat	sidewalk	terrain	manmade	vegetation
Method	Modality	IoU $\uparrow$	mIoU $\uparrow$
MonoScene [ 3 ]	C	24.0	7.3	4.0	0.4	8.0	8.0	2.9	0.3	1.2	0.7	4.0	4.4	27.7	5.2	15.1	11.3	9.0	14.9
C-CONet [ 46 ]	C	26.1	18.4	18.6	10.0	26.4	27.4	8.6	15.7	13.3	9.7	10.9	20.2	33.0	20.7	21.4	21.8	14.7	21.3
GaussianFormer [ 16 ]	C	29.8	19.1	19.5	11.3	26.1	29.8	10.5	13.8	12.6	8.7	12.7	21.6	39.6	23.3	24.5	23.0	9.6	19.1
BEVFormer [ 28 ]	C	30.5	16.8	14.2	6.6	23.5	28.3	8.7	10.8	6.6	4.1	11.2	17.8	37.3	18.0	22.9	22.2	13.8	22.2
TPVFormer [ 15 ]	C	30.9	17.1	16.0	5.3	23.9	27.3	9.8	8.7	7.1	5.2	11.0	19.2	38.9	21.3	24.3	23.2	11.7	20.8

Table 1: Quantitative comparison on SurroundOcc-nuScenes validation set. The best results are in bold , second best are underlined .

Method	mIoU $\uparrow$	Modality	others	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive. surf.	other flat	sidewalk	terrain	manmade	vegetation
Method	mIoU $\uparrow$	Modality
BEVDetOcc [ 12 ]	42.0	C	12.2	49.6	25.1	52.0	54.5	27.9	28.0	28.9	27.2	36.4	42.2	82.3	43.3	54.6	57.9	48.6	43.6
PanoOcc [ 47 ]	42.1	C	11.7	50.5	29.4	49.4	55.5	24.3	33.3	30.6	31.0	34.4	42.6	83.3	44.2	54.4	56.0	45.9	40.4
FB-OCC [ 29 ]	48.9	C	14.3	57.0	38.3	57.7	62.1	34.4	39.4	38.8	39.4	42.9	50.0	86.0	50.2	60.1	62.5	52.4	45.7
FlashOcc [ 53 ]	43.5	C	13.4	51.8	32.1	54.1	58.3	29.9	32.8	31.9	29.9	38.7	46.2	85.0	47.4	56.2	59.6	50.0	43.7
COTR [ 33 ]	46.2	C	14.9	53.3	35.2	50.8	57.4	34.1	33.5	31.9	33.1	39.1	45.0	84.5	48.3	57.6	61.1	51.6	46.7

Table 2: Quantitative comparison on Occ3D-nuScenes validation set. The best results are in bold , second best are underlined .

Figure 4: Qualitative results on the SurroundOcc-nuScenes validation set. Top: multi-view images (left), LiDAR input (center), and predicted image-view occupancy (right). Bottom: predicted 3D Gaussians, BEV occupancy, and front-view occupancy.

Figure 5: Qualitative results on the Occ3D-nuScenes validation set. Top: predicted occupancy. Bottom: ground-truth.

Figure 6: Qualitative comparison between Gaussianformer-2 [ 14 ] , DAOcc [ 52 ] and Gau-Occ on the SurroundOcc-nuScenes validation set.

Table 3: Ablation on the SurroundOcc-nuScenes validation set.

Figure 7: Visualization of ablations on Gaussian initialization.

Figure 8: Visualization of ablations on GAF components.

Method	Modality	Query	Lat.	Mem.	IoU $\uparrow$	mIoU $\uparrow$
Method	Modality	Number	(ms)	(GB)	IoU $\uparrow$	mIoU $\uparrow$
BEVFormer [ 28 ]	C	200×200	310	4.5	30.5	16.8
TPVFormer [ 15 ]	C	200×200×16	320	5.1	30.9	17.1
SurroundOcc [ 48 ]	C	200×200×16	340	5.9	31.5	16.3
GaussianFormer [ 16 ]	C	25600	195	4.8	28.7	16.0
GaussianFormer [ 16 ]	C	144000	372	6.2	29.8	19.1

Table 4: Comparison of inference efficiency on the nuScenes validation set. All results are measured with a batch size of 1 on a single NVIDIA RTX 4090 GPU.

Method	Modality	IoU $\uparrow$	mIoU $\uparrow$	car	bicycle	motorcycle	truck	other-vehicle	person	road	parking	sidewalk	other-ground	building	fence	vegetation	terrain	pole	traffic-sign	other-structure	other-object
Method	Modality	IoU $\uparrow$	mIoU $\uparrow$
LMSCNet [ 39 ]	L	47.5	13.7	20.9	0.0	0.0	0.3	0.0	0.0	63.0	13.5	33.5	0.2	43.7	0.3	40.0	26.8	0.0	0.0	3.6	0.0
SSCNet [ 41 ]	L	53.6	17.0	32.0	0.0	0.2	10.3	0.6	0.1	65.7	17.3	41.2	3.2	44.4	6.8	43.7	28.9	0.8	0.8	8.6	0.7
L2COcc-L [ 45 ]	L	57.6	25.2	40.4	4.1	4.2	26.2	10.3	10.0	70.0	22.6	46.2	8.8	51.0	14.3	47.6	31.3	24.9	21.6	14.0	6.4
MonoScene [ 3 ]	C	37.9	12.3	19.3	0.4	0.6	8.0	2.0	0.9	48.4	11.4	28.1	3.2	32.9	3.5	26.2	16.8	6.9	5.7	4.2	3.1
VoxFormer [ 26 ]	C	38.8	11.9	17.8	1.2	0.9	4.6	2.1	1.6	47.0	9.7	27.2	2.8	31.2	5.0	29.0	14.7	6.5	6.9	3.8	2.4

Table 5: Quantitative comparison on the KITTI-360 validation set. The best results are in bold , second best are underlined .

Figure 9: Qualitative results on KITTI-360.

Figure 10: Additional qualitative results on the SurroundOcc-nuScenes validation set under adverse weather scenarios. Top shows multi-view images (left), raw LiDAR input (center), and predicted image-view occupancy (right); Bottom presents predicted 3D Gaussians, BEV occupancy, and front-view occupancy.

Figure 11: Additional qualitative results on the Occ3D-nuScenes validation set under adverse weather scenarios. Top-left : multi-view images; top-right : predicted image-view occupancy; bottom-left : predicted 3D Gaussians; bottom-right : front-view occupancy; inset: LiDAR input.

Figure 12: Additional qualitative results on the SurroundOcc-nuScenes validation set under dense traffic scenarios.

Figure 13: Additional qualitative results on the Occ3D-nuScenes validation set under dense traffic scenarios.

Explore similar work

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction
Mar 3, 2026A. Enes Doruk, Hasan F. Ates3D Occupancy Prediction Autonomous Driving Perception
SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction
Jul 6, 2026Pin Tang, Zhongdao Wang, Guoqing Wang +23D Reconstruction Autonomous Driving Perception
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Noah’s Ark Lab, China
VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction
Jul 20, 2026Junhong Lin, Xianda Guo, Kangli Wang +4Autonomous Driving 3D Occupancy Prediction
1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor

Mar 3, 2026J/K move · Enter open · S save

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

A. Enes Doruk, Hasan F. Ates

3D Occupancy Prediction Autonomous Driving Perception Camera-LiDAR Fusion VLMs for Autonomous Driving

Open paper AskPDF arXiv

This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.

Abstract

Explore similar work

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction

VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction