CGOcc: Learning Cross-Gaussian Dependencies for 3D Semantic Occupancy Prediction
Authors: Zirui Huang, Jiongle He, Yunfan Zeng, Yuanzhi Lu, Guangye Li, Boyan Tang, Weite SU, Pedro V. Sander, Amy Hui Zhang
Organizations: Hong Kong University of Science and Technology · Nanyang Technological University · The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Beijing Normal-Hong Kong Baptist University
In Gaussian-based semantic occupancy prediction, overlapping primitives can provide ambiguous class predictions, making their joint interpretation important for voxel classification. However, current methods using spatially weighted mixtures neglect explicit modeling of cross-primitive dependencies, limiting the readout's ability to utilize category-specific relational evidence. Hence, we introduce CGOcc, a supervised multi-camera framework that explicitly models Cross-Gaussian dependencies for 3D semantic Occupancy prediction. CGOcc integrates appearance and geometric observations to refine Gaussian states and encodes cross-Gaussian dependencies in a diagonal-plus-low-rank representation. To incorporate these dependencies into voxel classification, CGOcc uses a class-conditional projection to extract diagonal and signed cross-Gaussian statistics without constructing a dense dependency matrix. The readout then refines occupied-class probabilities using these statistics while preserving a learned reference's occupied mass and free probability. Extensive experiments on nuScenes demonstrate CGOcc achieves state-of-the-art performance, with larger gains at high Gaussian overlap and improved predictions near boundaries. Ablation studies validate the effectiveness of covariance-aware prediction and support its synergy with geometric interaction.
Figures & tables
Figure 1: Cross-Gaussian dependencies for semantic occupancy. A schematic vehicle–road example compares weighted aggregation (middle) with CGOcc (bottom), with input and ground truth above. Blue and magenta denote vehicle and road; yellow and black circles mark the input region and enlarged regions, respectively. Pale yellow edges indicate selected cross-Gaussian dependencies.
Figure 2: Overview of CGOcc. Depth-initialized Gaussians undergo two geometry-conditioned multi-source interaction blocks before dependency-aware semantic readout (Fig. 3 ). Only Block 1 uses local graph attention; snowflakes denote frozen modules.
Figure 3: Overview of Dependency-Aware Semantic Prediction. Class-conditional projection extracts diagonal and cross-Gaussian statistics from a diagonal-plus-low-rank representation. Shared-head differencing guides semantic reweighting of the learned reference, preserving its occupied mass and free probability.
Method
OccIoU ↑
mIoU ↑
others
barrier
bicycle
bus
car
const. veh.
motorcycle
pedestrian
traffic cone
trailer
truck
drive. surf.
other flat
sidewalk
terrain
manmade
vegetation
MonoScene [ 4 ]
29.55
6.06
1.75
7.23
4.26
4.93
9.38
5.67
3.98
3.01
5.90
4.45
7.17
14.91
6.32
7.92
7.43
1.01
7.65
BEVDet [ 13 ]
38.23
19.38
4.39
30.31
0.23
32.26
34.47
12.97
10.34
10.36
6.26
8.93
23.65
52.27
24.61
26.06
22.31
15.04
15.10
OccFormer [ 53 ]
41.95
21.93
5.94
30.29
12.32
34.40
39.17
14.44
16.45
17.22
9.27
13.90
26.36
50.99
30.96
34.66
22.73
6.76
6.97
BEVStereo [ 25 ]
44.33
24.51
5.73
38.41
7.88
38.70
41.20
17.56
17.33
14.69
10.31
16.84
29.62
54.08
28.92
32.68
26.54
18.74
17.49
BEVFormer [ 26 ]
50.78
26.88
5.85
37.83
17.87
40.44
42.43
7.36
23.88
21.81
20.98
22.38
30.70
55.35
28.36
36.0
28.06
20.04
17.69
CTF-Occ [ 40 ]
51.24
28.53
8.09
39.33
20.56
38.29
42.24
16.93
24.52
22.72
21.05
22.98
31.11
53.33
33.84
37.98
33.23
20.79
18.0
Table 1: 3D semantic occupancy prediction results on the Occ3D-nuScenes validation set. † GaussianFormer-2 uses the 128-channel configuration ( Ch.=128 ). ‡ RenderOcc denotes the variant trained with both 3D voxel labels and LiDAR-derived 2D semantic and depth labels. Column-wise best results are bold and second best results are underlined.
Figure 4: Qualitative comparison on Occ3D-nuScenes validation set.
Figure 5: Visualization of prediction changes near boundaries on Occ3D-nuScenes. Local maps show changes from each baseline to CGOcc. Corrections occur along semantic interfaces and occupied/free edges, alongside residual and new errors.
Figure 6: Visualization of Gaussian overlap and prediction changes from CGOcc (Diag) to CGOcc. Overlap is measured using a fixed CGOcc (Diag) reference. Blue and red mark corrected predictions and new errors, respectively.
Figure 8
Decoder variant
Diag
Cov *
Sj
Full (F)
29.42±0.19
31.10±0.20
—
Within-source (W)
29.23±0.20
29.87±0.24
1.04±0.23
w/o geom. bias (G)
29.11±0.17
30.08±0.24
0.71±0.19
w/o neighbor msg. (N)
29.25±0.18
29.71±0.22
1.22±0.23
Table 4: Decoder–covariance interaction. Sj=ΔF−Δj , where Δj=Yj,Cov−Yj,Diag and Y denotes mIoU. Larger positive values indicate stronger positive synergy between decoder mechanism j and covariance-aware prediction. * denotes Full model.
Readout Variant
mIoU (%) ↑
OccIoU (%) ↑
P(0) -only
29.25
54.56
Diag
29.42
54.78
Class-agnostic
30.28
56.20
Cov (aligned factors)
30.43
56.24
Cov *
31.10
56.32
Cov (cross off)
30.30
56.13
Table 5: Ablation of semantic readout. Variants above the separator are trained independently. Cov (cross off) evaluates the trained Cov checkpoint with only the explicit cross-statistic input set to zero. * denotes Full model.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Gaussians / blocks / attention heads
7,200 / 2 / 4
Hidden / attention FFN width
256 / 512
Adapter / geom. bias hidden width
256 / 64
Dropout / sampling points / slots
0 / 2 / 48
Sampling-offset scale λoff
0.5
Graph neighbors K / radius R
16 / 1.6 m
Appendix
Table 6: Architecture and readout settings on Occ3D-nuScenes.
Table 7: 3D semantic occupancy prediction results on the SurroundOcc-nuScenes validation set. † GaussianFormer-2 uses the 128-channel configuration ( Ch.=128 ). Column-wise best results are bold and second best results are underlined.
Figure 9: Quantitative comparison across distances to ground-truth boundaries. Panels (a) and (b) report mIoU and OccIoU over common distance bins. Distances are unsigned 3D Euclidean distances to occupied-class and occupied/free interfaces.
Figure 10: Visualization of prediction changes near boundaries on Occ3D-nuScenes. Each scene shows two local regions, with change maps comparing CGOcc against CGOcc (Diag) and external baselines. Solid and dashed lines mark occupied-class and occupied/free interfaces, respectively; dotted lines indicate the 0.8 m boundary neighborhood. Colors distinguish corrected voxels and different error types.
Figure 11: Visualization of Gaussian overlap and prediction changes from CGOcc (Diag) to CGOcc on Occ3D-nuScenes. Overlap maps use the effective Gaussian contribution count from a fixed CGOcc (Diag) reference. Darker purple indicates greater overlap, and dotted contours mark high-overlap regions. In the change maps, blue and red denote corrected predictions and new errors, respectively.
Open-vocabulary 3D occupancy prediction represents scene geometry and semantics in a unified voxel grid, enabling recognition beyond predefined categories. One approach reconstructs scenes with 3D Gaussians and converts them into voxel occupancy. However, Gaussian scale inflation and excessive overlap can distort voxel occupancy predictions, while semantic mixing can arise within Gaussians spanning multiple categories and during 3D--2D semantic feature alignment. We propose GCSR-OCC, which combines geometry calibration and multi-mode query-conditioned semantic routing for indoor occupancy prediction from a single RGB image. The Evidence-Conserving Gaussian-to-Occupancy operator (EC-G2O) uses kernel mass normalization, p-norm aggregation, and a learnable gain to suppress scale inflation and excessive local evidence accumulation while calibrating occupancy responses. Multi-mode Query-First Gaussian Semantic Routing (MM-QGSR) matches multiple semantic modes per Gaussian to text queries and aggregates query-specific evidence into voxels through a learnable text-conditioned router. Fine-grained subclass queries and smoothed semantic aggregation further refine voxel-level predictions. Without 3D semantic annotations, GCSR-OCC achieves state-of-the-art performance in open-vocabulary occupancy prediction on Occ-ScanNet, with 63.31% geometric IoU and 27.44% semantic mIoU. It also reduces geometric aggregation latency from 19.76 ms to 12.88 ms. Our code will be made publicly available.
Camera-only 3D semantic occupancy prediction aims to infer voxel-wise scene semantics from multi-view images, yet remains fundamentally challenging due to depth ambiguity and occlusion. Along a single camera ray, multiple spatially separated surfaces may coexist, making occupancy inherently a multi-label existence problem rather than a single-depth estimation task. However, most existing approaches favor a single dominant depth hypothesis per ray, limiting their ability to model volumetric scenes under complex occlusion. To address this limitation, we introduce RayOcc, an occlusion-aware ray occupancy framework that reformulates ray modeling as multi-label existence prediction. Instead of predicting a categorical depth distribution, RayOcc estimates a non-normalized Gaussian mixture intensity along each ray and converts it into interval-wise occupancy probabilities via a Poisson event formulation, allowing multiple occupied hypotheses to coexist without enforcing mutual competition across depth. The predicted mixture components are interpreted as occupancy hypotheses to initialize sparse 3D Gaussian primitives, which are refined and rasterized for semantic occupancy prediction. Experiments on the nuScenes benchmark show that RayOcc achieves state-of-the-art overall IoU and mIoU among the compared Gaussian-based occupancy methods.
Junho Kim, Seongwon Lee
School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea
Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.
Junhong Lin, Xianda Guo, Kangli Wang +4
1SECE, Peking University · School of Computer Science, Wuhan University · 3Great Wall Motor