The prediction of 3D semantic occupancy enables autonomous vehicles (AVs) to perceive the fine-grained geometric and semantic scene structure for safe navigation and decision-making. Existing methods mainly rely on either voxel-based representations, which incur redundant computation over empty regions, or object-centric Gaussian primitives, whose ellipsoidal per-primitive geometry can limit representation efficiency for complex, non-convex, and asymmetric structures under a fixed primitive and computational budget. In this paper, we present TFusionOcc, a T-primitive-based object-centric multi-sensor fusion framework for 3D semantic occupancy prediction. Specifically, we introduce a family of Student's t-distribution-based T-primitives, including the plain T-primitive, T-Superquadric, and deformable T-Superquadric with inverse warping, where the deformable T-Superquadric serves as the key geometry-enhancing primitive. We further develop a unified probabilistic formulation based on the Student's t-distribution and the T-mixture model (TMM) to jointly model occupancy and semantics, and design a tightly coupled multi-stage fusion architecture to integrate camera and LiDAR cues effectively. Extensive experiments on nuScenes demonstrate competitive performance against recent state-of-the-art methods and strong performance among object-centric multi-sensor approaches, while additional evaluations on nuScenes-C show robust performance across most corruption scenarios. Further evaluation on the separate SemanticKITTI benchmark demonstrates cross-dataset applicability. The code will be available at: https://github.com/DanielMing123/TFusionOcc
Figures & tables
Fig. 1 : Pipeline of three approaches: Voxel-based approach (top), 3D-Gaussian-Primitive-based object-centric approach (middle), and our approach (bottom).
Fig. 2 : Overall architecture of TFusionOcc. The pipeline comprises a camera branch, a LiDAR branch, and a multi-stage fusion branch. The camera branch extracts multi-scale visual features and predicts a pseudo 3D point cloud from surround-view images. The pseudo 3D point cloud is further projected and cylindrically partitioned, resulting in camera-based, multi-scale, dense depth maps and a voxel volume defined under cylindrical coordinates. The LiDAR branch applies a cylindrical partition followed by a 3D encoder to extract the LiDAR feature. Meanwhile, the LiDAR point cloud is also projected to generate LiDAR-based, multi-scale, sparse depth maps. The feature fusion branch adopts a multi-stage fusion strategy to merge all outputs from the two-modality branch and leverages a proposed transformer to refine the T-primitives property through fused features.
Fig. 3 : Inner Structure of DepthNet. The 1/8-scale visual features are first used to generate a 1/8-scale depth map. Then, bilinear interpolation is leveraged to generate 1/16- and 1/32-scale depth maps. Meanwhile, the image-based pseudo-point cloud is generated solely from a 1/8-scale depth map.
Fig. 4 : Multi-Scale Fused Dense Depth Maps Generation. This demonstrates the detailed multi-source depth map fusion from a per-pixel ray perspective.
Fig. 5 : Skeleton Merge Module. The upper LiDAR-branch serves as the main skeleton to provide a foundation structure for the 3D scene, and the bottom camera-branch serves as an augmentation based on the main skeleton to provide more detailed local structure to compensate for the fine-grained geometry of the main skeleton.
Fig. 6 : Early-Fusion Module. Each occupied voxel center in VolCylinLidar serves as an anchor and is projected onto multi-scale visual features to aggregate semantic information, yielding a semantic-aware LiDAR feature.
Fig. 7 : Fused Depth Map Guided 3D Deformable Attention For the same visual feature at location (ui,vj) , several different depth values ranging between [0.0,1.0] along the depth-axis are used to weight the same pixel feature, resulting in the depth encoded visual feature to mitigate the projection ambiguity problem stemming from the projection process.
Fig. 8 : Gate-Concatenation and Weight-Summation Fusion Module. The left part exhibits the multi-modality weighted summation fusion part, and the right part exhibits the multi-modality gated concatenation fusion part. The outputs of the two major parts are further fused to produce the final fused feature for each T-Primitive.
Methods
Modality
mIoU
■ person
■ bicyclist
■ motorcyclist.
■ bicycle
■ motorcycle
■ traf.-sign
■ car
■ truck
■ other-veh.
■ vegetation
■ trunk
■ fence
■ road
■ sidewalk
■ parking
■ other-grnd
■ building
■ terrain
■ pole
JS3C-Net [ 52 ]
C
10.31
0.67
0.27
0.00
0.00
0.00
1.45
24.65
4.41
6.15
18.11
4.33
3.94
50.49
23.74
11.94
0.07
15.03
26.86
3.77
MonoScene [ 36 ]
C
11.08
1.86
1.20
0.00
0.61
0.45
2.25
23.26
6.98
1.48
17.89
2.81
5.84
56.52
26.72
14.27
0.46
14.09
29.64
4.14
VoxFormer [ 53 ]
C
12.35
1.78
3.32
0.00
0.59
0.51
4.18
25.79
5.63
3.77
24.39
5.08
7.64
54.76
26.35
15.50
0.70
17.65
29.96
7.11
LMSCNet [ 46 ]
L
6.70
0.00
0.00
0.00
0.0
0.0
0.00
18.33
0.00
0.0
13.66
0.02
1.21
40.68
18.22
4.38
0.00
10.31
20.54
0.00
M-CONet [ 40 ]
C+L
17.25
2.07
2.38
0.00
0.66
2.2
14.09
33.25
19.46
5.99
32.56
16.01
12.25
58.00
29.44
19.33
0.02
27.85
37.31
14.88
OccFusion [ 6 ]
C+L
21.64
2.33
3.07
0.00
2.00
3.39
14.36
45.72
19.41
7.92
41.21
19.18
15.74
65.73
36.46
21.35
0.00
39.32
46.28
27.60
TABLE IV : 3D semantic scene completion performance on the SemanticKITTI validation set . Classes are grouped to highlight performance on VRUs, Small Objects, and Complex Structures. Notation of modality: Camera (C), LiDAR (L).
Fig. 9 : Performance at different sector ranges for the 3D semantic occupancy prediction task. (a) mIoU performance at different sector ranges on the whole SurroundOcc-nuScenes validation set, (b) mIoU performance at different sector ranges on the SurroundOcc-nuScenes validation rainy scenario subset, (c) mIoU performance at different sector ranges on the SurroundOcc-nuScenes validation night scenario subset, (d) IoU performance at different sector ranges on the whole SurroundOcc-nuScenes validation set, (e) IoU performance at different sector ranges on the SurroundOcc-nuScenes validation rainy scenario subset, and (f) IoU performance at different sector ranges on the SurroundOcc-nuScenes validation night scenario subset. Better viewed when zoomed in.
Fig. 10 : Qualitative results compare against other SOTA algorithms for daytime, rainy, and nighttime scenarios displayed in the upper, middle, and bottom sections, respectively. Better viewed when zoomed in.
This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.
Vision-based 3D semantic occupancy prediction is essential for autonomous driving, yet dense voxel representations waste computation on largely empty space, while BEV and TPV projections compromise fine-grained 3D structure. Fully sparse representations offer an attractive alternative, but existing methods, including SparseOcc, entangle scene completion with semantic prediction by indiscriminately propagating high-dimensional features into empty regions and applying voxel-wise classification. This creates excessive activations, computational overhead, and geometric ambiguity. We present SparseOcc++, a geometry-aware sparse framework that explicitly decouples scene completion from semantic segmentation. SparseOcc++ reformulates completion as signed-distance regression on sparse anchor voxels through a scene completion field (SCF). To model complex outdoor geometry robustly, it combines orthogonal decomposition with discretized distance learning. A geometry-guided propagation module then converts the SCF into a complete volumetric scene and restricts semantic segmentation to geometrically verified regions. Experiments establish new state of the art: SparseOcc++ improves IoU by 2.3 points and is 3.9x faster than SparseOcc on nuScenes, while achieving a 5.9x speedup over OccFormer on SemanticKITTI.
Pin Tang, Zhongdao Wang, Guoqing Wang +2
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Noah’s Ark Lab, China
3D semantic occupancy prediction is crucial for autonomous driving. While multi-modal fusion improves accuracy over vision-only methods, it typically relies on computationally expensive dense voxel or BEV tensors. We present Gau-Occ, a multi-modal framework that bypasses dense volumetric processing by modeling the scene as a compact collection of semantic 3D Gaussians. To ensure geometric completeness, we propose a LiDAR Completion Diffuser (LCD) that recovers missing structures from sparse LiDAR to initialize robust Gaussian anchors. Furthermore, we introduce Gaussian Anchor Fusion (GAF), which efficiently integrates multi-view image semantics via geometry-aligned 2D sampling and cross-modal alignment. By refining these compact Gaussian descriptors, Gau-Occ captures both spatial consistency and semantic discriminability. Extensive experiments across challenging benchmarks demonstrate that Gau-Occ achieves state-of-the-art performance with significant computational efficiency.
Chengxin Lv, Yihui Li, Hongyu Yang +1
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China