3D semantic occupancy prediction estimates voxel-wise occupancy and semantic labels throughout a scene. Our experiments reveal that conventional sparse transposed-convolution upsampling yields poorly differentiated voxel responses and that successive decoder layers discard previously correct semantic predictions. To address these issues, we present Feature-Diverse Upsampling and Memory-Aware Decoding for 3D Occupancy Prediction (FM-Occ), comprising two components. First, 3D Sparse Subvoxel Shuffle (3S) mitigates feature homogenization by constructing candidate child features in the channel dimension, jointly transforming and normalizing them, and then rearranging them onto a finer sparse grid with explicit feature-coordinate correspondence. Second, Entropy-Regulated Layer-wise Memory (ER-LM) retains semantic information across decoder layers by maintaining a layer-wise memory for each query through gated recurrent updates and using prediction entropy to regulate the historical information propagated to the next layer. During training, a class-weighted entropy loss Lcwe provides label-guided regularization for matched semantic queries. On SemanticKITTI, 3S increases the spatial coefficient of variation of normalized voxel-wise root-mean-square (RMS) responses from 0.41% to 4.03%, while ER-LM reduces semantic degradation and improves error correction across decoder layers. The combined FM-Occ improves mIoU from 12.16% to 12.73% with negligible memory overhead. ER-LM further improves mIoU in the BEV-based DISC framework from 16.78% to 17.17%, supporting its applicability across 3D voxel-based and BEV-based architectures.
Figures & tables
Figure 1: Two challenges in occupancy prediction: feature homogenization and fragile semantic consistency. Left: Spatial variation of voxel-wise feature responses after upsampling, measured by spatial CV (higher indicates greater variation). Right: Mean class-wise semantic degradation (mD; lower is better) without and with ER-LM + Lcwe , comparing (a) adjacent decoder layers and (b) intermediate layers with the final layer.
Figure 2: Overall framework of FM-Occ. 3S performs sparse 3D feature upsampling, while ER-LM propagates information across successive decoder stages under entropy-guided regulation. Lcwe is applied only during training.
Figure 3: 3S. The upper branch expands feature channels through a submanifold convolution, applies group normalization and ReLU, and reshapes the features into r3 child representations per parent voxel. The lower branch generates the corresponding fine-grid coordinates. The two branches combine to form the upsampled sparse tensor, illustrated here for r=2 .
Method
CV (%)
Ratio
Transposed conv.
0.41
1.00×
3S
4.03
9.83×
Table 4: Comparison of spatial variation in voxel-wise RMS responses
Figure 4: Feature-response comparison after upsampling. Full images are shown on the left, with enlarged regions on the right. Each response map is independently min–max normalized for visualization.
Table 6
mD ↓
mR ↑
mD ↓
mR ↑
Transition
–
Ours
–
Ours
Transition
–
Ours
–
Ours
1→2
17.86
11.60
0.99
0.88
1→9
21.00
16.50
1.16
1.43
2→3
8.60
8.84
0.64
0.99
2→9
15.85
12.12
0.84
1.50
3→4
11.06
6.34
0.57
0.72
3→9
13.01
9.08
0.73
1.33
4→5
10.10
6.45
0.55
0.99
4→9
12.56
9.92
0.77
1.39
5→6
9.77
5.11
0.51
0.73
5→9
11.40
5.95
0.71
1.03
Table 8: Class-balanced semantic transition rates (%). Lower mD and higher mR are better
Vision-based 3D semantic occupancy prediction is essential for autonomous driving, yet dense voxel representations waste computation on largely empty space, while BEV and TPV projections compromise fine-grained 3D structure. Fully sparse representations offer an attractive alternative, but existing methods, including SparseOcc, entangle scene completion with semantic prediction by indiscriminately propagating high-dimensional features into empty regions and applying voxel-wise classification. This creates excessive activations, computational overhead, and geometric ambiguity. We present SparseOcc++, a geometry-aware sparse framework that explicitly decouples scene completion from semantic segmentation. SparseOcc++ reformulates completion as signed-distance regression on sparse anchor voxels through a scene completion field (SCF). To model complex outdoor geometry robustly, it combines orthogonal decomposition with discretized distance learning. A geometry-guided propagation module then converts the SCF into a complete volumetric scene and restricts semantic segmentation to geometrically verified regions. Experiments establish new state of the art: SparseOcc++ improves IoU by 2.3 points and is 3.9x faster than SparseOcc on nuScenes, while achieving a 5.9x speedup over OccFormer on SemanticKITTI.
Pin Tang, Zhongdao Wang, Guoqing Wang +2
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Noah’s Ark Lab, China
3D semantic occupancy prediction requires accurate 2D-to-3D feature lifting, yet current methods restrict camera geometry to initial projections. Subsequent operations like offset learning, attention weighting, and cross-camera aggregation remain geometry-agnostic, ignoring essential physical constraints. We propose VGGT-Occ, a framework that embeds geometric tokens throughout the entire pipeline. We introduce Projection-Aware Deformable Attention (PA-DA) to inject geometry into all attention stages. PA-DA projects 3D offsets back to image planes and leverages the projection Jacobian as an additive bias to suppress unreliable observations. Features are then integrated through a view-quality semantic gate for cross-view consistency. To optimize both efficiency and performance, we employ a sequential coarse-to-fine decoder with gated fusion, where low-resolution features are refined into higher resolutions, allocating computation by information density while substantially reducing decoder cost. Extensive evaluations demonstrate the effectiveness and accuracy of our approach. On SurroundOcc-nuScenes, VGGT-Occ achieves 33.00% IoU and 21.08% mIoU (T=1), and 33.64% IoU and 21.43% mIoU with T=2 inference, outperforming existing methods, with only ∼41M trainable parameters in the occupancy head. Code will be released publicly.
Xun Chen, Tianchen Deng, Rui Wang +5
Nanyang Technological University · Shanghai Jiao Tong University · ETH Zurich
Predicting 3D occupancy from multi-view images requires preserving geometric detail during 2D-to-3D lifting while reasoning over sparse, volumetric scene representations. We present DualPathOcc, a camera-based framework that combines a Spatial Enhancer for high-resolution feature aggregation before BEV compression, a SENet-augmented dual-path BEV encoder for local-global context modeling, and height-aware weighted cross-entropy for near-ground occupancy. The final model is optimized with occupancy supervision and no explicit depth loss. On single-frame Occ3D-nuScenes, DualPathOcc achieves 37.37 mIoU. We further analyze how surface-centered depth targets interact with volumetric occupancy learning.
Lihao Qiu, Jian Chen, Ruihao Wang +3
Department of Electrical and Computer Engineering, University of Nevada, Las Vegas · Institute of Logistics Science and Engineering, Shanghai Maritime University · School of Software Engineering, Xi’an Jiaotong University