Sparsity-Aware Voxel Attention and Foreground Modulation for 3D Semantic Scene Completion

Yu Xue; Longjun Gao; Yuanqi Su; HaoAng Lu; Xiaoning Zhang

cs.CVApr 7, 2026

Sparsity-Aware Voxel Attention and Foreground Modulation for 3D Semantic Scene Completion

Authors: Yu Xue, Longjun Gao, Yuanqi Su, HaoAng Lu, Xiaoning Zhang

Organizations: Xi’an Jiaotong University, China

Abstract

Monocular Semantic Scene Completion (SSC) aims to reconstruct complete 3D semantic scenes from a single RGB image, offering a cost-effective solution for autonomous driving and robotics. However, the inherently imbalanced nature of voxel distributions, where over 93% of voxels are empty and foreground classes are rare, poses significant challenges. Existing methods often suffer from redundant emphasis on uninformative voxels and poor generalization to long-tailed categories. To address these issues, we propose VoxSAMNet (Voxel Sparsity-Aware Modulation Network), a unified framework that explicitly models voxel sparsity and semantic imbalance. Our approach introduces: (1) a Dummy Shortcut for Feature Refinement (DSFR) module that bypasses empty voxels via a shared dummy node while refining occupied ones with deformable attention; and (2) a Foreground Modulation Strategy combining Foreground Dropout (FD) and Text-Guided Image Filter (TGIF) to alleviate overfitting and enhance class-relevant features. Extensive experiments on the public benchmarks SemanticKITTI and SSCBench-KITTI-360 demonstrate that VoxSAMNet achieves state-of-the-art performance, surpassing prior monocular and stereo baselines with mIoU scores of 18.2% and 20.2%, respectively. Our results highlight the importance of sparsity-aware and semantics-guided design for efficient and accurate 3D scene completion, offering a promising direction for future research.

Figures & tables

Figure 1: Motivation for VoxSAMNet. (a) Current methods such as BEVFormer [ 17 ] and Et-former [ 19 ] use deformable attention to update features for both empty and occupied voxels. (b) The ratio of occupied and empty voxels in the SemanticKITTI [ 3 ] dataset. (c) We designed a branch that uses two different feature update methods for empty voxels and occupied voxels.

Figure 2: Overview of VoxSAMNet and the structure of its modules. (a) The pipeline flow of the proposed VoxSAMNet. (b) The specific structure of the Text-Guided Image Filter. (c) The specific structure of Dummy Shortcut for Feature Refinement.

Figure 3: Result of VoxSAMNet. Qualitative visual comparisons with Monoscene [ 4 ] , Symphonies [ 10 ] , VisHall3D [ 29 ] , and Ground Truth (GT) on the SemanticKITTI [ 3 ] dataset.

Method	Date	IoU $\uparrow$	mIoU $\uparrow$	road [-0.2ex]	sidewalk [-0.2ex]	parking [-0.2ex]	other-grnd. [-0.2ex]	building [-0.2ex]	car [-0.2ex]	truck [-0.2ex]	bicycle [-0.2ex]	motorcycle [-0.2ex]	other-veh. [-0.2ex]	vegetation [-0.2ex]	trunk [-0.2ex]	terrain [-0.2ex]	person [-0.2ex]	bicyclist [-0.2ex]	motorcyclist [-0.2ex]	fence [-0.2ex]	pole [-0.2ex]	traf.-sign [-0.2ex]
Stereo / multi-view methods
ScanSSC [ 2 ]	CVPR2025	44.54	17.40	66.2	35.9	35.1	12.5	25.3	27.1	3.5	3.5	3.2	6.1	25.2	11.0	30.6	1.8	5.3	0.7	20.5	8.4	8.9
VLScene [ 36 ]	AAAI2025	45.14	17.52	64.7	34.7	32.4	13.1	27.3	26.1	6.5	4.2	3.8	8.3	26.4	10.0	29.4	2.8	5.1	0.9	20.0	8.9	8.4
Temporal methods
SOAP [ 11 ]	CVPR2025	46.09	19.09	63.6	36.2	36.8	17.2	28.7	28.9	3.3	5.0	5.3	7.0	29.8	15.1	32.3	3.7	2.3	0.2	22.5	11.7	13.1
FlowScene [ 37 ]	NIPS2025	45.20	17.70	64.1	35.0	33.7	13.0	27.7	26.4	10.0	4.2	3.1	7.0	26.3	10.0	30.2	3.1	5.1	1.1	22.2	8.9	9.1

Table 1: Quantitative results on the SemanticKITTI hidden test set. Bold denotes the best performance.

Method	Date	IoU $\uparrow$	mIoU $\uparrow$	car [-0.2ex]	bicycle [-0.2ex]	motorcycle [-0.2ex]	truck [-0.2ex]	other-veh. [-0.2ex]	person [-0.2ex]	road [-0.2ex]	parking [-0.2ex]	sidewalk [-0.2ex]	other-ground. [-0.2ex]	building [-0.2ex]	fence [-0.2ex]	vegetation [-0.2ex]	terrain [-0.2ex]	pole [-0.2ex]	traf.-sign [-0.2ex]	other-struct. [-0.2ex]	other-object. [-0.2ex]
Single-frame methods
MonoScene [ 4 ]	CVPR2023	37.87	12.31	19.3	0.4	0.6	8.0	2.0	0.9	48.4	11.4	28.1	3.3	32.9	3.5	26.2	16.8	6.9	5.7	4.2	3.1
OccFormer [ 54 ]	ICCV2023	40.27	13.81	22.6	0.7	0.3	9.9	3.8	2.8	54.3	13.4	31.5	3.6	36.4	4.8	31.0	19.5	7.8	8.5	7.0	4.6
VoxFormer [ 16 ]	CVPR2023	38.76	11.91	17.8	1.2	0.9	4.6	2.1	1.6	47.0	9.7	27.2	2.9	31.4	5.0	29.0	14.7	6.5	6.9	3.8	2.4
Symphonies [ 10 ]	CVPR2024	44.12	18.58	30.0	1.9	5.9	25.1	12.1	8.2	54.9	13.8	32.8	6.9	35.1	8.6	38.3	11.5	14.0	9.6	14.4	11.3
MixSSC [ 35 ]	TCSVT2025	45.33	17.15	26.3	3.6	4.5	14.8	5.4	5.8	57.9	13.6	35.2	3.6	37.9	7.0	35.2	22.1	11.4	12.8	6.56	5.0

Table 2: Quantitative results on the SSCBench-KITTI-360 test set. Bold denotes the best performance.

Method	TGIF	FD	DSFR	IoU(%)	mIoU(%)	Params(M)
Baseline				45.38	15.84	116.30M
1	✓			45.96	16.27	130.52M
2	✓	✓		46.91	16.66	130.52M
3			✓	46.65	17.42	129.48M
4	✓		✓	47.12	17.86	143.58M
5	✓	✓	✓	47.88	18.19	143.58M

Table 3: Ablation study of the architectural components on the test set of SemanticKITTI.

Backbone	TGIF	IoU(%)	mIoU(%)
ResNet50		44.94	15.67
ResNet50	✓	45.75	15.98
Swin Transformer		45.38	15.84
Swin Transformer	✓	45.96	16.27

Table 4: Backbone ablation on the SemanticKITTI test set.

Refine Strategy	Total	Deformable Attention	Ratio (%)
Dense	30.96	24.512	79.2
Sparsity-Aware	8.694	0.377	4.3
Reduction	71.9% $\downarrow$	98.5% $\downarrow$	-

Table 5: Computational cost of DSFR. Unit: FLOPs (G).

3D Scene Completion

3D scene completion infers missing geometry, occupancy, semantics, and appearance from partial images, point clouds, or depth scans.

See all 41 papers

Aug 9, 2026

RayLift: Lifting Complementary Ray-Wise Evidence with 3D Geometry Priors for Semantic Scene Completion

Camera-based 3D semantic scene completion (SSC) provides comprehensive scene understanding for autonomous driving and robotics. However, existing methods often treat stereo depth estimates as deterministic geometric constraints, causing depth uncertainty and local correspondence errors to propagate directly into voxel representations. To address this issue, we propose RayLift, a framework that uses stereo geometry as a metric reference while incorporating complementary ray evidence to recover reliable 3D structures adaptively. RayLift first employs a Complementary Context Encoder that extracts geometry-aware priors from a frozen 3D vision foundation model, thereby enriching the scene context. It then introduces a Depth Ray Evidence Lifter module that jointly models geometric dissimilarity, depth confidence, and spatial uncertainty to adaptively sample and weight candidate surface locations along each camera ray. Finally, a Semantic-Aware Voxel Integrator injects the resulting ray evidence into voxel features by explicitly modeling their spatial support. Extensive experiments on SemanticKITTI and SSCBench-KITTI-360 demonstrate that RayLift achieves competitive performance and consistently outperforms existing methods.

Meng Wang, Hongxia Yu, Wenzhe He +4

College of Computer Science and Electronic Engineering,Hunan University,Changsha,China

3D Semantic Segmentation 3D Scene Completion 3D Scene Understanding

Ask PDF

Aug 27, 2026

Generative Semantic Scene Completion

Outdoor LiDAR semantic scene completion (SSC) recovers a dense semantic voxel grid from a scan observing 1% of the target volume, under class imbalance beyond 7,000x. We recast SSC as generative semantic scene completion (GSSC): a single discrete-diffusion formulation in three roles. First, paired sparse-dense scene synthesis (PS

^3

) generates matched sparse LiDAR observations with their dense semantic completions, addressing the long tail at its source and yielding the PS

^3

-SemanticKITTI corpus we train on alongside SemanticKITTI. Second, semantic-guided generative scene completion (SGSC) generates the scene from noise with multinomial discrete diffusion, conditioned on the sparse scan through a bird's-eye-view semantic map and a sparse 3D feature stream. Third, the same framework instead refines an existing completion in one flow-matching step: structured source discrete diffusion (S

^2

^2

). S

^2

^2

improves the mIoU of SGSC's own output and every external SSC base tested, without base retraining or test-time adaptation. On the strongest base, one step without test-time augmentation reaches 38.8% mIoU on the SemanticKITTI hidden test. To our knowledge that is the best causal, single-sweep, single-sample result on that leaderboard, +2.1 pp over the previous best published score under the same restriction. Four correction steps with eight-view test-time augmentation reach 39.2%, outside that restriction.

Shi Chen, Weifeng Ge

College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA

3D Scene Completion 3D Scene Understanding Diffusion Models

Ask PDF

Mar 23, 2026

GaussianSSC: Triplane-Guided Directional Gaussian Fields for 3D Semantic Completion

We present GaussianSSC, a two-stage, grid-native and triplane-guided approach to semantic scene completion (SSC) that injects the benefits of Gaussians without replacing the voxel grid or maintaining a separate Gaussian set. We introduce Gaussian Anchoring, a sub-pixel, Gaussian-weighted image aggregation over fused FPN features that tightens voxel--image alignment and improves monocular occupancy estimation. We further convert point-like voxel features into a learned per-voxel Gaussian field and refine triplane features via a triplane-aligned Gaussian--Triplane Refinement module that combines local gathering (target-centric) and global aggregation (source-centric). This directional, anisotropic support captures surface tangency, scale, and occlusion-aware asymmetry while preserving the efficiency of triplane representations. On SemanticKITTI~\cite{behley2019semantickitti}, GaussianSSC improves Stage~~1 occupancy by +1.0% Recall, +2.0% Precision, and +1.8% IoU over state-of-the-art baselines, and improves Stage~~2 semantic prediction by +1.8% IoU and +0.8% mIoU.

Ruiqi Xian, Jing Liang, He Yin +2

3D Scene Completion 3D Scene Understanding 3D Occupancy Prediction 3D Scene Representation

Ask PDF

Abstract

3D Scene Completion

Explore similar work

RayLift: Lifting Complementary Ray-Wise Evidence with 3D Geometry Priors for Semantic Scene Completion

Generative Semantic Scene Completion

GaussianSSC: Triplane-Guided Directional Gaussian Fields for 3D Semantic Completion