Organizations: College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA
Outdoor LiDAR semantic scene completion (SSC) recovers a dense semantic voxel grid from a scan observing 1% of the target volume, under class imbalance beyond 7,000x. We recast SSC as generative semantic scene completion (GSSC): a single discrete-diffusion formulation in three roles. First, paired sparse-dense scene synthesis (PS3) generates matched sparse LiDAR observations with their dense semantic completions, addressing the long tail at its source and yielding the PS3-SemanticKITTI corpus we train on alongside SemanticKITTI. Second, semantic-guided generative scene completion (SGSC) generates the scene from noise with multinomial discrete diffusion, conditioned on the sparse scan through a bird's-eye-view semantic map and a sparse 3D feature stream. Third, the same framework instead refines an existing completion in one flow-matching step: structured source discrete diffusion (S2D2). S2D2 improves the mIoU of SGSC's own output and every external SSC base tested, without base retraining or test-time adaptation. On the strongest base, one step without test-time augmentation reaches 38.8% mIoU on the SemanticKITTI hidden test. To our knowledge that is the best causal, single-sweep, single-sample result on that leaderboard, +2.1 pp over the previous best published score under the same restriction. Four correction steps with eight-view test-time augmentation reach 39.2%, outside that restriction.
Figures & tables
Fig. 1: The challenge of semantic scene completion. (a) SSC infers dense structure and semantics from a sparse sweep. (b) Five failure modes of discriminative methods; the percentages are measured on the validation predictions of SCPNet [ 1 ] . (c) SemanticKITTI is severely long-tailed, vegetation 26.7% of labelled points against motorcyclist 0.0037% . (d) Our class frequency (log axis) and SCPNet’s published hidden-test IoU correlate at Pearson r=0.80 .
Fig. 2: PS 3 : the offline data-augmentation pipeline. Three cascaded coarse-to-fine multinomial diffusions, screened by a Jensen–Shannon divergence filter; a rare-class object bank and the Hardware-Aware LiDAR Observation ( Halo ) HDL-64E ray-tracer then turn each synthetic scene into a (sparse, dense) pair. Calibration, thresholds and the pasting pseudocode are in Appendix A of the supplementary material.
Fig. 3: One PS 3 pair. Pyramid diffusion generates a scene ( S1→S2→S3 , a–c); the DJS -screened sample receives object-bank rare classes, giving Y (d; pasted objects in colour), which Halo ray-traces into the sparse sweep X (e).
Fig. 4: The SGSC denoiser in its generation role. A four-level dense 3D U-Net (channels 32→256 ) generates the scene from the uniform prior xT=u , conditioned on a BEV map B and per-voxel LSK3DNet features F from one sweep, injected additively at every level ( Eq. 3 ); the timestep enters only through adaptive group normalisation (AdaGN), and the bottleneck stacks two residual blocks (RB). The same architecture, retrained against a structured source, drives S 2 D 2 refinement ( Fig. 5 ).
Fig. 5: S 2 D 2 refinement: the shared denoiser’s second role. The velocity field carries the base prediction xsrc to ground truth x0 in one Euler step on the per-voxel simplex (top); the identity and error bound rule out the amplification that would force the full chain (middle; Prop. 3, Thm. 1 and Cor. 1 of Appendix B); a motorcyclist scene, frozen base vs. base + S 2 D 2 at N=1 , one denoiser evaluation (bottom).
Method
Eval
mIoU
Comp
SC-m
VRU
LMSCNet [ 11 ]
test
17.6
56.7
12.4
0.0
SSA-SC [ 14 ]
test
23.5
58.8
17.8
2.6
JS3C-Net [ 15 ]
test
23.8
56.6
18.7
4.5
DiffSSC [ 32 ] ¶
test
27.4
63.4
n/a
n/a
S3CNet [ 13 ]
test
29.5
45.6
29.9
32.6
SCPNet [ 1 ] (published)
test
36.7
56.1
32.5
18.2
TABLE I: Main results on SemanticKITTI. Published baselines at their own test leaderboard; our rows on both splits. S3CNet, not ours, takes the vulnerable road user (VRU) column ( Section IV-E ).
Class
Base IoU
Released Δ
Retrain Δ
car
50.3
+1.1
+1.0
bicycle
23.7
+0.6
−0.1
motorcycle
35.6
−0.1
+0.2
truck
54.7
+5.3
+5.4
other-vehicle
42.0
+2.5
+2.3
person
22.0
+1.2
+1.9
TABLE II: Per-class val IoU against the frozen base (seq. 08 , N=1 ). “Released”: shipped checkpoint ( 38.54% ); “Retrain”: from-scratch ( 38.05% ).
Fig. 6: Qualitative comparison on val seq. 08. Three frozen sources (JS3C-Net, SCPNet, TALoS’s adaptation of SCPNet), our S 2 D 2 on the SCPNet base, and ground truth; top bicyclist, bottom motorcyclist. Chips give per-sample rare-class IoU and TP/FP voxel counts (from the N=4 , +D4 -TTA configuration).
Fig. 7: Our completions on val seq. 08 (rows: frames 004025, 001850). From a ∼1% -occupied sweep (a) and the base-derived BEV (b), one correction step recovers dense semantics (c) close to ground truth (d).
Configuration
mIoU
Comp
Δ
SC-m
VRU
LMSCNet ♭
14.8
46.7
–
10.4
0.0
+ S 2 D 2
16.6
46.7
+1.8
12.1
0.3
JS3C-Net ∗♭
22.7
53.1
–
18.6
3.8
+ S 2 D 2 †
24.3
50.9
+1.6
20.0
2.4
SCPNet ∗
36.17
49.9
–
32.5
14.7
+ S 2 D 2
38.54
52.7
+2.36
35.2
19.6
TABLE III: One denoiser architecture on three frozen bases , each trained on its own base’s output and refined by a single N=1 pass. Val seq. 08 ; Δ is the mIoU uplift over the paired base.
S 2 D 2 components ( 100 -frame probe, Δ vs. frozen base):
no BEV stream
n/a
+1.1
TABLE IV: Ablations. Most rows remove or replace a single element of the headline recipe; where a row differs in more than one, the text says so. Each block names its own reference and probe.
Fig. 8: The generator inherits the quality of its guidance. Ground-truth scene (left) and two 2562×32 SGSC completions (val seq. 08, frame 6 ) from one development checkpoint (from-noise T=100 , identical weights, only the BEV conditioning differs: oracle vs. predicted). A predicted BEV keeps the road surface but collapses the road users.
Fig. 9: PS 3 rebalances the long tail. Per-class voxel frequency (log scale, sorted ascending by real-only frequency), real-only vs. real + synth-32K. The annotated factors are the synthetic pool’s own frequency against real-only ( 4.6× – 6,239× ); the pooled bars gain 3.2× – 3,907× .
Fig. 10: Six rare-class val scenes , one per class, in descending order of D4 -TTA-vs-base IoU gain ( +40.1 to +20.1 ). Left : SCPNet base; right : S 2 D 2 ( 32 K multi-frame +D4 TTA), sharing one camera so differences are the predictions, not the framing. These are the scenes where the correction helps most; the regime where it hurts , a weaker base whose source under-supplies rare classes, is Fig. 12 .
Fig. 11: Qualitative BEV semantic predictions on SemanticKITTI validation. Each panel is a different val seq-08 frame, not another view of one scene; within a panel, left to right: the sparse LiDAR input, the ground-truth BEV, and our predicted BEV. Colour key (this BEV renderer’s own palette, not the voxel key): grey : road; dark purple : sidewalk; magenta : parking; cyan : building; dark green : vegetation; light green : terrain.
Fig. 12: Failure case: S 2 D 2 erases pedestrians on the JS3C-Net base (val seq. 08, frame 001810 ). Person and bicyclist voxels keep their semantic colour; the rest of the scene is muted to grey. Both pedestrians present in the ground truth (c) are recovered by the frozen base (a) and all but eliminated in the refined output (b), whose person IoU falls to 0.7% .