High-beam-count LiDAR sensors are costly, yet many perception pipelines require dense angular sampling. Using a pretrained Stable Diffusion model as the backbone, we fine-tune a LiDAR-conditioned depth model with pseudo-depth targets from a 2D foundation model. During training, the LiDAR conditioning is randomly decimated at different beam budgets. We then investigate how much of a LiDAR scan can be recovered from heavily decimated input and characterize performance across the input beam budget. We evaluate against physically held-out real beams on nuScenes and report recovery separately from fit accuracy. Our model yields its largest advantage in very sparse regimes, achieving a δ1.25 accuracy of 66.8% from 4-beam input where scattered interpolation reaches only 45.1%. A class-stratified error breakdown further reveals that planar surfaces recover first while objects introducing depth discontinuities degrade earliest. Together, these results quantify the recovery/resolution trade-off for foundation-model-guided LiDAR enhancement.
Figures & tables
Fig. 1: Partial view of the model output in point cloud space across 3 fused views (gray) given 8 input beams (red) compared to ground truth (yellow).
Budget
Method
AbsRel ↓
RMSE ↓
δ1.25↑
b=4
Nearest
0.692
12.62
0.451
Ours (no scene)
0.260
8.67
0.668
Ours (scene)
0.270
8.85
0.653
b=8
Nearest
0.311
8.14
0.670
Ours (no scene)
0.169
7.34
0.787
Ours (scene)
0.174
7.53
0.779
TABLE I: Held-out beam recovery vs. beam budget. “Nearest” applies nearest valid-pixel scattered interpolation. At b=32 no beams are held out. The fit is reported as a full-density reference.
Fig. 2: Recovery against input beam budget on nuScenes [ 21 ] validation split. Left (overall recovery): the proposed model outperforms nearest-neighbor interpolation at every budget, with performance largely plateauing beyond 16 beams. Right (recovery per semantic group): large-vehicle error (orange) falls steeply as additional beams constrain output geometry. Planar ground (green) shows marginal improvement.
Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.
Samed Doğan, Nico Leuze, Alfred Schöttl
Department of Electrical Engineering and Information Technology Munich University of Applied Sciences Munich, Bavaria 80335
Should a completion model spend extra test-time compute by iterating, or spend a similar parameter budget on a wider one-shot predictor? The answer is easily confounded by denoising curricula, corruption augmentation, capacity, and unpaired evaluation. We study this question in LiDAR semantic scene completion by comparing a one-shot predictor, a parameter-matched wider predictor, and a weight-tied multigrid refiner initialized from the same frozen predictor. The protocol separates coherent region removal, independent thinning, range-dependent attenuation, and additive clutter while preserving exact scene-condition pairing. Across five training seeds and 815 SemanticKITTI sequence-08 frames, the full iterative system improves mIoU over the wide control by 0.911 points under contiguous angular removal, with a 95% moving-block bootstrap interval of [0.804, 1.040] that clears a predeclared 0.5-point practical margin. Under independent 75% thinning, iteration adds only 0.300 points [0.166, 0.436], whereas observation-family augmentation adds 5.975 points [5.662, 6.140]. Neither intervention repairs additive clutter. The iterative system also costs 10.74 ms and 0.75 GiB per frame, versus 6.25 ms and 0.23 GiB for the wide control. These results establish a geometry-conditioned empirical boundary rather than a universal advantage: coherent gaps can justify fixed-depth refinement, broadly thinned evidence is addressed more effectively by training coverage, and spurious evidence requires a different robustness mechanism.
Shijie Hao, Weining Zhang
Phillips Exeter Academy · Cheung Kong Graduate School of Business
While accurate LiDAR depth has been shown to improve stereo matching, high-end LiDAR remains costly and difficult to deploy at scale, motivating guidance from sparse LiDAR measurements. In this paper, we revisit how extremely sparse LiDAR can guide iterative stereo by examining its cost-volume retrieval mechanism. Our analysis shows that LiDAR guidance degrades sharply when only a few hundred points are available, and we provide a signal-processing explanation for why iterative stereo fails to fully exploit such sparse input. This insight leads to a simple yet highly effective remedy: pre-filling the initial disparity map so that cost-volume retrieval becomes more reliable under extremely sparse LiDAR. We find that pre-filling is also effective when injecting LiDAR depth into image features via early fusion, but for a fundamentally different reason, necessitating a distinct pre-filling approach. Combining both forms of guidance, our proposed Guided RAFT-Stereo (GRAFT-Stereo) achieves substantial improvements over existing LiDAR-guided stereo methods across multiple datasets.