cs.CVNov 5, 2026

Extrapolation Matters: Sparse-Only Depth Completion Beyond LiDAR Coverage

Authors: Maximilian Jansen, Gasser Elazab, Praveen Kulkarni

Organizations: CARIAD SE · Bosch

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Extrapolation beyond LiDAR coverage is an important yet under-evaluated challenge of depth completion. In real camera--LiDAR systems, partial sensor overlap, occlusions, and limited field coverage often leave large image regions weakly observed or completely unobserved by LiDAR. Thus, we investigate depth completion in this beyond-coverage regime using only sparse supervision from raw single-sweep LiDAR, without dense depth labels, LiDAR accumulation, or pseudo-LiDAR generation. Our approach leverages a frozen monocular depth foundation model, lightweight per-frame alignment to sparse LiDAR geometry, and a refinement module that fuses the aligned prior with sparse LiDAR anchors to predict dense metric depth. To regularize learning under raw sparse supervision, we introduce a high-frequency normal-consistency loss defined as the image-space gradient of the per-pixel normal cosine discrepancy. This term penalizes spatial variation in scalar angular error; it does not directly match changes in neighboring normal vectors. We further introduce an evaluation protocol that explicitly separates LiDAR-supported and beyond-coverage regions, revealing extrapolative behavior that conventional depth-completion metrics can obscure. Our method achieves strong depth accuracy beyond LiDAR coverage while remaining competitive in LiDAR-covered regions, providing a practical way to deploy monocular foundation priors in multi-sensor systems.

Figures & tables

Figure 2: Architecture overview. A frozen monocular depth foundation model predicts a dense, scale-ambiguous depth prior from the input RGB image. After per-frame alignment to raw sparse LiDAR, the aligned prior and sparse LiDAR depth are fused to predict dense metric depth, including beyond LiDAR coverage. In our implementation, we use MoGe [ 28 ] as the foundation model and DySPN [ 14 ] as the fusion module.
Figure 3: KITTI beyond-coverage protocol. Left: cropped raw Velodyne depth used as input and sparse supervision during training. Right: complementary region used for evaluation only. The two regions are spatially disjoint, so performance in the evaluation region reflects extrapolation beyond direct LiDAR support.
 KITTI Extrap.KITTI
MethodRMSE ↓\downarrowAbs. Rel. ↓\downarrowRMSE ↓\downarrowAbs. Rel. ↓\downarrow
 (mm)(%)(mm)(%)
OMNI-DC †\dagger [ 37 ]397312.012241.5
DMD3C †\dagger [ 13 ]1122021.113292.2
DMD3C [ 13 ]731412.421354.2
DySPN1665065.71609264.6
Table 1: Results on standard KITTI depth completion and on the controlled beyond-coverage extrapolation protocol. The bottom sections isolate the effect of our fusion strategy against prior-only baselines. †\dagger denotes models pretrained on other datasets but evaluated zero-shot on KITTI. “Aligned” indicates per-frame metric alignment of the frozen monocular prior to sparse LiDAR anchors without our fusion refinement.
Figure 4: Qualitative results on the KITTI beyond-coverage protocol. We show the RGB input with the cropped Velodyne band (sparse supervision), followed by DMD3C and our predictions. Our approach yields more coherent surfaces, preserves thin structures, and reconstructs plausible geometry beyond direct LiDAR support.
MethodAbs Rel ↓\downarrowRMSE ↓\downarrowδ1↑\delta_{1}\uparrowδ2↑\delta_{2}\uparrowδ3↑\delta_{3}\uparrow
  (mm)   
DMD3C [ 13 ]0.02811410.9840.9960.998
Ours0.03011660.9470.9810.991
Table 2: Comparison on NYUv2 [ 23 ] with 500 sparse depth samples as input. DMD3C is trained with full dense ground-truth depth supervision, whereas our framework uses only the 500 sparse samples as supervision.
Figure 5: Qualitative results on KITTI-DC [ 26 ] . The first row shows the full raw Velodyne input provided to both the pretrained DMD3C in the second row and our method in the third row. The proposed framework produces more geometrically coherent reconstructions while reducing localized artifacts.
 KITTIKITTI Extrap.
VariantRMSE ↓\downarrowAbs. Rel. ↓\downarrowRMSE ↓\downarrowAbs. Rel. ↓\downarrow
 (mm)(%)(mm)(%)
w/o normal consist.19922.430136.9
w/o HF norm. consist.16492.429496.2
w/o RSA17822.529386.1
w/o depth input47393.3880716.9
Table 3: Component ablations on standard KITTI and on the beyond-coverage protocol. The w/o depth input variant removes the aligned dense prior from the fusion network, while w/o completion corresponds to the aligned monocular prior without refinement. The extrapolation metrics are evaluated only on the complementary beyond-coverage region, which explains why removing RSA can slightly improve this mask-specific score despite hurting performance on the full semi-dense validation set.
Figure 6: Qualitative robustness analysis on a challenging nuScenes scene. Left : input RGB image with reflective wet-road surfaces and large weakly textured regions. Center : point cloud reconstructed from the monocular prior together with raw LiDAR, showing both large-scale prior inconsistencies and local LiDAR outliers. Right : point cloud produced by the proposed framework together with filtered LiDAR points. The fused reconstruction follows sparse LiDAR geometry at large scales while suppressing short-scale artifacts, including below-ground reflections on the road surface.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

SettingAbs. Rel. ↓\downarrowRMSE (mm) ↓\downarrow
Rand., 5k pts, L1L_{1} only0.05202502
LiDAR-like, 16 beams0.05171920
LiDAR-like, 64 beams0.02141022
Rand., 500 pts0.03031166
Rand., 5k pts0.0270900
Appendix
Table 4: NYUv2 results under sparse-only supervision for different sparsity patterns and sampling densities. The first row replaces the geometric regularization with direct L1L_{1} depth supervision only. The remaining rows use the proposed formulation without pruning the L1L_{1} loss, which is not needed in this noise-free sparsification setting. Performance improves with denser sampling, while the LiDAR-like patterns show that the framework remains effective even under persistent structured coverage gaps.
Figure 7: Effect of the robust sparse anchor (RSA) loss under synthetic sparse outlier corruption. Top : RGB overlay with injected large-depth outliers highlighted by a bounding box. Bottom left : model trained without RSA; the corrupted sparse points are propagated into the reconstruction and distort the local geometry. Bottom right : full model with RSA; the same outliers are rejected and the local surface remains coherent. RSA trims the largest sparse depth residuals during training, encouraging the network to ignore spurious LiDAR measurements.
More in this topic

Depth Completion

Depth completion estimates dense, metrically grounded scene geometry from sparse depth measurements, often using aligned images and sensor data.

See all 29 papers

Explore similar work

CardsList
May 28, 2026

Large Depth Completion Model from Sparse Observations

This work presents the Large Depth Completion Model (LDCM), a simple, effective, and robust framework for single-view metric depth estimation with sparse observations. Without relying on complex architectural designs, LDCM generates metric-accurate dense depth maps using a transformer. It outperforms existing approaches across diverse datasets and sparse observations. We achieve this from two key perspectives: (1) leveraging existing monocular foundation models to improve the quality of sparse depth inputs, and (2) reformulating training objectives to better capture geometric structure and metric consistency. Specifically, a Poisson-based depth initialization strategy is first introduced to generate a uniform coarse dense depth map from diverse sparse observations, providing a strong structural prior for the network. Regarding the training objective, we replace the conventional depth head with a point map head that regresses per-pixel 3D coordinates in camera space, enabling the model to directly learn the underlying 3D scene structure instead of performing pixel-wise depth map restoration. Moreover, this design eliminates the need for camera intrinsic parameters, allowing LDCM to naturally produce metric-scaled 3D point maps. Extensive experiments demonstrate that LDCM consistently outperforms state-of-the-art methods across multiple benchmarks and varying sparsity levels in both depth completion and point map estimation, showcasing its effectiveness and strong generalization to unseen data distributions.

Zhu Yu, Zhengyi Zhao, Runmin Zhang +7

Zhejiang University · Internship at Tongyi Lab · Tongyi Lab, Alibaba Group +4

Metric Depth EstimationDepth EstimationDepth CompletionMonocular Depth Estimation
AskPDF
Mar 16, 2026

AURORA-KITTI: Any-Weather Depth Completion and Denoising

Robust depth completion is fundamental to real-world 3D scene understanding, yet existing RGB--LiDAR fusion methods degrade significantly under adverse weather, where both camera images and LiDAR measurements suffer from weather-induced corruption. In this paper, we introduce AURORA-KITTI, the first large-scale, physics-consistent synthetic RGB-LiDAR benchmark for any-weather depth completion and denoising. We further formulate Depth Completion and Denoising~(DCD) as a unified task that reconstructs clean dense metric depth from weather-corrupted RGB images and sparse LiDAR inputs while suppressing weather-induced noise. AURORA-KITTI contains over 82K weather-consistent RGB--LiDAR pairs with metric depth supervision, spanning diverse weather types, three severity levels, day/night scenes, paired clean references, lens occlusion conditions, and textual descriptions.Moreover, we provide DDCD as a strong and efficient distillation-based reference baseline. DDCD is trained on AURORA-KITTI using a frozen teacher applied to paired clean RGB images only during training; at inference, it uses weather-corrupted RGB and sparse LiDAR. Experiments on AURORA-KITTI and the real-world DENSE benchmark show that existing depth completion and foundation depth models remain vulnerable under severe weather, while DDCD improves over weather-aware retraining. Controlled experiments further show that physics-consistent adverse-weather data is the dominant driver of robustness, with clean-prior distillation providing complementary gains. Comparisons with off-the-shelf zero-shot methods use different training protocols; same-data retraining and component ablations separately quantify the contributions of weather-aware data and DDCD. Github: https://github.com/Warwick-Jocelyn/AURORA-KITTI.

Yiting Wang, Tim Brödermann, Hamed Haghighi +4

Warwick Manufacturing Group, University of Warwick, Coventry, CV4 7AL, United Kingdom · School of Engineering and Materials Science, Queen Mary University of London, London, E1 4NS, United Kingdom · Computer Vision Laboratory, ETH Zürich, Zurich, 8057, Switzerland +1

Depth CompletionMultimodal RobustnessCamera-LiDAR FusionImage Denoising
AskPDF
Apr 20, 2026

EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion

Depth completion from sparse LiDAR measurements and corresponding RGB images is a prerequisite for accurate 3D perception in robotic systems. Existing methods achieve high accuracy on standard benchmarks but rely on heavy backbone architectures that preclude real-time deployment on embedded hardware. We present EfficientPENet, a two-branch depth completion network that replaces the conventional ResNet encoder with a modernized ConvNeXt backbone, introduces sparsity-invariant convolutions for the depth stream, and refines predictions through a Convolutional Spatial Propagation Network (CSPN). The RGB branch leverages ImageNet-pretrained ConvNeXt blocks with Layer Normalization, 7x7 depthwise convolutions, and stochastic depth regularization. Features from both branches are merged via late fusion and decoded through a multi-scale deep supervision strategy. We further introduce a position-aware test-time augmentation scheme that corrects coordinate tensors during horizontal flipping, yielding consistent error reduction at inference. On the KITTI depth completion benchmark, EfficientPENet achieves an RMSE of 631.94 mm with 36.24M parameters and a latency of 20.51 ms, operating at 48.76 FPS. This represents a 3.7 times reduction in parameters and a 23 times speedup relative to BP-Net, while maintaining competitive accuracy. These results establish EfficientPENet as a practical solution for real-time depth completion on resource-constrained edge platforms such as the NVIDIA Jetson.

Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi +4

Canizaro Livingston Gulf States Center for Environmental Informatics, the University of New Orleans, New Orleans, USA · US Army Corps of Engineers, Engineer Research and Development Center, Vicksburg, Mississippi, USA

Efficient Neural Network InferenceDepth EstimationDepth CompletionCamera-LiDAR FusionConvolutional Neural Networks
AskPDF