Extrapolation Matters: Sparse-Only Depth Completion Beyond LiDAR Coverage

Maximilian Jansen; Gasser Elazab; Praveen Kulkarni

cs.CVNov 5, 2026

Extrapolation Matters: Sparse-Only Depth Completion Beyond LiDAR Coverage

Authors: Maximilian Jansen, Gasser Elazab, Praveen Kulkarni

Organizations: CARIAD SE · Bosch

Abstract

Extrapolation beyond LiDAR coverage is an important yet under-evaluated challenge of depth completion. In real camera--LiDAR systems, partial sensor overlap, occlusions, and limited field coverage often leave large image regions weakly observed or completely unobserved by LiDAR. Thus, we investigate depth completion in this beyond-coverage regime using only sparse supervision from raw single-sweep LiDAR, without dense depth labels, LiDAR accumulation, or pseudo-LiDAR generation. Our approach leverages a frozen monocular depth foundation model, lightweight per-frame alignment to sparse LiDAR geometry, and a refinement module that fuses the aligned prior with sparse LiDAR anchors to predict dense metric depth. To regularize learning under raw sparse supervision, we introduce a high-frequency normal-consistency loss defined as the image-space gradient of the per-pixel normal cosine discrepancy. This term penalizes spatial variation in scalar angular error; it does not directly match changes in neighboring normal vectors. We further introduce an evaluation protocol that explicitly separates LiDAR-supported and beyond-coverage regions, revealing extrapolative behavior that conventional depth-completion metrics can obscure. Our method achieves strong depth accuracy beyond LiDAR coverage while remaining competitive in LiDAR-covered regions, providing a practical way to deploy monocular foundation priors in multi-sensor systems.

Figures & tables

Figure 2: Architecture overview. A frozen monocular depth foundation model predicts a dense, scale-ambiguous depth prior from the input RGB image. After per-frame alignment to raw sparse LiDAR, the aligned prior and sparse LiDAR depth are fused to predict dense metric depth, including beyond LiDAR coverage. In our implementation, we use MoGe [ 28 ] as the foundation model and DySPN [ 14 ] as the fusion module.

Figure 3: KITTI beyond-coverage protocol. Left: cropped raw Velodyne depth used as input and sparse supervision during training. Right: complementary region used for evaluation only. The two regions are spatially disjoint, so performance in the evaluation region reflects extrapolation beyond direct LiDAR support.

	KITTI Extrap.		KITTI
Method	RMSE $\downarrow$	Abs. Rel. $\downarrow$	RMSE $\downarrow$	Abs. Rel. $\downarrow$
	(mm)	(%)	(mm)	(%)
OMNI-DC $\dagger$ [ 37 ]	3973	12.0	1224	1.5
DMD3C $\dagger$ [ 13 ]	11220	21.1	1329	2.2
DMD3C [ 13 ]	7314	12.4	2135	4.2
DySPN	16650	65.7	16092	64.6

Table 1: Results on standard KITTI depth completion and on the controlled beyond-coverage extrapolation protocol. The bottom sections isolate the effect of our fusion strategy against prior-only baselines.

\dagger

denotes models pretrained on other datasets but evaluated zero-shot on KITTI. “Aligned” indicates per-frame metric alignment of the frozen monocular prior to sparse LiDAR anchors without our fusion refinement.

Figure 4: Qualitative results on the KITTI beyond-coverage protocol. We show the RGB input with the cropped Velodyne band (sparse supervision), followed by DMD3C and our predictions. Our approach yields more coherent surfaces, preserves thin structures, and reconstructs plausible geometry beyond direct LiDAR support.

Method	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
		(mm)
DMD3C [ 13 ]	0.028	1141	0.984	0.996	0.998
Ours	0.030	1166	0.947	0.981	0.991

Table 2: Comparison on NYUv2 [ 23 ] with 500 sparse depth samples as input. DMD3C is trained with full dense ground-truth depth supervision, whereas our framework uses only the 500 sparse samples as supervision.

Figure 5: Qualitative results on KITTI-DC [ 26 ] . The first row shows the full raw Velodyne input provided to both the pretrained DMD3C in the second row and our method in the third row. The proposed framework produces more geometrically coherent reconstructions while reducing localized artifacts.

	KITTI		KITTI Extrap.
Variant	RMSE $\downarrow$	Abs. Rel. $\downarrow$	RMSE $\downarrow$	Abs. Rel. $\downarrow$
	(mm)	(%)	(mm)	(%)
w/o normal consist.	1992	2.4	3013	6.9
w/o HF norm. consist.	1649	2.4	2949	6.2
w/o RSA	1782	2.5	2938	6.1
w/o depth input	4739	3.3	8807	16.9

Table 3: Component ablations on standard KITTI and on the beyond-coverage protocol. The w/o depth input variant removes the aligned dense prior from the fusion network, while w/o completion corresponds to the aligned monocular prior without refinement. The extrapolation metrics are evaluated only on the complementary beyond-coverage region, which explains why removing RSA can slightly improve this mask-specific score despite hurting performance on the full semi-dense validation set.

Figure 6: Qualitative robustness analysis on a challenging nuScenes scene. Left : input RGB image with reflective wet-road surfaces and large weakly textured regions. Center : point cloud reconstructed from the monocular prior together with raw LiDAR, showing both large-scale prior inconsistencies and local LiDAR outliers. Right : point cloud produced by the proposed framework together with filtered LiDAR points. The fused reconstruction follows sparse LiDAR geometry at large scales while suppressing short-scale artifacts, including below-ground reflections on the road surface.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Abs. Rel. $\downarrow$	RMSE (mm) $\downarrow$
Rand., 5k pts, $L_{1}$ only	0.0520	2502
LiDAR-like, 16 beams	0.0517	1920
LiDAR-like, 64 beams	0.0214	1022
Rand., 500 pts	0.0303	1166
Rand., 5k pts	0.0270	900

Appendix

Table 4: NYUv2 results under sparse-only supervision for different sparsity patterns and sampling densities. The first row replaces the geometric regularization with direct

L_{1}

depth supervision only. The remaining rows use the proposed formulation without pruning the

L_{1}

loss, which is not needed in this noise-free sparsification setting. Performance improves with denser sampling, while the LiDAR-like patterns show that the framework remains effective even under persistent structured coverage gaps.

Figure 7: Effect of the robust sparse anchor (RSA) loss under synthetic sparse outlier corruption. Top : RGB overlay with injected large-depth outliers highlighted by a bounding box. Bottom left : model trained without RSA; the corrupted sparse points are propagated into the reconstruction and distort the local geometry. Bottom right : full model with RSA; the same outliers are rejected and the local surface remains coherent. RSA trims the largest sparse depth residuals during training, encouraging the network to ignore spurious LiDAR measurements.

Depth Completion

Depth completion estimates dense, metrically grounded scene geometry from sparse depth measurements, often using aligned images and sensor data.

See all 29 papers

Large Depth Completion Model from Sparse Observations
May 28, 2026Zhu Yu, Zhengyi Zhao, Runmin Zhang +7Metric Depth Estimation Depth Estimation
Zhejiang University · Internship at Tongyi Lab · Tongyi Lab, Alibaba Group +4
AURORA-KITTI: Any-Weather Depth Completion and Denoising
Mar 16, 2026Yiting Wang, Tim Brödermann, Hamed Haghighi +4Depth Completion Multimodal Robustness
Warwick Manufacturing Group, University of Warwick, Coventry, CV4 7AL, United Kingdom · School of Engineering and Materials Science, Queen Mary University of London, London, E1 4NS, United Kingdom · Computer Vision Laboratory, ETH Zürich, Zurich, 8057, Switzerland +1
EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion
Apr 20, 2026Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi +4Efficient Neural Network Inference Depth Estimation
Canizaro Livingston Gulf States Center for Environmental Informatics, the University of New Orleans, New Orleans, USA · US Army Corps of Engineers, Engineer Research and Development Center, Vicksburg, Mississippi, USA

May 28, 2026J/K move · Enter open · S save

Large Depth Completion Model from Sparse Observations

Zhu Yu, Zhengyi Zhao, Runmin Zhang, Lingteng Qiu+6

Zhejiang University · Internship at Tongyi Lab · Tongyi Lab, Alibaba Group+4

Metric Depth Estimation Depth Estimation Depth Completion Monocular Depth Estimation

Open paper AskPDF arXiv

This work presents the Large Depth Completion Model (LDCM), a simple, effective, and robust framework for single-view metric depth estimation with sparse observations. Without relying on complex architectural designs, LDCM generates metric-accurate dense depth maps using a transformer. It outperforms existing approaches across diverse datasets and sparse observations. We achieve this from two key perspectives: (1) leveraging existing monocular foundation models to improve the quality of sparse depth inputs, and (2) reformulating training objectives to better capture geometric structure and metric consistency. Specifically, a Poisson-based depth initialization strategy is first introduced to generate a uniform coarse dense depth map from diverse sparse observations, providing a strong structural prior for the network. Regarding the training objective, we replace the conventional depth head with a point map head that regresses per-pixel 3D coordinates in camera space, enabling the model to directly learn the underlying 3D scene structure instead of performing pixel-wise depth map restoration. Moreover, this design eliminates the need for camera intrinsic parameters, allowing LDCM to naturally produce metric-scaled 3D point maps. Extensive experiments demonstrate that LDCM consistently outperforms state-of-the-art methods across multiple benchmarks and varying sparsity levels in both depth completion and point map estimation, showcasing its effectiveness and strong generalization to unseen data distributions.

Abstract

Depth Completion

Explore similar work

Large Depth Completion Model from Sparse Observations

AURORA-KITTI: Any-Weather Depth Completion and Denoising

EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion