In this work we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or Time of Flight cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primaldual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.
Figures & tables
Figure 1: Upsampling of a low resolution depth image (a) using an additional high resolution intensity image (b) through image guided anisotropic Total Generalized Variation (c). Depth maps are color coded for better visualization.
Figure 2: Projection from a low resolution depth map DL to a high resolution sparse depth map DS in the intensity camera coordinate system.
Art
Books
Moebius
Avg.Time [s]
x2
x4
x8
x16
x2
x4
x8
x16
x2
x4
x8
x16
Nearest
4.65
5.01
5.71
7.10
4.30
4.68
4.85
5.23
5.08
5.20
5.31
5.65
-
Bilinear
3.09
3.59
4.39
5.91
2.91
3.12
3.34
3.71
3.21
3.45
3.62
4.00
-
Yang et al . [ 24 ]
1.36
1.93
2.45
4.52
1.12
1.47
1.81
2.92
1.25
1.63
2.06
3.21
-
He et al . [ 10 ]
1.92
2.40
3.32
5.08
1.60
1.82
2.31
3.06
1.77
2.03
2.60
3.34
23.89
Diebel and Thrun [ 6 ]
1.62
2.24
3.85
5.70
1.34
2.08
2.85
3.54
1.47
2.29
3.09
3.81
-
Table 1: Quantitative comparison on the Middlebury 2007 datasets with added noise. The error is measured as RMSE of the pixel disparity for four different magnification factors ( ×2 , ×4 , ×8 , ×16 ). The best result for each dataset and upscaling factor is highlighted and the second best is underlined.
Figure 3: Visual comparison of ×8 upsampling on a snippet of the Middlebury Art dataset including fine structures. (a) RGB intensity image, (b) low resolution input image (enlarged using nearest neighbor upsampling). (c) Upsampling using MRF proposed by Diebel and Thrun [ 6 ] . (d) Adaptive bilateral upsampling proposed by Chan et al . [ 4 ] . (e) Nonlocal means upsampling proposed by Park et al . [ 16 ] . (f) Our upsampling method using image guided anisotropic TGV. The results in (c) and (d) still suffer from noise. (e) removes noise but suffers from edge bleeding especially at small structure boundaries. Our method removes noise and preserves sharp object edges.
Figure 4: Correlation between measured IR amplitude and depth measurement error of TOF acquisitions. The standard deviation of the measurements increases with decreasing IR intensities.
Figure 5: Compensation for amplitude based depth error of TOF cameras. Depth estimates on dark surfaces are underestimated (see red/full box). Using our depth calibration, we can compensate for that error (see green/dashed box).
Books
Shark
Devil
RMSE nearest [ mm ]
18.21
21.83
19.36
RMSE bilinear [ mm ]
17.10
20.17
18.66
RMSE Kopf et al. [ 12 ] [ mm ]
16.03
18.79
27.57
RMSE He et al. [ 10 ] [ mm ]
15.74
18.21
27.04
Input depth density [ % ]
2.57
2.55
2.53
RMSE OURS [ mm ]
12.36
15.29
14.68
Table 2: Quantitative evaluation on the real datasets Books , Shark and Devil . The error is calculated as RMSE to the measured groundtruth in mm . We compare standard interpolation methods as well as joint bilateral filtering [ 12 ] and image guided filtering [ 10 ] to our approach. The input density value shows the percentage of sparse depth values which are projected into the high resolution image space. This corresponds to an upsampling factor of approximately ×6.25 .
Figure 6: Visual evaluation on the real datasets Books (first row), Shark (second row) and Devil (third row). In column (a) the low resolution ToF image and the high resolution intensity image are shown, whereas column (b) shows the high resolution groundtruth depth. The black areas are not correctly reconstructed due to occlusions in the stereo system and therefore set invalid for the RMSE calculation. In column (c) the upsampling result of our method is shown whereas in column (d) the relative depth error to the known groundtruth is shown.
Though there exists a reasonable forward model for blur based on optical physics, recovering depth from a collection of defocused images remains a computationally challenging optimization problem. In this paper, we show that with contemporary optimization methods and reasonable computing resources, a global optimization approach to depth from defocus is feasible. Our approach rests on alternating minimization. When holding the depth map fixed, the forward model is linear with respect to the all-in-focus image. When holding the all-in-focus image fixed, the depth at each pixel can be computed independently, enabling embarrassingly parallel computation. We show that alternating between convex optimization and parallel grid search can effectively solve the depth-from-defocus problem at higher resolutions than current deep learning methods. We demonstrate our approach on benchmark datasets with synthetic and real defocus blur and show promising results compared to prior approaches. Our code is available at github.com/hollyjackson/dfd.
We propose AdaDS, an Adaptive Diffusion Sampling framework for prompted metric depth estimation, which estimates high-resolution metric depth from an image and an arbitrarily degraded low-resolution depth prompt. This setting is commonly studied as depth super-resolution, where existing methods typically regress depth directly without an explicit prior over high-quality depth, and thus often produce artifacts under severe or unseen degradations. In contrast, AdaDS exploits the contraction property of Gaussian smoothing: as noise accumulates in the forward diffusion process, the noisy distributions of degraded prompts and their high-quality counterparts converge toward an isotropic Gaussian prior, while the prompt content is gradually erased. Accordingly, AdaDS refines the prompt and estimates its uncertainty to adaptively select a starting timestep, and injects learned noise to place the intermediate sample in a high-density region of the depth distribution learned by a pre-trained diffusion model. Denoising from this sample yields high-quality depth with fine-grained details and corrects residual errors, even under imperfect prompt refinement. Experiments on real-world and synthetic benchmarks demonstrate its superior zero-shot generalization and robustness to diverse degradations.
Kun Wang, Yun Zhu, Pan Zhou +1
IMPL, Singapore University of Technology and Design · PCA Lab, Nanjing University of Science and Technology · LV-Lab, Singapore Management University
Guided depth completion methods heavily depend on RGB quality and alignment, while unguided ones often suffer from limited precision due to the absence of explicit visual cues. In this paper, we present Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion (GUDC), a new completion paradigm that innovatively bridges advanced 2D generative models with unguided depth completion, enabling semantics-aware depth inference without real RGB inputs. Our key idea is to exploit ControlNet's powerful depth-conditioned generation capability to synthesize pseudo-images directly from sparse depth, effectively converting the original unguided setting into a semantics-guided one. To address the potential image-depth misalignment caused by depth sparsity, we propose a multi-level dense-to-sparse representation distillation strategy for ControlNet fine-tuning, where dense-depth features act as teacher signals to distill consistent structural representations for sparse-depth inputs. Furthermore, during pseudo-image-guided completion, we propose a pseudo-image semantic attention fusion module to adaptively extract informative semantic cues from pseudo-images while suppressing artifacts (e.g., texture hallucinations). Extensive experiments on KITTI and NYUv2 validate that our GUDC achieves superior accuracy and robustness over existing methods.