Image Guided Depth Upsampling using Anisotropic Total Generalized Variation
Organizations: Graz University of Technology, Institute for Computer Graphics and Vision, Inffeldgasse 16, 8010 Graz, AUSTRIA
Abstract
In this work we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or Time of Flight cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primaldual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.
Figures & tables
| Art | Books | Moebius | Avg.Time [s] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| x2 | x4 | x8 | x16 | x2 | x4 | x8 | x16 | x2 | x4 | x8 | x16 | ||
| Nearest | 4.65 | 5.01 | 5.71 | 7.10 | 4.30 | 4.68 | 4.85 | 5.23 | 5.08 | 5.20 | 5.31 | 5.65 | - |
| Bilinear | 3.09 | 3.59 | 4.39 | 5.91 | 2.91 | 3.12 | 3.34 | 3.71 | 3.21 | 3.45 | 3.62 | 4.00 | - |
| Yang et al . [ 24 ] | 1.36 | 1.93 | 2.45 | 4.52 | 1.12 | 1.47 | 1.81 | 2.92 | 1.25 | 1.63 | 2.06 | 3.21 | - |
| He et al . [ 10 ] | 1.92 | 2.40 | 3.32 | 5.08 | 1.60 | 1.82 | 2.31 | 3.06 | 1.77 | 2.03 | 2.60 | 3.34 | 23.89 |
| Diebel and Thrun [ 6 ] | 1.62 | 2.24 | 3.85 | 5.70 | 1.34 | 2.08 | 2.85 | 3.54 | 1.47 | 2.29 | 3.09 | 3.81 | - |
| Books | Shark | Devil | |
| RMSE nearest [ ] | 18.21 | 21.83 | 19.36 |
| RMSE bilinear [ ] | 17.10 | 20.17 | 18.66 |
| RMSE Kopf et al. [ 12 ] [ ] | 16.03 | 18.79 | 27.57 |
| RMSE He et al. [ 10 ] [ ] | 15.74 | 18.21 | 27.04 |
| Input depth density [ ] | 2.57 | 2.55 | 2.53 |
| RMSE OURS [ ] | 12.36 | 15.29 | 14.68 |
Depth Estimation
Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.
See all 207 papers