Organizations: School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China · School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China · Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China · Research Institute of Interdisciplinary Innovation, Zhejiang Lab, Hangzhou, China
Self-supervised monocular depth estimation (MDE) has received increasing interests in the last few years. The objects in the scene, including the object size and relationship among different objects, are the main clues to extract the scene structure. However, previous works lack the explicit handling of the changing sizes of the object due to the change of its depth. Especially in a monocular video, the size of the same object is continuously changed, resulting in size and depth ambiguity. To address this problem, we propose a Depth-converted-Scale Convolution (DcSConv) enhanced monocular depth estimation framework, by incorporating the prior relationship between the object depth and object scale to extract features from appropriate scales of the convolution receptive field. The proposed DcSConv focuses on the adaptive scale of the convolution filter instead of the local deformation of its shape. It establishes that the scale of the convolution filter matters no less (or even more in the evaluated task) than its local deformation. Moreover, a Depth-converted-Scale aware Fusion (DcS-F) is developed to adaptively fuse the DcSConv features and the conventional convolution features. Our DcSConv enhanced monocular depth estimation framework can be applied on top of existing CNN based methods as a plug-and-play module to enhance the conventional convolution block. Extensive experiments with different baselines have been conducted on the KITTI benchmark and our method achieves the best results with an improvement up to 11.6% in terms of SqRel reduction. Ablation study also validates the effectiveness of each proposed module.
Figures & tables
Fig. 1: Illustration of the object size and depth change at successcive frames of a monocular video.
Fig. 2: The framework of the proposed adaptive Depth-converted-Scale Convolution (DcSConv) enhanced monocular depth estimation. The progressively generated depth maps of different resolutions are used to further update the network by constructing adaptive-scale convolution based on the depth value.
Fig. 3: Illustration of the relationship between the object scale in an image and its scene depth. The length lr in the 3D is projected to different scales in an image with different depths.
Fig. 4: Illustration of the Depth-converted Multiple Scale convolution Fusion (DMSF).
Fig. 5: Illustration of Depth-converted-Scale Convolution (DcSConv) in comparison with the standard convolution among the successive frames. “ Po ” denotes each location on the input feature map and the center of each convolution operation, and “ Pr ”, “ Pd ” denote the sampling locations in the local grid, in fixed receptive field as the conventional convolution and the depth-converted-scale receptive field in the proposed DcSConv, respectively. With the object scale information provided by the corresponding depth map, our DcSConv can adaptively capture object scale changes with adaptive local receptive field.
Fig. 6: Illustration of the Depth-converted-Scale aware Fusion (DcS-F) module. It consists of a DcS-channel attention block and a DcS-spatial attention block based on the depth information to fully fuse the DcSConv features and conventional convolution features.
Fig. 7: Architecture of the Depth-converted-Scale aware feature Decoding (DcS-D) module. Different from the conventional skip-connection based feature decoding, the features from the encoder and the previous decoder layers are first enhanced with the DcSConv and then fused with the conventional convolution features using DcS-F module.
Evaluation metric
Para
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Baseline(MD2)
14.34M
0.115
0.903
/
4.863
0.877
0.959
Baseline + DMSF
25.99M
0.112
0.831
7.9%
4.759
0.880
0.960
Baseline + Fixed-scale conv
17.52M
0.116
0.864
4.3%
4.814
0.875
0.959
Baseline + Deformable conv
17.69M
0.114
0.918
-1.6%
4.879
0.879
0.958
Baseline + GLSConv (proposed)
18.30M
0.112
0.823
8.8%
4.789
0.879
0.960
Baseline + DcSConv (proposed)
17.52M
0.112
0.811
10.2%
4.732
0.879
0.960
TABLE I: Comparison of different convolution methods based on the Monodepth2 (MD 2) [7]. The best results are shown in bold.
Evaluation metric
Para
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel ↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Baseline
14.34M
0.115
0.903
/
4.863
0.877
0.959
Concatenation
17.52M
0.112
0.811
10.2%
4.732
0.879
0.960
Summation
15.75M
0.113
0.824
8.7%
4.754
0.878
0.960
CBAM [ 47 ]
17.57M
0.114
0.824
8.7%
4.764
0.877
0.959
DcS-F
18.42M
0.111
0.798
11.6%
4.701
0.882
0.961
TABLE II: Comparison of different feature fusion methods. The best results are shown in bold.
Method
Dataset
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel ↑
RMSE ↓
Monodepth2 [ 12 ]
Make3D
0.322
3.589
-
7.417
Proposed+Monodepth2
0.309
3.137
12.59%
7.083
Monodepth2 [ 12 ]
NYU V2
0.384
0.753
-
1.276
Proposed+Monodepth2
0.328
0.617
18.06%
1.228
TABLE III: Results of proposed method in comparison with the baseline on Make3D and NYUv2.
Fig. 8: Sample qualitative results from the NYU V2 indoor benchmark dataset.
Method
Backbone
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Monodepth2 [ 12 ] (ICCV)
ResNet 18
0.115
0.903
/
4.863
0.877
0.959
Proposed+Monodepth2
ResNet 18
0.112
0.798
11.6%
4.701
0.882
0.961
Zhou et al [ 59 ] (ICCV)
ResNet 18
0.121
0.837
7.3%
4.945
0.853
0.955
Sun et al [ 42 ] (TNNLS)
ResNet 18
0.117
0.863
4.4%
4.813
0.871
0.959
Guizilini et al [ 13 ] (ICLR)
ResNet 18
0.117
0.854
5.4%
4.714
0.873
0.963
SGDepth [ 19 ] (ECCV)
ResNet 18
0.113
0.835
7.5%
4.693
0.879
0.961
TABLE IV: Experimental results on the self-supervised monocular depth estimation in comparison with the state-of-the-art methods using KITTI Eigen split. All methods are trained with monocular images of resolution 640×192 . Improvement percentage in terms of Sq Rel over the Monodepth2 is used as an extra measurement. The best method is shown in bold for each category of backbone.
Fig. 9: Qualitative compassion of different methods on the KITTI Eigen split. Our model produces better quality depth maps with clearer object edges.
School of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China · Key Laboratory of Advanced Marine Communication and Information Technology, Harbin 150001, China · Department of Computer Science, University of Toronto, Toronto ON M5S 2E4, Canada +1