Organizations: School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China. · School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China · the State Key Laboratory of Dynamic Testing Technology and School of Information and Communication Engineering, North University of China, Taiyuan 030051, China. · Research Institute of Interdisciplinary Innovation, Zhejiang Lab, Hangzhou, China.
Monocular depth estimation (MDE) has attracted increasing interest in the past few years, owing to its important role in 3D vision. MDE is the estimation of a depth map from a monocular image/video to represent the 3D structure of a scene, which is a highly ill-posed problem. To solve this problem, in this paper, we propose a LiftFormer based on lifting theory topology, for constructing an intermediate subspace that bridges the image color features and depth values, and a subspace that enhances the depth prediction around edges. MDE is formulated by transforming the depth value prediction problem into depth-oriented geometric representation (DGR) subspace feature representation, thus bridging the learning from color values to geometric depth values. A DGR subspace is constructed based on frame theory by using linearly dependent vectors in accordance with depth bins to provide a redundant and robust representation. The image spatial features are transformed into the DGR subspace, where these features correspond directly to the depth values. Moreover, considering that edges usually present sharp changes in a depth map and tend to be erroneously predicted, an edge-aware representation (ER) subspace is constructed, where depth features are transformed and further used to enhance the local features around edges. The experimental results demonstrate that our LiftFormer achieves state-of-the-art performance on widely used datasets, and an ablation study validates the effectiveness of both proposed lifting modules in our LiftFormer.
Figures & tables
Fig. 1: Illustration of the feature flow in our LiftFormer. Existing MDE methods predict the depth values by decoding the image spatial features and regressing the depth values (blue dashed line). Our LiftFormer lifts the depth prediction to the DGR subspace via SF-DGR transformation to generate depth features to better correspond to the depth bin centres (orange line). The depth features are further lifted to the ER subspace via the DF-ER transformation to enhance the edge information in the features (green line).
Fig. 2: Overview of the proposed LiftFormer architecture. The image spatial features are lifted to the depth-oriented geometric representation (DGR) subspace via the SF-DGR transformation. SF-DGR-based lifting is used at different scales to transform the encoder features to the decoder. DF-ER-based lifting is used to enhance the depth features with the local edge information. Finally, the depth features are progressively decoded with the DGR features and ER enhancement to generate the depth map in an AdaBin-style prediction.
Fig. 3: Illustration of the SF-DGR subspace transformation -based lifting module.
Fig. 4: t-SNE visualization of the encoder and DGR features in our LiftFormer versus the decoder features obtained by PixelFormer [ 23 ] at two scales (top: 1/32 and bottom: 1/16). Two example images from the KITTI dataset are used. The DGR features are continuously changed instead of being clustered, which better corresponds to continuous depth value prediction.
Fig. 5: Illustration of the DF-ER subspace transformation -based lifting module.
Fig. 6: Visualization of the ER coefficients obtained on the KITTI dataset (left: input image; right: ER coefficients). The brighter the color is, the larger the ER coefficient.
Method
RMSE ↓
Δ RMSE ↓
Abs Rel ↓
Sq Rel ↓
RMSElog ↓
ζ1↑
ζ2↑
ζ3↑
Eigen et al. [ 15 ]
6.307
-209.470%
0.203
1.548
-
-
-
-
Naderi et al. [ 66 ]
3.223
-58.145%
0.070
-
-
0.944
0.991
0.998
Focal-Wnet [ 67 ]
3.076
-50.932%
0.082
-
0.120
0.926
0.986
0.997
P3Depth [ 68 ]
2.842
-39.450%
0.071
0.270
0.103
0.953
0.993
0.998
DORN [ 20 ]
2.727
-33.808%
0.072
0.307
0.120
0.932
0.984
0.995
Wang et al. [ 69 ]
2.273
-11.531%
0.055
0.224
-
-
-
-
TABLE I: Results of the proposed LiftFormer in comparison with those of existing methods on the Eigen split of the KITTI dataset. The depth is divided from 0–80 m. ‘ ↓ ’ and ‘ ↑ ’ indicate that the lower and the higher the metric is, respectively, the better the results. Δ RMSE is calculated via comparison with our method.
Fig. 7: Qualitative results of the proposed LiftFormer in comparison with those of the PixelFormer [ 23 ] on the KITTI dataset.
Method
RMSE ↓
Δ RMSE
Abs Rel ↓
log10 ↓
ζ1↑
ζ2↑
ζ3↑
Eigen et al. [ 15 ]
0.641
-104.792%
0.158
0.039
0.789
0.950
0.988
Naderi et al. [ 66 ]
0.444
-41.853%
0.097
0.042
0.897
0.982
0.996
Focal-Wnet [ 67 ]
0.398
-27.157%
0.116
0.048
0.875
0.980
0.995
IronDepth [ 72 ]
0.352
-12.460%
0.101
0.043
0.910
0.985
0.997
Meta-Initialization [ 73 ]
0.348
-11.182%
0.093
0.043
0.908
0.980
0.995
DepthFormer [ 24 ]
0.339
-8.307%
0.096
0.041
0.921
0.989
0.998
TABLE II: Results of the proposed LiftFormer in comparison with those of existing methods on the Eigen split of the NYUV2 dataset. The depth is divided from 0–10 m. ‘ ↓ ’ and ‘ ↑ ’ indicate that the lower and the higher the metric is, respectively, the better the results. Δ RMSE is calculated via comparison with our method..
Fig. 8: Qualitative results of the proposed LiftFormer in comparison with those of PixelFormer [ 23 ] on the NYUV2 dataset.
Setting
Redundancy ( n/c )
RMSE ↓
Abs Rel ↓
Sq Rel ↓
ζ1↑
ζ2↑
DepthFormer. [ 24 ]
-
2.143
0.052
0.158
0.975
0.997
PixelFormer. [ 23 ]
-
2.081
0.051
0.149
0.976
0.997
SF-DGR (128)
128/32=4
2.054
0.051
0.147
0.976
0.997
SF-DGR (128)+DF-ER
128/32=4
2.038
0.050
0.143
0.978
0.998
SF-DGR (32)+DF-ER
32/32=1
2.058
0.051
0.145
0.977
0.997
SF-DGR (64)+DF-ER
64/32=2
2.051
0.050
0.144
0.977
0.998
TABLE III: Ablation study of the two proposed lifting modules and different configurations of the SF-DGR-based lifting module.
Fig. 9: Error map visualization on the KITTI dataset. The first column shows the error map of PixelFormer [ 23 ] . The second column shows the error map of the proposed method. The errors are represented with different colors mapped on the images, where dark and light colors represent small and large errors, respectively. The third column shows the improvement of the proposed method over PixelFormer, where the blue to red colors represent error reductions from small to large.
Method
Dec
RMSE ↓
Abs Rel ↓
Sq Rel ↓
ζ1↑
BinsF. [ 22 ]
CNN
2.141
0.052
0.156
0.974
PixelF. [ 23 ]
TF
2.081
0.051
0.149
0.976
LiftFormer
TF
2.059
0.051
0.148
0.976
LiftFormer
CNN
2.038
0.050
0.143
0.978
TABLE IV: Result comparison when different decoders are used in the LiftFormer. TF refers to the cross-attention-based transformer architecture, and CNN refers to the use of convolutional layers for the decoder.
Method
Range
RMSE ↓
Abs Rel ↓
Sq Rel ↓
ζ1↑
Fu et al. [ 20 ]
0-50m
2.271
0.071
0.268
0.936
PWA [ 78 ]
0-50m
1.872
0.057
0.161
0.965
P3Depth [ 68 ]
0-50m
1.651
0.055
0.130
0.974
LiftFormer
0-50m
1.531
0.048
0.109
0.981
TABLE V: Result comparison in the depth range of 0–50 m.
School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China · School of Control Science and Engineering, Shandong University, Jinan, China +4
Faculty of Mathematics, Physics and Informatics, Comenius University in Bratislava · Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague