Organizations: School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China · School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China · Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China · Research Institute of Interdisciplinary Innovation, Zhejiang Lab, Hangzhou, China
Self-supervised monocular depth estimation (MDE) has received increasing interests in the last few years. The objects in the scene, including the object size and relationship among different objects, are the main clues to extract the scene structure. However, previous works lack the explicit handling of the changing sizes of the object due to the change of its depth. Especially in a monocular video, the size of the same object is continuously changed, resulting in size and depth ambiguity. To address this problem, we propose a Depth-converted-Scale Convolution (DcSConv) enhanced monocular depth estimation framework, by incorporating the prior relationship between the object depth and object scale to extract features from appropriate scales of the convolution receptive field. The proposed DcSConv focuses on the adaptive scale of the convolution filter instead of the local deformation of its shape. It establishes that the scale of the convolution filter matters no less (or even more in the evaluated task) than its local deformation. Moreover, a Depth-converted-Scale aware Fusion (DcS-F) is developed to adaptively fuse the DcSConv features and the conventional convolution features. Our DcSConv enhanced monocular depth estimation framework can be applied on top of existing CNN based methods as a plug-and-play module to enhance the conventional convolution block. Extensive experiments with different baselines have been conducted on the KITTI benchmark and our method achieves the best results with an improvement up to 11.6% in terms of SqRel reduction. Ablation study also validates the effectiveness of each proposed module.
Figures & tables
Fig. 1: Illustration of the object size and depth change at successcive frames of a monocular video.
Fig. 2: The framework of the proposed adaptive Depth-converted-Scale Convolution (DcSConv) enhanced monocular depth estimation. The progressively generated depth maps of different resolutions are used to further update the network by constructing adaptive-scale convolution based on the depth value.
Fig. 3: Illustration of the relationship between the object scale in an image and its scene depth. The length lr in the 3D is projected to different scales in an image with different depths.
Fig. 4: Illustration of the Depth-converted Multiple Scale convolution Fusion (DMSF).
Fig. 5: Illustration of Depth-converted-Scale Convolution (DcSConv) in comparison with the standard convolution among the successive frames. “ Po ” denotes each location on the input feature map and the center of each convolution operation, and “ Pr ”, “ Pd ” denote the sampling locations in the local grid, in fixed receptive field as the conventional convolution and the depth-converted-scale receptive field in the proposed DcSConv, respectively. With the object scale information provided by the corresponding depth map, our DcSConv can adaptively capture object scale changes with adaptive local receptive field.
Fig. 6: Illustration of the Depth-converted-Scale aware Fusion (DcS-F) module. It consists of a DcS-channel attention block and a DcS-spatial attention block based on the depth information to fully fuse the DcSConv features and conventional convolution features.
Fig. 7: Architecture of the Depth-converted-Scale aware feature Decoding (DcS-D) module. Different from the conventional skip-connection based feature decoding, the features from the encoder and the previous decoder layers are first enhanced with the DcSConv and then fused with the conventional convolution features using DcS-F module.
Evaluation metric
Para
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Baseline(MD2)
14.34M
0.115
0.903
/
4.863
0.877
0.959
Baseline + DMSF
25.99M
0.112
0.831
7.9%
4.759
0.880
0.960
Baseline + Fixed-scale conv
17.52M
0.116
0.864
4.3%
4.814
0.875
0.959
Baseline + Deformable conv
17.69M
0.114
0.918
-1.6%
4.879
0.879
0.958
Baseline + GLSConv (proposed)
18.30M
0.112
0.823
8.8%
4.789
0.879
0.960
Baseline + DcSConv (proposed)
17.52M
0.112
0.811
10.2%
4.732
0.879
0.960
TABLE I: Comparison of different convolution methods based on the Monodepth2 (MD 2) [7]. The best results are shown in bold.
Evaluation metric
Para
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel ↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Baseline
14.34M
0.115
0.903
/
4.863
0.877
0.959
Concatenation
17.52M
0.112
0.811
10.2%
4.732
0.879
0.960
Summation
15.75M
0.113
0.824
8.7%
4.754
0.878
0.960
CBAM [ 47 ]
17.57M
0.114
0.824
8.7%
4.764
0.877
0.959
DcS-F
18.42M
0.111
0.798
11.6%
4.701
0.882
0.961
TABLE II: Comparison of different feature fusion methods. The best results are shown in bold.
Method
Dataset
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel ↑
RMSE ↓
Monodepth2 [ 12 ]
Make3D
0.322
3.589
-
7.417
Proposed+Monodepth2
0.309
3.137
12.59%
7.083
Monodepth2 [ 12 ]
NYU V2
0.384
0.753
-
1.276
Proposed+Monodepth2
0.328
0.617
18.06%
1.228
TABLE III: Results of proposed method in comparison with the baseline on Make3D and NYUv2.
Fig. 8: Sample qualitative results from the NYU V2 indoor benchmark dataset.
Method
Backbone
Abs Rel ↓
Sq Rel ↓
Δ Sq Rel↑
RMSE ↓
δ<1.25 ↑
δ<1.252 ↑
Monodepth2 [ 12 ] (ICCV)
ResNet 18
0.115
0.903
/
4.863
0.877
0.959
Proposed+Monodepth2
ResNet 18
0.112
0.798
11.6%
4.701
0.882
0.961
Zhou et al [ 59 ] (ICCV)
ResNet 18
0.121
0.837
7.3%
4.945
0.853
0.955
Sun et al [ 42 ] (TNNLS)
ResNet 18
0.117
0.863
4.4%
4.813
0.871
0.959
Guizilini et al [ 13 ] (ICLR)
ResNet 18
0.117
0.854
5.4%
4.714
0.873
0.963
SGDepth [ 19 ] (ECCV)
ResNet 18
0.113
0.835
7.5%
4.693
0.879
0.961
TABLE IV: Experimental results on the self-supervised monocular depth estimation in comparison with the state-of-the-art methods using KITTI Eigen split. All methods are trained with monocular images of resolution 640×192 . Improvement percentage in terms of Sq Rel over the Monodepth2 is used as an extra measurement. The best method is shown in bold for each category of backbone.
Fig. 9: Qualitative compassion of different methods on the KITTI Eigen split. Our model produces better quality depth maps with clearer object edges.
Accurate monocular depth estimation serves as a core enabler for single camera scene understanding. However, existing self-supervised monocular depth estimation methods generally suffer from the bottleneck of inefficient cross-scale information interaction and difficulty in balancing local and global spatial modeling. In this paper, we propose CMambaDepth, a self-supervised framework that achieves efficient multi-scale feature fusion and fine-grained contextual modeling via channel-wise selective state propagation. Specifically, Bidirectional Channel Mamba (Bi-CMamba) aligns encoder features across scales and enables bidirectional information exchange among ordered scale groups. Unidirectional Channel Mamba (Uni-CMamba) progressively aggregates decoder features and retains fine-grained scale groups through a group selection mechanism for subsequent fusion. Furthermore, a Hybrid Attention Module (HAM) is introduced to combine large-kernel local context and Manhattan self-attention for complementary spatial modeling. Experimental results demonstrate that our method achieves highly competitive performance. Specifically, our model achieves an AbsRel of 0.094 and an RMSE of 4.156 on KITTI, and an AbsRel of 0.140 on DDAD. In the zero-shot cross-dataset generalization test on NYUv2, it attains an AbsRel of 0.232, outperforming the baseline RA-Depth by 7.2%.
Xuezhi Xiang, Jiayao Liu, Heqi Xiang +3
School of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China · Key Laboratory of Advanced Marine Communication and Information Technology, Harbin 150001, China · Department of Computer Science, University of Toronto, Toronto ON M5S 2E4, Canada +1
Self-Supervised Monocular Depth Estimation (MDE) has garnered attention in recent years due to its independence from ground truth. However, most existing models are limited to a single scale and exhibit considerable performance degradation in complex driving environments. Networks specifically designed to handle dynamic traffic participants tend to be overly complex, hindering their deployment on resource-constrained automotive edge devices. To address these limitations and move towards robust driving perception, we propose FlexDepth, a scale-driven and flexible family of self-supervised MDE models tailored for challenging road scenarios. FlexDepth employs a two-stage static-dynamic decoupled training strategy, enabling the independent assessment of confidence for both static backgrounds and dynamic road objects. Furthermore, it introduces a meticulously designed Scale-Driven Decoder (SDD) to dynamically select components based on scale size, facilitating efficient feature fusion and the output of high-precision depth maps. Extensive experiments on standard driving benchmarks demonstrate that without any auxiliary information, our model achieves state-of-the-art performance across arbitrary scales with minimal computational overhead. Our smallest model, Flex-Nano, requires only 0.7 GFLOPs and achieves 37.6 FPS on mobile platforms, ensuring reliable real-time perception while maintaining excellent zero-shot generalization. Our source code is avalible: https://github.com/startnew/flexdepth
Zhaowen Zhu, Li Zhang, Yujie Chen +3
Hefei University of Technology, Hefei, China · Chengpin Home Tech, Nanjing, China · Differential Robotics, Hangzhou, China +1
Self-supervised depth estimation from monocular sequences relies on the joint learning of a depth and a pose network. Despite abundant research done to improve the depth network, efforts on the pose remain limited. In this context, even when depth is estimated up to scale, we highlight the importance of the alignment between the scene scales estimated by the pose and depth nets. Then, we introduce SA4Depth, an approach to improve this alignment and boost the depth predictions while keeping the inference time unchanged. Our proposed method uses the depth estimated during training to reproject learnable visual features across consecutive frames and refine the pose estimates by reducing feature alignment residuals. With our method, the estimated scene scales by the separate depth and pose networks are aligned, and the prediction scale consistency is improved across different sequences. Our differentiable refinement integrates seamlessly into existing self-supervised pipelines and substantially improves their depth estimates. We demonstrate this with extensive experiments both outdoors and indoors on KITTI, Cityscapes, and NYUv2. Additionally, results on KITTI Odometry confirm the effectiveness of our pose refinement. Our code is available at https://github.com/Runningchauncey/SA4Depth .
Changxuan Li, Nadine Berner, Nassir Navab +2
Technical University of Munich. · Munich Center for Machine Learning (MCML). · BMW Group. +2