RoSAMDepth: Robust Self-supervised Depth Estimation Leveraging Segment Anything Model

Xuanang Gao; Zhiwei Ning; Gengming Zhang; Jiaxi Cao; Runze Yang; Zhonglong Zheng; Jie Yang; Rong Xiao

cs.CVOct 28, 2026

RoSAMDepth: Robust Self-supervised Depth Estimation Leveraging Segment Anything Model

Authors: Xuanang Gao, Zhiwei Ning, Gengming Zhang, Jiaxi Cao, Runze Yang, Zhonglong Zheng, Jie Yang, Rong Xiao,

Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University · School of Computing, Macquarie University · School of Computer Science, Zhejiang Normal University ·

Abstract

Robust depth estimation aims to maintain high-quality depths across diverse conditions. However, most existing methods estimate depth without taking into account the object-level information. As a result, the predicted depth may easily deviate within objects and become blurred under adverse conditions. To overcome this weakness, we propose RoSAMDepth, a novel framework that can assist robust self-supervised depth estimation in leveraging rich and diverse object-level priors from the Segment Anything Model (SAM). We focus on incorporating object-level information across three key aspects: a segment-guided representation contrasting method that injects object-level awareness into the feature representation space; an adaptive regional outlier masking strategy combined with a regional Gaussian likelihood loss that enforces regional depth smoothness; and an object-level reliability estimation strategy that mitigates the influence of unreliable supervision. Extensive experiments across multiple datasets and diverse weather conditions demonstrate that our method produces sharper, more accurate depth predictions, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/xagao/RoSAMDepth.

Figures & tables

Figure 1: Our RoSAMDepth utilizes object-level information from SAM [ 14 ] to generate sharp and accurate depths across diverse conditions.

Figure 2: Illustration of the proposed method. Given the SAM masks, we leverage object-level priors to improve depth estimation $D_{s}$ in two complementary ways: 1) Object-aware representation learning: SRC injects segment priors into multi-scale features. 2) Depth learning with object-level priors, AROM generates an outlier-suppressed mask $M_{out}$ , which is paired with $L_{rgl}$ to enforce consistent depth within each region. Concurrently, ORE estimates object-level reliability to form a reliability-weighted map $W_{rel}$ , combined with $M_{out}$ to modulate the distillation loss $L_{d}$ .

Figure 3: An example of why conventional boundary-aware smoothness loss performs poorly. (a) Original image. (b) Depth from $\Phi_{t}$ overlaid with the SAM boundary (black). (c) Weak and local gradients of the conventional boundary-aware smoothness loss. (d) Our outlier-suppressed mask. (e) Strong, non-local gradients from our proposed regional Gaussian likelihood loss guided by the outlier-suppressed mask.

Figure 4: Pixel-level photometric error is insufficient for evaluating the object-level depth reliability. (a) Image patch. (b) Depth predictions from $\Phi_{t}$ , with the car’s depth scaled from 0.25 to 4.0. (c) The corresponding photometric error map, where brighter intensity denotes larger error, fails to capture the object-level depth reliability. (d) Depth map obtained by selecting, for each pixel, the scaled depth that minimizes the pixel-level photometric error. (e) Our object-level reliability correctly identifies the entire mispredicted car as an unreliable region.

Method	Test	day-clear – nuScenes				night – nuScenes				day-rain – nuScenes
Method	frames	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\pagecolor{up}\delta_{1}\uparrow$	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
Monodepth2 [ 8 ]	1	0.1333	1.820	6.459	85.88	0.2419	2.776	10.922	58.17	0.1572	2.273	7.453	79.49
R4Dyn ${}^{\textbf{\textdagger}}$ (Radar) [ 6 ]	1	0.1259	1.661	6.434	86.97	0.2194	2.889	10.542	62.28	0.1337	1.938	7.131	83.91
RNW [ 35 ]	1	0.2872	3.433	9.185	56.21	0.3333	4.066	10.098	43.72	0.2952	3.796	9.341	57.21
robust-depth [ 28 ]	1	0.1436	1.862	6.802	82.95	0.2101	2.691	8.673	69.58	0.1458	1.891	7.371	80.21
md4all-AD [ 7 ]	1	0.1523	2.141	6.853	83.11	0.2187	2.991	9.003	68.84	0.1601	2.259	7.832	78.97

Table 1: Quantitative comparison on nuScenes [ 3 ] under different conditions (day-clear, nighttime, and day-rain). † denotes the method [ 6 ] that leverages additional Radar information. ‘Test frames’ specifies whether the evaluation is conducted in a multi-frame or single-frame setting. The best and the second best results are highlighted in bold and underline , respectively.

Method	Test	day – Robotcar				night – Robotcar
Method	frames	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\pagecolor{up}\delta 1\uparrow$	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\pagecolor{up}\delta 1\uparrow$
Monodepth2 [ 8 ]	1	0.1209	0.723	3.335	86.61	0.3909	3.547	8.227	22.51
DeFeatNet [ 30 ]	1	0.2470	2.980	7.884	65.00	0.3340	4.589	8.606	58.60
ADIDS [ 18 ]	1	0.2390	2.089	6.743	61.40	0.2870	2.569	7.985	49.00
RNW [ 35 ]	1	0.2970	2.608	7.996	43.10	0.1850	1.710	6.549	73.30
WSGD [ 35 ]	1	0.1760	1.603	6.036	75.00	0.1740	1.637	6.302	75.40

Table 2: Quantitative comparison on the Robotcar [ 20 ] test set under daytime and nighttime conditions.

Figure 5: Qualitative comparison of depth predictions under diverse conditions. The first two rows show results on nuScenes [ 3 ] (rain and nighttime), and the last two rows show results on Robotcar [ 20 ] (daytime and nighttime).

SRC	AROM	$L_{rgl}$	ORE	night – nuScenes				day-rain – nuScenes
SRC	AROM	$L_{rgl}$	ORE	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\delta 1\uparrow$	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	$\delta 1\uparrow$
✓				0.1725	1.833	7.828	73.21	0.1270	1.623	6.710	82.82
	✓			0.1711	1.806	7.774	73.24	0.1263	1.612	6.681	83.03
		✓		0.1734	1.841	7.819	73.14	0.1289	1.657	6.794	82.87
	✓	✓		0.1699	1.804	7.756	73.67	0.1259	1.603	6.639	83.17
	✓	✓	✓	0.1673	1.798	7.736	73.89	0.1249	1.608	6.621	83.45

Table 3: Ablations study of our components on nuScenes [ 3 ] . The checkmark indicates the inclusion of the corresponding component.

Smoothness Loss	night – nuScenes		day-rain – nuScenes
Smoothness Loss	AbsRel $\downarrow$	$\delta 1\uparrow$	AbsRel $\downarrow$	$\delta 1\uparrow$
Image-edge-aware	0.1735	73.15	0.1280	82.75
SAM-boundary-aware	0.1732	73.17	0.1273	82.78
AROM & $L_{rgl}$ (Ours)	0.1699	73.67	0.1259	83.17

Table 4: Ablation study on different smoothness loss definitions, SAM-boundary-aware smoothness loss indicates replacing

I

with

M_{sam}

in Eq. 3 .

Method	night – nuScenes		day-rain – nuScenes
Method	AbsRel $\downarrow$	$\delta 1\uparrow$	AbsRel $\downarrow$	$\delta 1\uparrow$
SRC on $D_{s}$	0.1694	73.68	0.1261	83.05
SRC (Ours)	0.1667	73.95	0.1236	83.53

Table 5: Ablation study on different implementations of Eq. 2 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Example of the SAM masks generation.

Method	night – nuScenes		day-rain – nuScenes
Method	AbsRel $\downarrow$	$\delta 1\uparrow$	AbsRel $\downarrow$	$\delta 1\uparrow$
S=1	0.1674	73.88	0.1250	83.45
S=2	0.1671	73.91	0.1244	83.47
S=3	0.1667	73.95	0.1236	83.53
S=4	0.1665	73.95	0.1234	83.55

Appendix

Table S1 : Ablation study on the number of feature scales (

S

) used in Eq. 2 .

Figure S2 : More qualitative results on nuScenes [ 3 ]

Figure S3 : More qualitative results on Robotcar [ 20 ]

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 199 papers

SA4Depth: Consistent Pose-Depth Scale Alignment for Self-Supervised Monocular Depth Estimation
May 27, 2026Changxuan Li, Nadine Berner, Nassir Navab +2Self-Supervised Learning Camera Pose Estimation
Technical University of Munich. · Munich Center for Machine Learning (MCML). · BMW Group. +2
CMAMBADEPTH: Self-supervised Monocular Depth Estimation with Channel Mamba and Hybrid Attention
Sep 21, 2026Xuezhi Xiang, Jiayao Liu, Heqi Xiang +3Monocular Depth Estimation Multi-Scale Feature Fusion
School of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China · Key Laboratory of Advanced Marine Communication and Information Technology, Harbin 150001, China · Department of Computer Science, University of Toronto, Toronto ON M5S 2E4, Canada +1
Weather-Conditioned Depth Anything
Sep 4, 2026Zhaoming Xu, Chan-Wei Hu, Kuan-Ru Huang +4Depth Estimation Monocular Depth Estimation
Texas A&M University, College Station, TX 77843, USA

May 27, 2026J/K move · Enter open · S save

SA4Depth: Consistent Pose-Depth Scale Alignment for Self-Supervised Monocular Depth Estimation

Changxuan Li, Nadine Berner, Nassir Navab, Federico Tombari+1

Technical University of Munich. · Munich Center for Machine Learning (MCML). · BMW Group.+2

Self-Supervised Learning Camera Pose Estimation Monocular Depth Estimation Self-Supervised Monocular Depth Estimation

Open paper AskPDF arXiv

Self-supervised depth estimation from monocular sequences relies on the joint learning of a depth and a pose network. Despite abundant research done to improve the depth network, efforts on the pose remain limited. In this context, even when depth is estimated up to scale, we highlight the importance of the alignment between the scene scales estimated by the pose and depth nets. Then, we introduce SA4Depth, an approach to improve this alignment and boost the depth predictions while keeping the inference time unchanged. Our proposed method uses the depth estimated during training to reproject learnable visual features across consecutive frames and refine the pose estimates by reducing feature alignment residuals. With our method, the estimated scene scales by the separate depth and pose networks are aligned, and the prediction scale consistency is improved across different sequences. Our differentiable refinement integrates seamlessly into existing self-supervised pipelines and substantially improves their depth estimates. We demonstrate this with extensive experiments both outdoors and indoors on KITTI, Cityscapes, and NYUv2. Additionally, results on KITTI Odometry confirm the effectiveness of our pose refinement. Our code is available at https://github.com/Runningchauncey/SA4Depth .

Abstract

Depth Estimation

Explore similar work

SA4Depth: Consistent Pose-Depth Scale Alignment for Self-Supervised Monocular Depth Estimation

CMAMBADEPTH: Self-supervised Monocular Depth Estimation with Channel Mamba and Hybrid Attention

Weather-Conditioned Depth Anything