UniDAC: Universal Metric Depth Estimation for Any Camera

Girish Chandar Ganesan; Yuliang Guo; Liu Ren; Xiaoming Liu

Figure 1: We propose UniDAC , a universal, domain-agnostic metric depth estimation framework that generalizes to any camera. Unlike prior methods that either rely on large-FoV data during training or require separate models for indoor and outdoor domains, UniDAC is trained solely on perspective images yet generalizes effectively to large-FoV inputs, leveraging a universal model to robustly handle both indoor and outdoor environments.

Figure 2: We show the Abs.Rel error between the predicted relative depth and the ground truth by performing (a) no scaling, (b) median scaling, and (c) depth guided scaling. Theoretically, the relative and metric depth are aligned with a single scale

s

. Practically, (b) we observe that the irregularities in the relative depth cannot be compensated with a single scalar

s

. Thus, to tackle this, (c) we propose a Depth-Guided Scale Estimation module that predicts a high-resolution scale map

\mathbf{S}

respecting local variations.

Figure 3: Overview of proposed method. UniDAC decouples metric depth estimation into relative depth and scale estimation. Relative depth relies on local scene information, while scene scale is domain-specific and depends on global scene information. Therefore, given an ERP image

\mathbf{I}

, we split the features from the encoder into local

\mathbf{F}_{l}

and the global features

\mathbf{F}_{g}

. We predict the relative depth

\mathbf{D}^{\text{rel}}

using the local features

\mathbf{F}_{l}

. We predict a scale map

\mathbf{S}

from the global features

\mathbf{F}_{g}

to account for the irregularities in

\mathbf{D}^{\text{rel}}

. We first predict a low-resolution scale map

\mathbf{S}_{r}

and obtain the high-resolution

\mathbf{S}

through our proposed Depth-Guided Scale (DGS) estimation module. The DGS upsamples

\mathbf{S}_{r}

by using the

\mathbf{D}^{\text{rel}}

as a guide to ensure the upsampling process respects object boundaries. The final metric depth

\mathbf{D}^{\text{m}}

is calculated using

\mathbf{D}^{\text{rel}}

and

\mathbf{S}

as shown in Eq. 8 . We introduce distortion-aware positional embedding, termed RoPE-

\phi

, that applies a weight

w(\phi)

to the RoPE rotations based on the latitude

\phi

. We train using two losses

\mathcal{L}_{\text{rel}}

and

\mathcal{L}_{\text{m}}

applied on

\mathbf{D}^{\text{rel}}

and

\mathbf{D}^{\text{m}}

, respectively.

Figure 4: Depth-Guided Upsampling. We leverage the predicted relative depth

\mathbf{D}^{\text{rel}}

as a guide to upsample the predicted low-resolution scale map

\mathbf{S}_{r}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}}

to get

\mathbf{S}\in\mathbb{R}^{H\times W}

. We compare

\mathbf{D}^{\text{rel}}

and its downsampled version

\mathbf{D}_{r}

to get the local information in the form of weights

\mathbf{W}\in\mathbb{R}^{H\times W\times 9}

. We compare the spatial mapping between

\mathbf{S}

and

\mathbf{S}_{r}

and combine it with

\mathbf{W}

to obtain

\mathbf{S}

. The Depth-Guided Upsampling is non-parametric and thus does not add computational overhead.

Figure 5: Motivation for RoPE-

\phi

. We show the difference between (a) the pixel distance in ERP and (b) the corresponding geodesic distance on the curvature of the sphere. Although

|\mathbf{p}_{11}-\mathbf{p}_{12}|=|\mathbf{p}_{21}-\mathbf{p}_{22}|

in the ERP, we see that

\mathcal{G}(\mathbf{p}_{11},\mathbf{p}_{12})<\mathcal{G}(\mathbf{p}_{21},\mathbf{p}_{22})

on the sphere. Geodesic distance respects the actual separation in the 3D space. Thus, we modify 2D-RoPE to reflect the geodesic distance to get RoPE-

\phi

.

Methods	Dataset	ScanNet++			Pano3D-GV2			KITTI-360
Methods	Size	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
UniK3D [ 39 ] - V	$7.94$ M	$0.651$	$0.253$	$0.285$	$0.785$	$0.170$	$0.400$	$0.817$	$0.244$	$2.400$
Metric3Dv2 [ 23 ] - V	$16.20$ M	$0.536$	$0.223$	$0.895$	$0.404$	$0.307$	$0.855$	$0.716$	$0.200$	$4.580$
UniDepth [ 42 ] - V	$3.83$ M	$0.364$	$0.497$	$1.166$	$0.247$	$0.789$	$1.268$	$0.481$	$0.294$	$6.564$
DAC U [ 19 ] - S	$0.79$ M	$0.658$	$0.233$	$0.464$	$0.684$	$0.203$	$0.507$	$0.708$	$0.186$	$5.079$
UniDAC- V	$1.45$ M	$0.918$	$0.097$	$0.277$	$0.768$	$0.161$	$0.394$	$0.836$	$0.141$	$3.977$

Table 1: Zero-shot evaluation of universal domain robustness . We evaluate all unified methods on a zero-shot setting across both indoor and outdoor domains. All models are trained on a mix of indoor and outdoor datasets. UniK3D is trained on large FoV images, while the rest of the methods are only trained on perspective images. [Key: Best , Second Best , S: Swin-L [ 32 ] , V: ViT-L [ 7 ] ]

Methods	ScanNet++			Pano3D-GV2			Matterport3D			KITTI-360
Methods	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
DAC I [ 19 ] - R	$0.852$	$0.132$	$0.309$	$0.812$	$0.139$	$0.478$	$0.773$	$0.156$	$0.619$	$0.082$	$0.464$	$10.046$
DAC I [ 19 ] - S	$0.854$	$0.128$	$0.287$	$0.729$	$0.184$	$0.483$	$0.723$	$0.179$	$0.591$	$0.245$	$0.365$	$9.508$
DAC O [ 19 ] - R	$0.256$	$0.901$	$2.312$	$0.340$	$0.616$	$1.713$	$0.390$	$0.548$	$1.630$	$0.786$	$0.156$	$4.361$
DAC O [ 19 ] - S	$0.109$	1.412	$3.539$	$0.323$	$0.870$	$2.056$	$0.330$	$0.924$	$2.164$	$0.822$	$0.149$	$3.751$
DAC U [ 19 ] - S	$0.658$	$0.233$	$0.464$	$0.684$	$0.203$	$0.507$	$0.662$	$0.215$	$0.662$	$0.708$	$0.186$	$5.079$

Table 2: Zero-shot evaluation of cross-camera generalization . We evaluate all methods on a zero-shot setting across diverse cameras. *

{}_{\text{I}}

, *

{}_{\text{O}}

indicate models trained only on the indoor and outdoor datasets, respectively. [Key: Best , Second Best , R: ResNet101 [ 20 ] , S: Swin-L [ 32 ] , V: ViT-L [ 7 ] ]

Figure 6: Qualitative Results. Every pair of consecutive rows corresponds to a single sample. Odd rows display the input RGB image, and A.Rel error between predicted and GT depth maps. Even rows display the GT depth map and predicted depth maps. UniDAC reduces error in the distorted region as seen around the table in ScanNet++ compared to DAC U and on the walls in Pano3D-GV2 compared to UniK3D.

Scale	ScanNet++			KITTI-360
Scale	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
$-$	$0.782$	$0.152$	$0.401$	$0.563$	$0.286$	$5.461$
$s\in\mathbb{R}$	$0.773$	$0.166$	$0.426$	$0.601$	$0.245$	$5.257$
$\mathbf{S}\in\mathbb{R}^{H\times W}$	$0.792$	$0.140$	$0.396$	$0.622$	$0.239$	$5.057$

Table 3: Ablation on Scale Estimation. We study the impact of depth-guided scale estimation by comparing it against single-scale estimation. The model performances are compared in a zero-shot setting. ‘-’ indicates we directly estimate metric depth without decoupling relative depth and metric scale. [Key: Best ]

Pos. Emb	ScanNet++			KITTI-360
Pos. Emb	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
RoPE	$0.750$	$0.177$	$0.452$	$0.592$	$0.254$	$5.313$
RoPE- $\phi$	$0.792$	$0.140$	$0.396$	$0.622$	$0.239$	$5.057$

Table 4: Ablation on RoPE. We evaluate the benefit of RoPE-

\phi

, comparing it against a model trained with 2D RoPE. The model performances are compared in a zero-shot setting. [Key: Best ]

Dataset	#Images	Scene	Acquisition
HM3D-tiny [ 43 ]	$310$ K	Indoor	Rec
Taskonomy-tiny [ 68 ]	$300$ K	Indoor	RGB-D
Hypersim [ 47 ]	$54$ K	Indoor	Syn
DDAD [ 17 ]	$80$ K	Outdoor	LiDAR
LYFT [ 22 ]	$50$ K	Outdoor	LiDAR
Argoverse2 [ 59 ]	$300$ K	Outdoor	LiDAR

Table 5: Training Datasets. List of training datasets with the following attributes: number of images, scene type, and acquisition method. [Key: Syn=Synthetic, Rec=Mesh reconstruction]

Dataset	Cam.Type	Scene	Acquisition
ScanNet++ [ 63 ]	Fisheye	Indoor	Rec
Matterport3D [ 5 ]	ERP	Indoor	Rec
Pano3D-GV2 [ 2 ]	ERP	Indoor	Rec
KITTI-360 [ 31 ]	Fisheye	Outdoor	LiDAR

Table 6: Testing Datasets. List of testing datasets with the following attributes: camera type, scene type, and acquisition method. [Key: Rec=Mesh reconstruction]

Method	ScanNet++			KITTI-360
Method	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
UniK3D	$0.651$	$0.253$	$0.285$	$0.817$	$0.244$	$2.400$
UniDAC- P	$0.894$	$0.110$	$0.274$	$0.706$	$0.198$	$4.397$
+ A2D2	$0.917$	$0.104$	$0.279$	$0.815$	$0.154$	$4.091$
UniDAC- G	$0.905$	$0.104$	$0.274$	$0.757$	$0.169$	$4.470$
+ A2D2	$0.918$	$0.097$	$0.277$	$0.836$	$0.141$	$3.977$

Table 7: Effect of camera intrinsics on depth performance. We study the impact of utilizing predicted and ground-truth {P,G} camera intrinsics during inference. [Key: Best , Second Best ]

Method	Training	KITTI			NYU-v2			nuScenes			IBims-1
Method	Size	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
Metric3Dv2	$16$ M	$0.974$	$0.053$	$2.493$	$0.972$	$0.067$	$0.262$	$0.841$	$0.236$	$9.400$	$0.684$	$0.207$	$0.700$
UniDepth	$3$ M	$0.964$	$0.116$	$2.788$	$0.988$	$0.052$	$0.194$	$0.846$	$0.127$	$4.560$	$0.157$	$0.410$	$1.250$
UniK3D	$8$ M	$0.833$	$0.159$	$4.323$	$0.899$	$0.133$	$0.400$	$0.840$	$0.189$	$10.830$	$0.919$	$0.104$	$0.406$
DAC U	$0.8$ M	$0.767$	$0.180$	$5.332$	$0.816$	$0.140$	$0.505$	$0.631$	$0.225$	$8.321$	$0.808$	$0.370$	$1.182$
UniDAC	$1.1$ M	$0.872$	$0.122$	$3.784$	$0.934$	$0.093$	$0.354$	$0.801$	$0.151$	$6.335$	$0.845$	$0.129$	$0.577$

Table 8: Zero-shot evaluation on perspective datasets. We evaluate all unified models on perspective datasets. All models are trained on a mix of indoor and outdoor datasets. [Key: Best ]

Method	ScanNet++			KITTI-360
Method	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
DAC U - D2	$0.368$	$0.305$	$0.939$	$0.334$	$0.318$	$6.872$
DAC U - D3	$0.707$	$0.211$	$0.471$	$0.428$	$0.342$	$5.305$
UniDAC- D2	$0.533$	$0.242$	$0.703$	$0.445$	$0.287$	$6.924$
UniDAC- D3	$0.792$	$0.140$	$0.396$	$0.622$	$0.239$	$5.057$

Table 9: Ablation on Encoder Weight . D2, D3 indicate the ViT encoders have been initialized with DINOv2 and DINOv3 pre-trained weights, respectively. [Key: Best ]

Shift	ScanNet++			KITTI-360
Shift	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	A.Rel $\downarrow$	RMSE $\downarrow$
$t\in\mathbb{R}$	$0.792$	$0.140$	$0.396$	$0.622$	$0.239$	$5.057$
$\mathbf{T}\in\mathbb{R}^{H\times W}$	$0.798$	$0.139$	$0.393$	$0.630$	$0.235$	$4.985$

Table 10: Ablation on Shift Estimation. We study the impact of depth-guided shift map estimation. [Key: Best ]

Figure 7: Qualitative Results on ScanNet++ [ 63 ] . Every pair of consecutive rows corresponds to a single sample. Odd rows display the input RGB image, and A.Rel error between predicted and GT depth maps. Even rows display the GT depth map and predicted depth maps.

Figure 8: Qualitative Results on Pano3D-GV2 [ 2 ] . Every pair of consecutive rows corresponds to a single sample. Odd rows display the input RGB image, and A.Rel error between predicted and GT depth maps. Even rows display the GT depth map and predicted depth maps.

Figure 9: Qualitative Results on KITTI-360 [ 31 ] . Every pair of consecutive rows corresponds to a single sample. Odd rows display the input RGB image, and A.Rel error between predicted and GT depth maps. Even rows display the GT depth map and predicted depth maps.

UniDAC: Universal Metric Depth Estimation for Any Camera

Abstract

Figures & tables

Metric Depth Estimation

DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images

DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection

The Midas Touch for Metric Depth: Interpretable Metricization for Generalizable Depth Perception

Abstract

Metric Depth Estimation

Explore similar work

DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images

DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection

The Midas Touch for Metric Depth: Interpretable Metricization for Generalizable Depth Perception