UA-Depth: Uncertainty-Aware Depth Restoration for Material-Agnostic Object Manipulation

Rui Cao; Jia-Hui Pan; Yin Zhang; Qiwei Meng; Jiapeng Li; Gang Wang; Runsong Zhu; Wei Chen

cs.CVNov 1, 2026

UA-Depth: Uncertainty-Aware Depth Restoration for Material-Agnostic Object Manipulation

Authors: Rui Cao, Jia-Hui Pan, Yin Zhang, Qiwei Meng, Jiapeng Li, Gang Wang, Runsong Zhu, Wei Chen,

Organizations: Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · The University of Hong Kong, Hong Kong, China. · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China.

Abstract

Robotic manipulation requires accurate metric depth, yet commodity RGB-D cameras often fail on non-Lambertian objects such as transparent, reflective, and translucent materials, producing holes, structured noise, and biased depth. Most restoration methods produce only a single depth map, treating noisy and reliable regions uniformly for downstream tasks while ignoring regional uncertainty. We present UA-Depth, an uncertainty-aware depth restoration framework that iteratively co-refines depth and pixel-wise uncertainty, and uses uncertainty-gated features to suppress unreliable cues for stable updates. To strengthen zero-shot behavior, UA-Depth leverages a frozen depth foundation model as a prior to initialize missing or unreliable depth, and applies two-stage MAD-weighted log-depth scale normalization to reduce outlier bias in relative-to-metric alignment. We introduce an eight-dataset zero-shot benchmark across diverse scenes, materials, and sensors. Across these benchmarks, UA-Depth achieves the best mean rank among representative baselines, with particularly strong gains on datasets dominated by transparent and reflective objects. Beyond depth metrics, we demonstrate downstream improvements in category-level 6D pose estimation and 6D grasp synthesis by exploiting the predicted uncertainty. Code, models, and data for training and evaluation will be released upon publication.

Figures & tables

Fig. 2: Overview of UA-Depth. Given an RGB observation and a sensor depth map, we first extract RGB features with relative depth using a frozen DAV2 backbone [ 24 ] . We then derive two feature streams for refinement: an RGB context feature obtained by projecting DAV2 features through a lightweight feature transfer MLP, and a sensor-depth context feature produced by a context network (Sec. III-B ). In parallel, the proposed ADF module (Sec. III-C ) initializes metric depth and uncertainty by aligning the relative depth to valid sensor measurements and filling missing regions. Starting from this initialization, the Depth-Uncertainty Co-Refinement module (Sec. III-D ) jointly updates depth and uncertainty over several iterations, and finally upsamples the predictions to the input resolution via convex sampling [ 32 ] . We discuss uncertainty modeling in Sec. III-E and robust scale normalization in Sec. III-F .

Fig. 3: Detailed overview of Depth-Uncertainty Co-Refinement with Uncertainty-aware Context Feature Calibration (UCFC). We run $T$ ConvGRU refinement steps: at step $t$ , the ConvGRU updates the hidden state $\mathbf{H}^{t}$ and predicts residual updates that refine depth $\mathbf{D}^{t}$ and uncertainty $\mathbf{U}^{t}$ . UCFC is applied at each iteration to calibrate both RGB and depth context features using the predicted uncertainty, gating unreliable regions before they are fused into the ConvGRU update.

Fig. 4: Effect of different choices of error proxy $\mathcal{E}$ for learning the uncertainty map on samples of DREDS [ 11 ] . From left to right: RGB input, uncertainty maps trained with $L_{2}$ , $L_{1}$ , Grad, and $L_{1}{+}$ Grad losses. Red arrows highlight a transparent object where $L_{1}$ tends to emphasize interior details, Grad emphasizes only the boundary, and $L_{1}{+}$ Grad responds to both. All uncertainty maps are normalized with the same per-image percentile scaling for visualization.

Fig. 5: Risk–coverage curves on object pixels of the DREDS test split [ 11 ] for uncertainty learned with different error proxies. Pixels are ranked by predicted uncertainty and retained from low to high uncertainty; we plot mean REL (risk) versus retained fraction (coverage). Lower AURC indicates better uncertainty ranking.

Fig. 6: Box plots of the per-frame mean absolute error of the estimated median depth (in meters) on testset of HAMMER [ 26 ] and TransCG [ 25 ] . We observe that the vanilla scale normalization used in OMNI-DC [ 20 ] is sensitive to noisy or partially missing sensor depth, which is common in datasets containing transparent or reflective objects, leading to larger errors and higher variance compared to our robust scale normalization.

Split	Dataset	Size	Object	Scene	Depth Sensor
Training	DREDS-CatKnown [ 11 ]	100K	1536 objects, 7 categories	$\sim$ 74 scenes	Synthetic
	ClearPose [ 56 ]	222K	63 objects, 7 categories	24 scenes	RealSense L515
	HISS [ 14 ]	65K	350 objects	160 scenes	Synthetic
Testing	HAMMER [ 26 ]	1414	20 objects, 8 categories	9 scenes	RealSense D435
	HouseCat6D [ 27 ]	2929	50 objects, 10 categories	5 scenes	RealSense D435
	TransCG [ 25 ]	11K	46 objects	52 scenes	RealSense D435

TABLE I: Training/testing datasets used in zero-shot generalization evaluation. All dataset statistics are taken from the corresponding original papers unless otherwise specified.

Method	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}\uparrow$	$\delta_{1.10}\uparrow$	$\delta_{1.25}\uparrow$
SwinDR [ 11 ]	0.0433	0.0364	0.0282	78.13	90.06	98.24
$\text{D}^{3}\text{RoMa}$ [ 14 ]	0.0405	0.0494	0.0265	76.95	91.26	98.22
DP [ 36 ]	0.0587	0.0448	0.0274	82.98	89.97	95.89
OGNI-DC [ 19 ]	0.0491	0.0338	0.0242	82.17	92.73	98.44
PromptDA [ 21 ]	0.0523	0.0546	0.0317	69.73	85.89	95.39
PPFT [ 58 ]	0.0351	0.0302	0.0201	85.23	94.23	98.83

TABLE II: Quantitative results on HAMMER [ 26 ] . Metrics are computed over all valid pixels.

Methods	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}$ $\uparrow$	$\delta_{1.10}$ $\uparrow$	$\delta_{1.25}$ $\uparrow$
NLSPN [ 35 ]	0.010	0.009	0.006	97.48	99.51	99.97
LIDF [ 59 ]	0.016	0.018	0.011	93.60	98.71	99.92
SwinDR [ 11 ]	0.010	0.008	0.005	98.04	99.62	99.98
ASGrasp [ 23 ]	0.007	0.006	0.004	-	-	-
$\text{D}^{3}$ RoMa [ 14 ]	0.0101	0.0075	0.0050	98.11	99.49	99.97
PromptDA [ 21 ]	0.0138	0.0141	0.0090	95.96	99.12	99.93

TABLE III: Quantitative results on DREDS-CatKnown [ 11 ] . Metrics are computed over object pixels.

Methods	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}$ $\uparrow$	$\delta_{1.10}$ $\uparrow$	$\delta_{1.25}$ $\uparrow$
STD-CatKnown
SwinDR [ 11 ]	0.015	0.013	0.008	96.66	99.03	99.92
$\text{D}^{3}$ RoMa [ 14 ]	0.0176	0.0133	0.0096	96.04	98.42	99.74
PromptDA [ 21 ]	0.3427	0.5497	0.3353	71.63	77.29	82.26
Ours	0.0155 $\pm 0.0003$	0.0125 $\pm 0.0002$	0.0083 $\pm 0.0001$	96.92 $\pm 0.12$	98.87 $\pm 0.06$	99.89 $\pm 0.01$
STD-CatNovel

TABLE IV: Quantitative results on STD [ 11 ] . Metrics are computed over object pixels.

Testset	Method	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}\uparrow$	$\delta_{1.10}\uparrow$	$\delta_{1.25}\uparrow$
New Background	ImplicitDepth [ 59 ]	0.07	0.05	0.04	67.00	87.03	97.50
	TransCG [ 25 ]	0.03	0.03	0.02	86.50	97.02	99.74
	$\text{D}^{3}\text{RoMa}$ [ 14 ]	0.05	0.01	0.01	96.71	98.84	99.74
	PromptDA [ 21 ]	0.240	0.045	0.061	85.03	93.66	97.49
	Ours	0.057 $\pm 0.008$	0.009 $\pm 0.002$	0.011 $\pm 0.004$	96.18 $\pm 1.13$	98.55 $\pm 0.19$	99.55 $\pm 0.09$
Heavy Occlusion	ImplicitDepth [ 59 ]	0.11	0.09	0.08	41.43	66.52	91.96

TABLE V: Quantitative results on ClearPose [ 56 ] . Metrics are computed over all valid pixels.

Method	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank
Method	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
SwinDR [ 11 ]	0.0429	76.35	0.0463	76.84	0.0307	81.10	0.1078	61.75	0.0241	88.82	0.0151	93.53	0.0247	85.19	0.0318	77.95	3.2
$\text{D}^{3}\text{RoMa}$ [ 14 ]	0.0655	58.43	0.0498	76.45	0.0419	67.56	0.1231	50.27	0.0494	81.61	0.0511	73.36	0.0650	47.27	0.0575	50.07	5.3
PromptDA [ 21 ]	0.5156	6.62	0.3444	24.75	0.2130	8.97	0.3535	17.28	0.4402	15.52	0.1351	62.97	0.3060	38.45	0.6614	10.03	7.0
OMNI-DC [ 20 ]	0.0586	67.47	0.0530	71.34	0.0360	79.06	0.1642	52.00	0.0215	94.59	0.0260	91.52	0.0047	99.03	0.0063	99.40	3.1
PriorDA [ 22 ]	0.0762	41.42	0.0720	45.51	0.0418	71.37	0.1492	46.55	0.0338	87.47	0.0420	73.41	0.0205	95.26	0.0182	95.15	4.9

TABLE VI: Zero-shot generalization results on object pixels. We report REL and

\delta_{1.05}

on eight datasets. Bold and underlined entries indicate the best and second-best results, respectively.

Fig. 7: Qualitative zero-shot comparison on in-the-wild RGB-D scenes captured with a RealSense D435. We show the raw depth and the back-projected point clouds using the same intrinsics, with insets highlighting challenging regions. Our method reduces holes and structured noise on transparent/reflective objects and better preserves fine-scale geometry and boundaries compared with baselines.

Model Variant	ADF	Unc. Modeling	UCFC	HAMMER				DREDS-CatKnown
Model Variant	ADF	Unc. Modeling	UCFC	$\delta_{1.05}\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	REL $\downarrow$
Baseline	✗	✗	✗	86.55	0.0182	0.0322	0.0272	98.15	0.0046	0.0094	0.0070
+ ADF	✓	✗	✗	87.81	0.0170	0.0321	0.0263	98.17	0.0045	0.0095	0.0069
+ Unc. Modeling	✓	✓	✗	86.46	0.0184	0.0329	0.0271	98.19	0.0045	0.0094	0.0069
+ UCFC (Ours)	✓	✓	✓	89.56	0.0164	0.0309	0.0243	98.32	0.0044	0.0089	0.0067

TABLE VII: Ablation study on our framework components on the test set of HAMMER [ 26 ] and DREDS-CatKnown [ 11 ] . Metrics are computed over Object pixels.

Model Variant	REL $\downarrow$	RMSE $\downarrow$	$\delta_{1.05}\uparrow$	REL		RMSE		$\delta_{1.05}$
Model Variant	REL $\downarrow$	RMSE $\downarrow$	$\delta_{1.05}\uparrow$	AUSE $\downarrow$	AURG $\uparrow$	AUSE $\downarrow$	AURG $\uparrow$	AUSE $\downarrow$	AURG $\uparrow$
Baseline + ADF	0.00685	0.00945	98.17	–	–	–	–	–	–
+ $L_{1}$	0.00794	0.01038	97.70	0.00021	0.00055	0.00036	0.00075	0.00040	0.00259
+ $L_{2}$	0.00642	0.00924	98.26	0.00059	0.00003	0.00090	0.00011	0.00200	0.00020
+ Grad	0.00653	0.00921	98.26	0.00021	0.00041	0.00028	0.00073	0.00043	0.00181
+ $L_{1}$ + Grad (Ours)	0.00672	0.00898	98.32	0.00019	0.00046	0.00024	0.00073	0.00032	0.00188

TABLE VIII: Effect of different error proxy

\mathcal{E}

on depth estimation and uncertainty estimation performance on DREDS-CatKnown [ 11 ] . Metrics are computed over Object pixels.

Model Variant	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank
Model Variant	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
Plain refinement	0.0365	78.05	0.0346	85.14	0.0221	86.98	0.0566	72.23	0.0295	91.30	0.0149	95.33	0.0090	97.87	0.0124	98.17	4.00
Residual gate	0.0393	75.92	0.0391	81.85	0.0228	86.24	0.0682	66.67	0.0267	93.31	0.0148	94.67	0.0065	99.38	0.0116	98.48	3.88
Simple conv fusion	0.0351	82.36	0.0330	85.80	0.0214	87.75	0.0727	64.39	0.0223	94.11	0.0144	95.07	0.0057	99.59	0.0114	98.47	2.50
UCFC-PM	0.0345	79.46	0.0322	86.83	0.0220	87.43	0.0545	72.47	0.0404	79.91	0.0146	95.40	0.0081	99.01	0.0121	98.54	2.75
UCFC (Ours)	0.0326	82.38	0.0334	86.86	0.0199	89.52	0.0701	67.28	0.0262	93.94	0.0137	95.73	0.0069	98.35	0.0103	98.74	1.88

TABLE IX: Ablation of UCFC feedback. Metrics are computed over Object pixels.

Model Variant	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank
Model Variant	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
Vanilla (w/o SN)	0.0349	80.84	0.0424	83.05	0.0221	87.13	0.0671	66.67	0.0252	93.60	0.0120	96.32	0.0078	99.20	0.0199	90.36	3.25
SN-Median	0.0327	83.19	0.0351	86.84	0.0198	89.46	0.0657	69.16	0.0285	93.85	0.0159	95.45	0.0077	98.90	0.0109	98.49	2.63
SN-LogBias	0.0343	81.52	0.0350	85.68	0.0204	88.95	0.0648	69.21	0.0255	94.01	0.0175	95.64	0.0073	99.06	0.0105	98.66	2.44
SN-LogBias-Robust	0.0324	82.96	0.0345	86.09	0.0193	89.66	0.0677	67.20	0.0259	94.35	0.0143	95.83	0.0063	99.25	0.0096	99.12	1.69

TABLE X: Effect of different scale normalization models. Metrics are computed over Object pixels.

Fig. 8: Effect of UCFC on predicted uncertainty. We show the RGB input and the uncertainty maps predicted without and with UCFC. With UCFC, the uncertainty map is less noisy and exhibits more coherent high-uncertainty responses near transparent surfaces and depth discontinuities. All uncertainty maps use identical visualization scaling.

Fig. 9: Training REL curves on HAMMER ( left ) and DREDS-CatKnown ( right ). UCFC yields lower training error and smoother convergence. Curves are shown with a moving average for readability.

Fig. 10: Visualization of the two–stage robust scale normalization on T-LESS [ 29 ] . (a) RGB input. (b) Absolute error of sensor depth (brighter indicates larger errors). (c) Consistency weight map $w(x,y)$ , where high values (green) keep reliable sensor pixels and low values (purple) suppress inconsistent measurements. The red arrow highlights a multi-object junction where sensor depth exhibits large errors and is therefore down-weighted by $w(x,y)$ . (d) Per-pixel absolute error comparison. blue indicates SN-LogBias-Robust has lower error, red indicates SN-LogBias has lower error.

	w/o Unc		w/ Unc
Dataset	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	AUSE(REL) $\downarrow$
Scene pixels
HAMMER	0.0398	79.05	0.0379 ( $\downarrow$ 4.77%)	80.98 ( $\uparrow$ 2.44%)	0.00131
TransCG	0.0115	94.55	0.0085 ( $\downarrow$ 26.09%)	96.81 ( $\uparrow$ 2.39%)	0.00069
ROBI	0.0053	96.94	0.0024 ( $\downarrow$ 54.72%)	99.61 ( $\uparrow$ 2.75%)	0.00021
Object pixels

TABLE XI: Effect of uncertainty modeling on zero-shot benchmarks. values in red denote improvements.

(a) Overall methods Method Time (ms) SwinDR [ 11 ] 73.0 PromptDA [ 21 ] 103.4 OMNI-DC [ 20 ] 277.0 PriorDA [ 22 ] 297.0 CDM [ 15 ] 383.4 D 3 RoMa [ 14 ] 2116.6 Ours 225.6

(b) Deployment cost Item Value Params 349.2M Trainable params 13.8M Checkpoint size 1.43 GB FLOPs 1367.2G Infer. peak VRAM 2.09 GB Train peak VRAM 19.4 GB/GPU Training cost 24 GPU-days

(c) Latency breakdown Component Time (ms) DAV2 backbone 118.5 Robust SN 6.1 Context networks 57.4 ConvGRU 43.5 Total 225.6

TABLE XII: Inference time and deployment-cost. (a) End-to-end latency of different methods. (b) Model size, FLOPs, memory, and training cost of UA-Depth. (c) Runtime breakdown of UA-Depth.

Depth Type	$\text{3D}_{\text{25}}$	$\text{3D}_{\text{50}}$	$\text{3D}_{\text{75}}$	$5^{\circ}$ 2cm	$5^{\circ}$ 5cm	$10^{\circ}$ 2cm	$10^{\circ}$ 5cm
Raw	77.3	52.6	17.2	6.7	8.9	17.6	25.4
SwinDR [ 11 ]	77.3	57.8	21.9	7.5	9.9	20.9	30.1
D 3 RoMa [ 14 ]	71.7	42.6	12.4	2.8	4.6	10.6	17.9
CDM [ 15 ]	80.7	58.4	18.4	10.0	14.7	22.4	35.6
Ours	79.9 $\pm 3.1$	60.5 $\pm 4.3$	30.3 $\pm 2.8$	13.3 $\pm 0.6$	15.0 $\pm 0.5$	28.9 $\pm 1.9$	36.7 $\pm 2.1$
Ours w/ Unc - Sampling	79.4 $\pm 1.1$	60.2 $\pm 1.5$	31.9 $\pm 2.3$	14.3 $\pm 1.8$	16.4 $\pm 1.9$	30.8 $\pm 2.1$	39.7 $\pm 2.8$

TABLE XIII: Category-level 6D pose accuracy of VI-Net [ 62 ] on HouseCat6D [ 27 ] using different depth inputs.

Fig. 11: VI-Net [ 62 ] results on HouseCat6D [ 27 ] . From left to right: using raw sensor depth, our restored depth, and restored depth with uncertainty-guided sampling. GT poses are shown as green 3D bounding boxes and predictions as red boxes. Red arrows highlight improved pose estimation on transparent and reflective objects when using restored depth and uncertainty-guided sampling.

Depth Type	Seen			Similar			Novel			Mean AP
Depth Type	AP	AP 0.8	AP 0.4	AP	AP 0.8	AP 0.4	AP	AP 0.8	AP 0.4	Mean AP
Raw	23.15	27.73	19.81	17.97	21.86	15.15	6.48	8.05	3.36	15.87
SwinDR [ 11 ]	28.95	35.45	22.41	22.51	27.82	17.32	10.20	12.64	4.70	20.55
$\text{D}^{3}\text{RoMa}$ (RGB - D) [ 14 ]	26.52	33.86	16.19	22.45	28.38	15.02	9.53	11.59	3.79	19.50
$\text{D}^{3}\text{RoMa}$ (RGB - D Stereo) [ 14 ]	33.01	40.31	26.32	25.82	31.91	19.63	10.89	13.41	4.92	23.24
CDM [ 15 ]	28.28	35.66	20.42	24.56	29.27	17.62	9.78	11.80	5.52	20.87

TABLE XIV: 6D grasp synthesis performance of AnyGrasp [ 6 ] on GraspNet-Trans under different depth inputs.

Fig. 12: Robot platform and the full set of test objects used for real-world evaluation of 6D grasp synthesis. Our test set emphasizes challenging household items with transparent and reflective surfaces.

Input Depth	Success Rate $\uparrow$	Completion Rate $\uparrow$
Raw	54.42% $\pm$ 9.49	88.00% $\pm$ 6.50
D 3 RoMa (RGB-D-Stereo)	69.05% $\pm$ 6.99	96.00% $\pm$ 3.50
CDM (RGB-D)	72.73% $\pm$ 6.07	98.00% $\pm$ 2.50
Ours (RGB-D)	76.65% $\pm$ 5.12	99.00% $\pm$ 1.50
Ours + Unc-Reweighting (RGB-D)	71.08% $\pm$ 6.74	94.50% $\pm$ 4.00

TABLE XV: Real-robot pick-and-place results on the same 20 fixed scenes. We report scene-level mean

\pm

95% confidence-interval half-width. Our method and CDM use RGB-D inputs, while D 3 RoMa uses RGB-D with active infrared stereo.

Fig. 13: Failure case of uncertainty-guided reweighting: the predicted uncertainty is locally low on an off-center region of a multi-material object (highlighted), biasing the selection toward an unstable off-center grasp.

Fig. 14: Execution sequences of real-world manipulation tasks: the top row shows pick-and-place, and the bottom row shows a long-horizon bimanual pouring demo.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. A-1: Material distribution across datasets used for zero-shot evaluation. Best view in color.

Dataset	$1^{\circ}$ , 1mm	$2^{\circ}$ , 2mm	$5^{\circ}$ , 5mm
YCB-V	0.0111 / 0.0100	0.0190 / 0.0170	0.0356 / 0.0314
T-LESS	0.0136 / 0.0119	0.0219 / 0.0191	0.0385 / 0.0326

Appendix

TABLE A-I: Rendered depth robustness analysis on YCB-V [ 28 ] and T-LESS [ 29 ] . We report REL between depths rendered from the ground truth poses and perturbed poses at three pose noise levels. Values are shown as mean / median.

Split	Dataset	#Cat.	Canonical Categories
Train	DREDS-CatKnown	15	airplane, bag, bottle, bowl, camera, can, car, clock, faucet, guitar, hat, knife, lamp, mug, phone
Train	HISS	11	airplane, bag, bottle, bowl, camera, can, car, guitar, hat, mug, phone
Train	ClearPose	20	beaker, bottle, bowl, container, cup, drinking glass, dropper, flask, fork, funnel, graduated cylinder, knife, mug, pan, pitcher, plate, reagent bottle, rod, spoon, syringe
Test	HAMMER	12	bottle, box, can, cup, jar, knife, remote control, shoe, spoon, teapot, trophy, tube
Test	HouseCat6D	12	bottle, box, can, cup, drinking glass, fork, knife, remote control, shoe, spoon, teapot, tube
Test	TransCG	10	bottle, bowl, can, cup, drinking glass, fruit dish, jelly container, soap container, spoon, wash container

Appendix

TABLE A-II: Canonical category inventory of the training and zero-shot testing datasets.

Dataset	#Obj.	Exact Overlap	Category Cov.	Material Cov.	Category–Material Cov.
HAMMER	20	0/20	7/20 (35.0%)	20/20 (100.0%)	4/20 (20.0%)
HouseCat6D	50	0/50	25/50 (50.0%)	50/50 (100.0%)	17/50 (34.0%)
TransCG	46	0/46	33/46 (71.7%)	46/46 (100.0%)	29/46 (63.0%)
GraspNet-Trans	76	0/76	15/76 (19.7%)	76/76 (100.0%)	10/76 (13.2%)
YCB-V	21	0/21	9/21 (42.9%)	21/21 (100.0%)	8/21 (38.1%)
T-LESS	30	0/30	0/30 (0.0%)	30/30 (100.0%)	0/30 (0.0%)

Appendix

TABLE A-III: Object-level transfer audit for zero-shot evaluation. Category coverage counts target objects whose canonical category appears in the training mixture. Category–material coverage additionally requires the target object’s material regime to be represented for the same category in the training mixture.

Method	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}\uparrow$	$\delta_{1.10}\uparrow$	$\delta_{1.25}\uparrow$
SwinDR [ 11 ]	0.0503	0.0403	0.0321	74.23	83.61	98.70
$\text{D}^{3}\text{RoMa}$ [ 14 ]	0.0592	0.0544	0.0348	68.43	84.95	96.81
DP [ 36 ]	0.0484	0.0445	0.0271	70.59	87.95	98.64
OGNI-DC [ 19 ]	0.0506	0.0515	0.0302	67.35	86.86	98.32
PromptDA [ 21 ]	0.0500	0.0532	0.0308	63.05	86.84	98.42
PPFT [ 58 ]	0.0389	0.0435	0.0241	78.36	87.76	98.80

Appendix

TABLE A-V: Quantitative results on HAMMER [ 26 ] . Metrics are computed over object pixels.

Methods	RMSE $\downarrow$	REL $\downarrow$	MAE $\downarrow$	$\delta_{1.05}\uparrow$	$\delta_{1.10}\uparrow$	$\delta_{1.25}\uparrow$
NLSPN [ 35 ]	0.026	0.039	0.015	78.90	89.02	97.86
LIDF [ 59 ]	0.082	0.183	0.069	23.70	42.77	75.44
SwinDR [ 11 ]	0.022	0.034	0.013	81.90	92.18	98.39
$\text{D}^{3}$ RoMa [ 14 ]	0.0247	0.0358	0.0141	79.56	89.94	98.29
PromptDA [ 21 ]	0.0463	0.0983	0.0365	42.79	66.71	91.75
Ours	0.0220 $\pm 0.0003$	0.0322 $\pm 0.0004$	0.0123 $\pm 0.0001$	82.30 $\pm 0.27$	91.82 $\pm 0.12$	98.37 $\pm 0.05$

Appendix

TABLE A-VI: Quantitative results on DREDS-CatNovel [ 11 ] . Metrics are computed over object pixels.

Method	HAMMER		TransCG		GraspNet-Trans		XYZ-IBD		ROBI		Mean Rank
Method	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
SwinDR [ 11 ]	0.0492	72.81	0.0138	94.26	0.0438	85.96	0.0283	84.01	0.0099	95.05	3.3
$\text{D}^{3}\text{RoMa}$ [ 14 ]	0.0804	54.46	0.0330	81.70	0.0546	81.05	0.0664	50.28	0.0347	76.97	5.4
PromptDA [ 21 ]	0.3642	33.42	0.1250	54.73	0.2083	56.18	0.2998	32.94	0.2000	67.97	7.0
OMNI-DC [ 20 ]	0.0665	68.65	0.0121	95.54	0.0752	81.35	0.0055	98.56	0.0057	99.52	2.8
PriorDA [ 22 ]	0.0670	49.79	0.0371	74.70	0.0690	76.05	0.0312	82.69	0.0286	79.63	5.4

Appendix

TABLE A-VII: Zero-shot generalization results on all valid pixels. We report REL

\downarrow

and

\delta_{1.05}\uparrow

on five datasets. Bold and underlined entries indicate the best and second-best results, respectively.

Dataset	Method	RMSE $\downarrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$
XYZ-IBD	CDM-D435	0.0128	0.0145	96.60
	CDM-L515	0.0146	0.0172	96.47
	CDM-Kinect	0.0163	0.0182	93.73
	Ours	0.0067	0.0063	99.25
YCB-V	CDM-D435	0.1216	0.0575	67.17
	CDM-L515	0.1202	0.0501	75.19

Appendix

TABLE A-VIII: Comparison of CDM variants and UA-Depth on object pixels for datasets without dedicated CDM models.

Sensor	Method	RMSE $\downarrow$	REL $\downarrow$	$\delta_{1.05}$ $\uparrow$
D435 (IR Stereo)	OMNI-DC	0.0616	0.0586	67.47
	CDM-D435	0.0497	0.0427	77.95
	Ours	0.0338	0.0324	82.96
L515 (D-ToF)	OMNI-DC	0.0661	0.0655	66.61
	CDM-L515	0.0438	0.0407	77.48
	Ours	0.0387	0.0320	83.83

Appendix

TABLE A-IX: Cross-sensor results on HAMMER. Metrics are computed over object pixels.

Panel of Figure 38 — Fig. A-2: Comparison between UA-Depth and OMNI-DC on object regions across eight datasets, reporting REL error separately on sensor-valid pixels (pixels with reported depth measurements) and missing pixels (pixels where the sensor fails to return depth).

Method	Input Res.	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI
Method	Input Res.	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$
SwinDR	518 $\times$ 518	0.0429	76.35	0.0463	76.84	0.0307	81.10	0.1078	61.75	0.0241	88.82	0.0151	93.53	0.0247	85.19	0.0318	77.95
D 3 RoMa	518 $\times$ 518	0.0655	58.43	0.0498	76.45	0.0419	67.56	0.1231	50.27	0.0494	81.61	0.0511	73.36	0.0650	47.27	0.0575	50.07
Ours	518 $\times$ 518	0.0324	82.96	0.0345	86.09	0.0193	89.66	0.0677	67.20	0.0259	94.35	0.0143	95.83	0.0063	99.25	0.0096	99.12
SwinDR	360 $\times$ 640	0.0394	77.85	0.0441	76.90	0.0242	83.30	0.1017	62.09	0.0169	90.62	0.0134	93.87	0.0257	82.34	0.0393	66.71
D 3 RoMa	360 $\times$ 640	0.0393	75.16	0.0414	79.59	0.0505	58.26	0.0744	60.34	0.0625	73.53	0.0453	77.02	0.0869	28.59	0.0895	24.07

Appendix

TABLE A-X: Effect of input resolution for SwinDR, D 3 RoMa, and UA-Depth on zero-shot benchmarks on object pixels. We report REL

\downarrow

and

\delta_{1.05}\uparrow

over object pixels. All methods are evaluated at 320

\times

180; only the input resizing differs.

Method	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank	$\Delta$ Rank
Method	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank	$\Delta$ Rank
SwinDR	0.0401	75.53	0.0503	74.77	0.0200	90.41	0.0815	64.81	0.0182	91.41	0.0175	92.35	0.0255	84.72	0.0346	74.21	3.1	-0.06
$\text{D}^{3}\text{RoMa}$	0.0393	75.18	0.0414	79.59	0.0505	58.26	0.0744	60.34	0.0625	73.53	0.0453	77.02	0.0869	28.59	0.0895	24.07	4.8	-0.50
PromptDA	0.5383	6.70	0.3792	20.16	0.2960	14.69	0.3474	10.99	0.4364	14.96	0.1390	59.47	0.3373	33.47	0.7275	6.27	6.9	-0.06
OMNI-DC	0.0585	67.84	0.0523	72.14	0.0356	79.14	0.1640	51.88	0.0214	94.59	0.0253	91.53	0.0047	99.03	0.0065	99.34	3.3	+0.25
PriorDA	0.0762	41.41	0.0720	45.51	0.0418	71.36	0.1492	46.55	0.0338	87.47	0.0420	73.41	0.0205	95.26	0.0182	95.15	5.0	+0.06

Appendix

TABLE A-XI: Official-resolution zero-shot generalization results on object pixels. For each baseline, inference follows the official/preferred input-resolution setting. All predictions are evaluated at

320\times 180

\Delta

Rank denotes the change in mean rank relative to the standardized-resolution protocol in main paper.

Model Variant	Unc. Sup.	Feedback Signal	Fusion Type
Single-step depth	–	–	–
Single-step depth + Unc.	✓	–	–
Plain refinement	–	–	–
Uncertainty-only	✓	–	–
Static learned gate	✓	Learned static	Gate
Validity-mask gate	✓	Valid mask	Gate

Appendix

TABLE A-XII: Design comparison of ablation variants for UCFC feedback.

Model Variant	Object Pixels			All-valid Pixels
Model Variant	REL $\downarrow$	$\delta_{1.05}$ $\uparrow$	REL-AUSE $\downarrow$	REL $\downarrow$	$\delta_{1.05}$ $\uparrow$	REL-AUSE $\downarrow$
Single-step depth	0.0341	80.60	–	0.0324	83.67	–
Single-step depth + Unc.	0.0372	76.19	0.00152	0.0346	80.95	0.00105
Plain refinement	0.0320	82.55	–	0.0259	88.15	–
Uncertainty-only	0.0340	80.16	0.00132	0.0316	83.85	0.00111
Static learned gate	0.0292	85.53	0.00117	0.0264	87.69	0.00097

Appendix

TABLE A-XIII: In-domain HAMMER ablation of UCFC feedback. REL-AUSE is computed using REL as the pixel-wise error.

ADF+Robust-SN	UCFC	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank
ADF+Robust-SN	UCFC	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
–	–	0.0372	78.83	0.0355	84.25	0.0214	87.69	0.0773	64.53	0.0227	92.64	0.0161	95.17	0.0060	98.51	0.0106	98.32	3.41
–	✓	0.0349	81.50	0.0347	85.16	0.0206	88.56	0.0646	69.42	0.0234	94.06	0.0145	95.35	0.0054	99.50	0.0108	98.59	1.94
✓	–	0.0334	80.56	0.0345	84.62	0.0208	87.70	0.0665	68.32	0.0230	91.44	0.0153	95.32	0.0060	99.43	0.0113	98.01	2.78
✓	✓	0.0326	82.38	0.0334	86.86	0.0199	89.52	0.0701	67.28	0.0262	93.94	0.0137	95.73	0.0069	98.35	0.0103	98.74	1.88

Appendix

TABLE A-XIV: Zero-shot object-pixel evaluation on the

2\times 2

contribution decomposition of ADF+Robust-SN and UCFC. Mean rank is computed over REL and

\delta_{1.05}

across eight target datasets.

Model Variant	HAMMER		HouseCat6D		TransCG		GraspNet-Trans		YCB-V		T-LESS		XYZ-IBD		ROBI		Mean Rank
Model Variant	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	REL $\downarrow$	$\delta_{1.05}\uparrow$	Mean Rank
Single-step depth	0.0482	70.02	0.0396	75.75	0.0259	83.09	0.0779	62.12	0.0409	72.35	0.0223	89.22	0.0094	98.68	0.0147	95.83	6.38
Single-step depth + Unc.	0.0509	73.96	0.0458	81.26	0.0261	86.16	0.0808	61.82	0.0423	83.86	0.0248	92.30	0.0083	99.12	0.0155	96.52	6.19
Plain refinement	0.0365	78.05	0.0346	85.14	0.0221	86.98	0.0566	72.23	0.0295	91.30	0.0149	95.33	0.0090	97.87	0.0124	98.17	4.19
Residual gate	0.0393	75.92	0.0391	81.85	0.0228	86.24	0.0682	66.67	0.0267	93.31	0.0148	94.67	0.0065	99.38	0.0116	98.48	3.88
Simple conv fusion	0.0351	82.36	0.0330	85.80	0.0214	87.75	0.0727	64.39	0.0223	94.11	0.0144	95.07	0.0057	99.59	0.0114	98.47	2.50

Appendix

TABLE A-XV: Full Ablation of UCFC feedback. Metrics are computed over Object pixels.

Failure mode	Failed Pixels (% Obj.)	CW Rate
Biased sensor	8.4%	15.3%
Missing sensor, inaccurate init.	4.5%	7.0%
Missing sensor, accurate init.	0.2%	4.4%
Reliable sensor, prediction error	3.8%	9.9%
All failed pixels	16.9%	11.7%

Appendix

TABLE A-XVI: Failure-mode taxonomy on HAMMER object pixels.

Condition	Condition Pixels	Failed Pixels		Mean REL		Improved	Degraded
Condition	Condition Pixels	UCFC-PM	UCFC	UCFC-PM	UCFC	Improved	Degraded
Biased sensor	19.1%	10.3%	8.4%	0.0611	0.0546	4.4%	2.1%
Missing sensor, inaccurate init.	16.3%	5.5%	4.5%	0.0473	0.0453	3.8%	2.4%
Missing sensor, accurate init.	2.0%	0.2%	0.2%	0.0252	0.0267	0.2%	0.2%
Reliable sensor	62.7%	2.9%	3.8%	0.0208	0.0212	2.5%	3.1%
All object pixels	100.0%	18.8%	16.9%	0.0345	0.0326	10.8%	7.8%

Appendix

TABLE A-XVII: Condition-wise effect of recurrent uncertainty feedback on zero-shot HAMMER object pixels. Improved/Degraded denote pixels whose REL decreases/increases by more than

0.02

under UCFC relative to UCFC-PM.

Fig. A-3: Qualitative comparisons on zero-shot datasets. We show RGB, sensor depth, SwinDR, D 3 RoMa, CDM, and our absolute depth error.

Fig. A-4: UA-Depth iterative refinement visualization. We show RGB with raw depth input and the predicted depth and uncertainty at each refinement step.

Train Depth	Test Depth	IoU@25 $\uparrow$	IoU@50 $\uparrow$	IoU@75 $\uparrow$	$5^{\circ}$ 2cm $\uparrow$	$5^{\circ}$ 5cm $\uparrow$	$10^{\circ}$ 2cm $\uparrow$	$10^{\circ}$ 5cm $\uparrow$
Raw	Raw	77.3	52.6	17.2	6.7	8.9	17.6	25.4
Raw	Restored	77.9	52.2	11.4	4.4	7.1	12.4	21.9
Raw	Restored+Unc-Sampling	78.4	52.3	11.0	4.2	7.1	12.0	22.1
GT	Raw	60.6	27.0	3.0	1.2	2.8	3.6	8.0
GT	Restored	60.7	25.5	3.3	2.7	4.2	6.6	11.9
GT	Restored+Unc-Sampling	69.0	43.9	13.7	4.3	5.6	13.4	18.1

Appendix

TABLE A-XVIII: Category-level 6D pose results under different training/testing depth on HouseCat6D [ 27 ] . Results are grouped by Raw, GT, and restored-depth training; the last block reports mean

\pm

sample standard deviation over three depth seeds.

Depth Type	Seen			Similar			Novel			Mean
Depth Type	AP $\uparrow$	CR ${}_{10}\downarrow$	CR ${}_{50}\downarrow$	AP $\uparrow$	CR ${}_{10}\downarrow$	CR ${}_{50}\downarrow$	AP $\uparrow$	CR ${}_{10}\downarrow$	CR ${}_{50}\downarrow$	AP $\uparrow$	CR ${}_{10}\downarrow$	CR ${}_{50}\downarrow$
Ours (Restored)	32.67	47.32	58.74	25.48	56.32	62.87	11.26	64.31	65.10	23.14	55.98	62.23
Ours (Restored+Unc - Sampling)	31.65	48.60	59.84	24.58	57.73	63.62	10.55	65.76	61.81	22.26	57.36	61.76
Ours (Restored+Unc - Reweighting)	33.54	45.44	58.41	26.62	53.98	62.51	11.89	62.36	64.42	24.02	53.93	61.78

Appendix

TABLE A-XIX: Effects of uncertainty utilization strategies on 6D grasp synthesis on GraspNet-Trans, following the GraspNet-1B evaluation protocol [ 57 ] .

Input Depth	SR (%) $\uparrow$	CR (%) $\uparrow$
Raw	54.42 [45.33, 64.31]	88.00 [81.00, 94.00]
D 3 RoMa (RGB-D-Stereo)	69.05 [62.36, 76.34]	96.00 [92.00, 99.00]
CDM (RGB-D)	72.73 [66.55, 78.69]	98.00 [95.00, 100.00]
Ours (RGB-D)	76.65 [71.57, 81.82]	99.00 [97.00, 100.00]
Ours + Unc-Reweighting (RGB-D)	71.08 [64.21, 77.69]	94.50 [90.00, 98.00]

Appendix

TABLE A-XX: Scene-level mean SR and CR with 95% percentile-bootstrap confidence intervals for real-robot pick-and-place. Results are computed over 20 scenes with 10,000 bootstrap resamples.

Comparison	$\Delta$ SR (pp) $\uparrow$	$\Delta$ CR (pp) $\uparrow$
Ours (RGB-D) $-$ Raw	+22.23 [12.00, 32.12]	+11.00 [5.00, 18.00]
Ours (RGB-D) $-$ D 3 RoMa (RGB-D-Stereo)	+7.60 [-0.15, 14.97]	+3.00 [-1.00, 7.00]
Ours (RGB-D) $-$ CDM (RGB-D)	+3.92 [-3.09, 12.25]	+1.00 [0.00, 3.00]
Ours (RGB-D) $-$ Ours + Unc-Reweighting (RGB-D)	+5.56 [-3.37, 13.33]	+4.50 [0.00, 9.50]

Appendix

TABLE A-XXI: Paired scene-level differences in real-robot pick-and-place. Differences are computed as UA-Depth minus each baseline on the same scene. We report mean differences in percentage points (pp) and 95% percentile-bootstrap confidence intervals.

Scene	Raw	D 3 RoMa	CDM	Ours	Ours + Unc.
01	6/5/5	7/4/5	7/5/5	8/5/5	9/4/5
02	11/3/5	8/5/5	6/5/5	6/5/5	6/5/5
03	6/5/5	6/5/5	6/5/5	6/5/5	6/5/5
04	12/4/5	8/5/5	6/5/5	7/5/5	8/4/5
05	6/5/5	6/5/5	6/5/5	7/5/5	6/3/4
06	10/3/5	8/4/5	8/5/5	7/5/5	6/5/5

Appendix

TABLE A-XXII: Per-scene grasp execution counts in the real-robot evaluation.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 201 papers

Jun 24, 2026

AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior

Reliable depth perception is critical for robotic manipulation, especially for non-Lambertian objects such as transparent or highly specular surfaces, where raw depth measurements are often corrupted or missing. These failures frequently propagate to motion planning, resulting in invalid grasp poses and execution errors. We propose AISPO, a depth completion framework that improves depth reliability for manipulation in challenging sensing conditions. AISPO combines multi-scale RGB-D feature fusion with an affine-invariant shape prior to enforce geometric consistency and mitigate catastrophic depth failures. Unlike methods that focus primarily on average depth accuracy, our approach emphasizes physical plausibility and structural integrity of the predicted depth maps. Extensive benchmark evaluations demonstrate competitive performance and strong generalization to unseen objects and novel scenes. Real-world grasping experiments further show that enhanced depth reliability significantly improves manipulation success rates, particularly for transparent objects where many existing methods fail to produce physically usable depth estimates.

Zhiming Chen, Linfang Zheng, Kun Zhang +4

The Hong Kong University of Science & Technology · LimX Dynamics · The University of Hong Kong +2

Depth Completion Robotic Manipulation

Ask PDF

May 4, 2026

AnchorD: Metric Grounding of Monocular Depth Using Factor Graphs

Dense and accurate depth estimation is essential for robotic manipulation, grasping, and navigation, yet currently available depth sensors are prone to errors on transparent, specular, and general non-Lambertian surfaces. To mitigate these errors, large-scale monocular depth estimation approaches provide strong structural priors, but their predictions can be potentially skewed or mis-scaled in metric units, limiting their direct use in robotics. Thus, in this work, we propose a training-free depth grounding framework that anchors monocular depth estimation priors from a depth foundation model in raw sensor depth through factor graph optimization. Our method performs a patch-wise affine alignment, locally grounding monocular predictions in metric real-world depth while preserving fine-grained geometric structure and discontinuities. To facilitate evaluation in challenging real-world conditions, we introduce a benchmark dataset with dense scene-wide ground truth depth in the presence of non-Lambertian objects. Ground truth is obtained via matte reflection spray and multi-camera fusion, overcoming the reliance on object-only CAD-based annotations used in prior datasets. Extensive evaluations across diverse sensors and domains demonstrate consistent improvements in depth performance without any (re-)training. We make our implementation publicly available at https://anchord.cs.uni-freiburg.de.

Simon Dorer, Martin Büchner, Nick Heppert +1

Department of Computer Science, University of Freiburg, Germany. · 2Zuse School ELIZA

Metric Depth Estimation Depth Estimation Monocular Depth Estimation

Ask PDF

May 21, 2026

UfM: Uncertainty from Motion for DNN Depth Estimation Using Gaussians

Reliable uncertainty estimation is critical for deploying monocular depth deep neural networks (DNNs) in safety-critical robotic systems. Conventional uncertainty methods such as ensembles and sampling-based approaches require multiple inferences per image, incurring substantial compute and memory overhead. Moreover, uncertainty predicted from a single image misses out on measuring disagreement between predictions across views of the same region. We propose Uncertainty from Motion* (UfM*), an uncertainty estimation algorithm that measures multiview disagreement efficiently by comparing previous and current views using a compact Gaussian mixture, requiring only a single DNN inference per image. Using Gaussians to compute multiview disagreement is not only more compute- and memory-efficient than a prior approach using a point cloud, but also improves uncertainty by measuring disagreement across regions of 3D space. UfM* paired with aleatoric uncertainty improves expected calibration error by 24-28% compared to an ensemble, while requiring only 3% of the energy and 0.02% of the memory on 100 out-of-distribution ScanNet sequences. We demonstrate UfM* consumes only 63 mJ per 224x224 image while running real-time at 30 FPS on an Arm Cortex-A76 CPU onboard a miniature energy-constrained robot, highlighting that measuring multiview disagreement using Gaussians enables efficient uncertainty for resource-constrained robotic systems.

Soumya Sudhakar, Sertac Karaman, Vivienne Sze

Massachusetts Institute of Technology, Cambridge, MA 02139, USA

Uncertainty Quantification Depth Estimation Monocular Depth Estimation Uncertainty Calibration

Ask PDF

Abstract

Depth Estimation

Explore similar work

AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior

AnchorD: Metric Grounding of Monocular Depth Using Factor Graphs

UfM*: Uncertainty from Motion* for DNN Depth Estimation Using Gaussians

UfM: Uncertainty from Motion for DNN Depth Estimation Using Gaussians