Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration

Yansong Du; Yutong Deng; Yuting Zhou; Zhancong Xu; Yingjia Lu; Mengdi Wang; Feiyu Jiao; Bangyao Wang

physics.opticsAug 29, 2026

Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration

Authors: Yansong Du, Yutong Deng, Yuting Zhou, Zhancong Xu, Yingjia Lu, Mengdi Wang, Feiyu Jiao, Bangyao Wang,

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518060, China

Abstract

Indirect time-of-flight (iToF) cameras provide compact and cost-effective dense depth measurements, but their ranging accuracy is often degraded by sensor-intrinsic uncertainty under practical imaging conditions. Spatially uniform or range-only Gaussian perturbations cannot accurately reproduce the range-dependent and signal-dependent noise characteristics of real iToF measurements, leading to a synthetic-to-real gap for learning-based restoration. To address this problem, we propose a joint depth-uncertainty modeling and restoration framework for reliable iToF sensing. A sensor-intrinsic depth-uncertainty model is first developed from calibrated tap responses, returned-signal levels, and sensor noise statistics through a depth-oriented weighted least-squares formulation. The resulting pixel-wise uncertainty is used for heteroscedastic depth synthesis and uncertainty-aware restoration supervision. Based on this heteroscedastic data synthesis, we further develop a U-shaped restoration network with Depth Visual State Space (DVSS) blocks, which combine long-range state-space modeling with convolutional spatial-channel refinement for structure-preserving depth recovery. Experiments on synthetic data and measurements captured by an in-house iToF prototype validate the proposed uncertainty model under varying range and returned-signal conditions. Controlled comparisons with fixed and range-aware Gaussian noise, together with evaluations on U-Net, Restormer, and DVSS, further demonstrate that the proposed synthesis consistently benefits different restoration backbones. The complete framework achieves 40.85~dB PSNR and 2.54 mm MAE on the synthetic test set, and 35.42 dB PSNR and 4.87 mm MAE on real iToF measurements.

Figures & tables

Fig. 1 : Overall workflow of the proposed reliable iToF depth sensing framework. Sensor-intrinsic uncertainty modeling produces heteroscedastic noisy-depth observations and uncertainty-aware supervision, which are used to train the restoration network equipped with DVSS blocks for accurate and structure-preserving depth recovery.

Fig. 2 : Overview of the proposed denoising network and its core modules. The left side shows the U-shaped architecture with stacked DVSS blocks and skip connections. The right side illustrates the proposed DVSS block and its VSSB submodule.

Fig. 3 : Illustration of the proposed SCR module. A global branch and a local branch interact in a coarse-to-fine manner to jointly refine channel and spatial responses.

Fig. 4 : Experimental setup and in-house iToF imaging system, including the acquisition configuration, camera module, and measurement rig.

Panel of Figure 5 — Fig. 5 : Validation of real-camera depth uncertainty. (a) Temporal depth standard deviation over the 500-1500 mm range. (b) Depth uncertainty under varying returned-signal levels at a fixed distance. (c) Predicted versus measured depth standard deviation over all evaluated conditions.

Fig. 6 : Qualitative comparison on the synthetic test set. From left to right: rendered RGB, DVSS-Range, DeepToF, ToF-KPN, RCF, our method, and ground-truth depth. DVSS-Range denotes the same DVSS backbone trained with range-aware Gaussian noise.

Method	PSNR (dB) $\uparrow$	SSIM $\uparrow$	MSE (mm 2 ) $\downarrow$	MAE (mm) $\downarrow$
DeepToF	37.99	0.9951	158.94	6.12
ToF-KPN	39.19	0.9969	120.52	4.88
RCF	39.53	0.9972	111.41	4.63
Ours w/ $\mathcal{L}_{1}$	40.37	0.9985	91.75	2.78
Ours w/ $\mathcal{L}_{\mathrm{full}}$	40.85	0.9987	82.30	2.54

TABLE I : Quantitative comparison with existing iToF denoising methods on the synthetic test set. The proposed method consistently achieves lower reconstruction errors while maintaining higher structural fidelity than the compared baselines.

Variant	Loss	PSNR (dB) $\uparrow$	SSIM $\uparrow$	MSE (mm 2 ) $\downarrow$	MAE (mm) $\downarrow$
DVSS structure ablation with $\mathcal{L}_{1}$
w/o DWConv & SCR	$\mathcal{L}_{1}$	39.76	0.9954	105.66	3.55
w/o DWConv	$\mathcal{L}_{1}$	40.00	0.9976	99.89	3.19
w/ SE	$\mathcal{L}_{1}$	40.16	0.9983	96.35	3.06
Full DVSS	$\mathcal{L}_{1}$	40.37	0.9985	91.75	2.78
Training objective ablation with the full DVSS block

TABLE II : Ablation study on the proposed DVSS block and training objective. Different structural variants and loss configurations are evaluated to quantify their contributions to depth restoration accuracy and structural preservation.

Method	PSNR (dB) $\uparrow$	MAE (mm) $\downarrow$
Raw iToF	31.68	7.21
DeepToF	33.29	6.18
ToF-KPN	34.76	5.31
RCF	34.48	5.52
Ours	35.42	4.87

TABLE III : Quantitative comparison on real iToF measurements. PSNR and MAE are reported to evaluate the restoration accuracy of different methods under practical imaging conditions.

Fig. 7 : Real-world results of iToF depth denoising. Each row shows one captured scene. From left to right: raw iToF measurement, DeepToF, ToF-KPN, RCF, our method, and the low-noise reference obtained by high-exposure multi-frame averaging. Compared with the baselines, our method more effectively suppresses sensor noise while preserving object boundaries and planar structures.

Backbone	Training-noise model	Synthetic PSNR (dB) $\uparrow$	Synthetic MAE (mm) $\downarrow$	Real PSNR (dB) $\uparrow$	Real MAE (mm) $\downarrow$
Noise-model ablation on the DVSS backbone
DVSS	Fixed Gaussian noise	39.40	3.42	34.41	5.82
DVSS	Range-aware Gaussian noise	39.89	3.08	34.88	5.36
DVSS	Proposed heteroscedastic synthesis	40.37	2.78	35.12	5.04
Cross-backbone generalization of the proposed synthesis
U-Net	Range-aware Gaussian noise	38.56	4.18	33.67	6.34

TABLE IV : Controlled comparison of training-noise models and their cross-backbone generalization. The results quantify the effectiveness and cross-backbone consistency of the proposed heteroscedastic synthesis under the standard

\mathcal{L}_{1}

reconstruction loss.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 198 papers

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors
Aug 5, 2026Hakyeong Kim, Ruicheng Wang, Chengtang Yao +2Metric Depth Estimation Depth Estimation
KAIST · USTC · Microsoft Research Asia
ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera
Mar 21, 2026Juncheng Chen, Tiancheng Lai, Xingpeng Wang +4Depth Completion Depth Estimation
State Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou, China. · Huzhou Institute of Zhejiang University, Huzhou, China. · The Chinese University of Hong Kong, Shenzhen, China.
PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration
Oct 21, 2026Xiujian Liang, Jiacheng Liu, Mingyang Sun +4Sim-to-Real Transfer Physics-Informed Generative Modeling
Fudan University · Shanghai Inovation Institute · Zhejiang University +2

Aug 5, 2026J/K move · Enter open · S save

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors

Hakyeong Kim, Ruicheng Wang, Chengtang Yao, Jiaolong Yang+1

KAIST · USTC · Microsoft Research Asia

Metric Depth Estimation Depth Estimation Depth Completion

Open paper AskPDF arXiv

Direct Time-of-Flight (dToF) sensors provide highly accurate metric depth and are more robust than indirect ToF systems in challenging real-world conditions. However, their high manufacturing cost and limited photodiode array size produce depth maps that are extremely sparse, low-resolution, and noisy, making them unsuitable for VR/XR, robotics, and 3D perception tasks that require dense metric depth. Existing monocular and depth completion methods struggle to handle the unique sampling patterns and hardware artifacts of dToF devices, and their performance often deteriorates significantly under severe sparsity or noise. We present a generalizable framework for dense metric depth completion from sparse dToF measurements, capable of operating across diverse sensor types, sparsity levels, and noise conditions. Our model employs a depth-guided dual-branch Vision Transformer encoder that processes RGB images and sparse dToF measurements separately, while a masked joint attention module allows depth tokens to reliably guide image features without being overwritten by them. A lightweight decoder reconstructs dense metric depth efficiently, without diffusion-based or refinement-heavy post-processing. To address the scarcity of paired training data, we introduce a comprehensive dToF simulation pipeline that reproduces the characteristics of flash, sub-VGA flash, and rotating sensors, including hardware-induced degradation, irregular sparsity, and realistic noise distributions. Trained entirely on synthetic data, our model achieves strong zero-shot generalization across 6 datasets and 3 real dToF devices, outperforming state-of-the-art approaches in both accuracy and computational efficiency. This establishes a robust and practical solution for dense metric depth completion from sparse direct ToF sensors. Our code and models are open-sourced. See https://vclab.kaist.ac.kr/cvpr2026p3.

Abstract

Depth Estimation

Explore similar work

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors

ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera

PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration