ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera

Juncheng Chen; Tiancheng Lai; Xingpeng Wang; Bingxin Liao; Baozhe Zhang; Chao Xu; Yanjun Cao

cs.ROMar 21, 2026

ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera

Authors: Juncheng Chen, Tiancheng Lai, Xingpeng Wang, Bingxin Liao, Baozhe Zhang, Chao Xu, Yanjun Cao

Organizations: State Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou, China. · Huzhou Institute of Zhejiang University, Huzhou, China. · The Chinese University of Hong Kong, Shenzhen, China.

Abstract

Time-of-Flight (ToF) cameras possess compact design and high measurement precision to be applied to various robot tasks. However, their limited sensing range restricts deployment in large-scale scenarios. Depth completion has emerged as a potential solution to expand the sensing range of ToF cameras, but existing research lacks dedicated datasets and struggles to generalize to ToF measurements. In this paper, we propose a full-stack framework that enables depth completion in large-scale scenarios for short-range ToF cameras. First, we construct a multi-sensor platform with a reconstruction-based pipeline to collect real-world ToF samples with dense large-scale ground truth, yielding the first LArge-ScalE scenaRio ToF depth completion dataset (LASER-ToF). Second, we propose a sensor-aware depth completion network that incorporates a novel 3D branch with a 3D-2D Joint Propagation Pooling (JPP) module and Multimodal Cross-Covariance Attention (MXCA), enabling effective modeling of long-range relationships and efficient 3D-2D fusion under non-uniform ToF depth sparsity. Moreover, our network can utilize the sparse point cloud from visual SLAM as a supplement to ToF depth to further improve prediction accuracy. Experiments show that our method achieves an 8.6% lower mean absolute error than the second-best method, while maintaining lightweight design to support onboard deployment. Finally, to verify the system's applicability on real robots, we deploy proposed method on a quadrotor at a 10Hz runtime, enabling reliable large-scale mapping and long-range planning in challenging environments for short-range ToF cameras.

Figures & tables

Fig. 1: (a) Previous paradigms typically acquire ground truth through a depth camera and subsample it to synthesis uniform sparse inputs for training. Sensor-agnostic networks developed under this paradigm struggle to transfer to real-world ToF completion. (b) Our framework first focuses on obtaining real-world ToF sampling and generating dense depth ground truth in large-scale scenarios. Then, we propose a sensor-aware depth completion network. This framework expands the applicability of ToF cameras in large-scale robot applications in (c) .

Dataset	Scene	Ground Truth Acquisition Method	Max Reliable Range	Sequence?	ToF data?	Avg. Return Density	Resolution	Quantity
NYU-Depth V2 [ 5 ]	Indoor	Depth-Camera-Based	Near, <6m	✓	✗	68%	304*228	48,238
TOFDC [ 6 ]	In/Outdoor		Near, <6m	✗	✓	98.5%	512*384	10,560
VOID [ 12 ]	In/Outdoor		Near, <3m	✓	✗	95.1%	640*480	40,800
DenseLivox [ 13 ]	In/Outdoor	Accumulation-Based	Far, >20m	✗	✗	88.3%	Unknown	19,428
DIODE [ 14 ]	In/Outdoor			✗	✗	99.6%/66.9%	1024*768	27,858
KittiDC [ 7 ]	Outdoor			✓	✗	22%	1216*256	87,898

TABLE I: Statistics of LASER-ToF compared to other depth completion datasets.

Fig. 2: Data synchronization and dataset production pipeline.

Fig. 3: Our multi-sensor data collection platform.

Fig. 4: Dataset visualization: RGB image (input), sparse depth (input), and ground truth. Color bars on the right indicate the farthest distance of depth maps.

Fig. 5: ToFormer Network Architecture. Given an input RGBD image, a multi-scale encoder is adopted for 2D RGB-D fusion and 3D-2D cross-modal fusion. Point clouds are backprojected from RGBD image, aligned with 2D features, and fed into the 3D branch. Early fusion is conducted between the 3D feature map $X_{3d}$ and 2D feature map $X_{2d}$ through the MXCA module. Then, multi-level depth maps are upsampled by the decoder, and the full-resolution depth map $\hat{D}_{1/1}$ will be refined by the SPN module.

Fig. 6: (a) Illustration of our 3D branch design details. (b) Non local Propagation Properties of proposed 3D Branch: (1)-(2) Ground truth and sparse depth. (3)-(4) As the number of EdgeConv iterations increases, the aggregation basis of point clouds shifts from distance to geometric structure. (5) Overlayed dense activation map of $X_{3d}$ . (c) Previous 3D-2D fusion patterns of [ 22 , 23 ] .

Fig. 7: Visualized comparison of raw depth map acquired through different method. Our reconstruction-based method has significantly higher overall quality compared to the accumulation-based method.

Fig. 8: Our direct validation method, designed specifically for comparing depth map from accumulation (stationary phase) and reconstruction (moving phase).

Scene	$D_{static}$ as Ground Truth			Return Density
Scene	RMSE (m)	REL	Max Depth (m)	$D_{static}$	$D_{motion}$
Outdoor-1	0.241	0.007	32.48	67.8%	92.7%
Outdoor-2	0.108	0.009	97.41	50.5%	56.8%
Outdoor-3	0.232	0.004	21.74	78.6%	88.0%
Outdoor-4	0.079	0.008	36.58	72.3%	79.0%
Indoor-1	0.023	0.002	24.88	75.4%	80.7%

TABLE II: Quantitative evaluation of

D_{motion}

when take

D_{static}

as ground truth and return density comparison.

	RGB with ToF-Only as Input				RGB with ToF&Visual as Input
Method	RMSE $\downarrow$ (mm)	MAE $\downarrow$ (mm)	REL $\downarrow$	$\delta_{1}\uparrow$	RMSE $\downarrow$ (mm)	MAE $\downarrow$ (mm)	REL $\downarrow$	$\delta_{1}\uparrow$	Params. $\downarrow$ (M)	FLOPs $\downarrow$ (G)
NLSPN [ 8 ]	1140.49	496.24	0.0605	94.7	1033.02	412.91	0.0530	95.7	26.23	972.48
MDANet [ 17 ]	1212.54	619.98	0.0791	93.9	1102.83	521.72	0.0725	94.2	3.04	323.52
PENet [ 16 ]	1105.80	528.64	0.0674	94.5	949.98	421.33	0.0582	95.7	131.92	585.61
DySPN [ 9 ]	1170.58	547.37	0.0701	94.1	1058.74	467.00	0.0634	94.8	26.80	934.18
CFormer [ 10 ]	1086.06	526.65	0.0694	94.2	987.86	447.64	0.0630	95.0	82.51	769.88

TABLE III: Quantitative results on LASER-ToF benchmark. ( xx : best; xx : second-best. Horizontal group 1 and 3: supervised methods. Horizontal group 2: generalizable methods.)

Fig. 9: Qualitative results on the LASER-ToF benchmark. Upper part: An outdoor staircase scene, where only ToF is used as sparse depth input. Depth is available only for the few steps directly in front of the sensor. Lower part: An office scene, near-range and far-range depths are provided by ToF and visual SLAM respectively, yet the textureless white wall yields no valid depth points.

Fig. 10: Visualized comparison of geometric details for four best methods in indoor and outdoor scenarios. The zoomed-in regions show fine-grained differences.

Fig. 11: Relative mean absolute error (REL) across depth ranges. As the depth increases toward far range, all methods exhibit a rising error trend. Our method maintains the lowest error across nearly all depth ranges.

Fig. 12: Error maps comparison. (a) A large hall scene in which the far-range depth is provided by visual SLAM but contains noise. (b) A challenging scene involving outliers caused by ground reflections and multiple large missing regions. (c) The same scene as in (b) when only ToF depth is used and SLAM depth is unavailable.

Method	RMSE $\downarrow$ (m)	REL $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
CSPN [ 19 ]	0.117	0.016	99.2	99.9	100.0
GuideNet [ 54 ]	0.101	0.015	99.5	99.9	100.0
NLSPN [ 8 ]	0.092	0.012	99.6	99.9	100.0
CFormer [ 10 ]	0.091	0.012	99.6	99.9	100.0
DySPN [ 9 ]	0.090	0.012	99.6	99.9	100.0
BP-Net [ 27 ]	0.089	0.012	99.6	99.9	100.0

TABLE IV: Quantitative evaluation under uniform sparse depth setting on NYUv2.

	Base	SPN	MXCA	3D	RMSE $\downarrow$ (mm)	REL $\downarrow$	GFLOPs $\downarrow$
(A)	$\checkmark$	$\times$	$\times$	$\times$	957.15	0.0541	417.73
(B)	$\checkmark$	$\checkmark$	$\times$	$\times$	950.87	0.0526	429.82
(C)	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	942.23	0.0525	495.40
(D)	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	935.15	0.0510	483.90
(E)	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	936.39	0.0522	440.52
(F)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	924.07	0.0501	507.49

TABLE V: Ablation studies of ToFormer’s main components on LASER-ToF Depth Dataset.

	3D Branch
Interaction	Sparse-to-Dense (w/o JPP)	Dense-to-Dense (w/ JPP)
RMSE (mm)	974.32	924.07

TABLE VI: Ablation studies of the 3D branch without and with JPP module.

SLAM method	Keypoint	RMSE $\downarrow$ (mm)	MAE $\downarrow$ (mm)
ORB-SLAM [ 42 ]	Oriented FAST [ 43 ]	924.07	379.06
Vins-Mono [ 41 ]	Shi-Tomasi [ 44 ]	1025.37	435.45
Superpoint-SLAM [ 57 ]	Superpoint [ 45 ]	1000.62	408.71
No backend optimization		1068.61	540.42
Tracking degradation (50% visual points)		946.88	393.34
Tracking degradation (20% visual points)		974.10	422.63

TABLE VII: Ablation studies on the types and behaviors of SLAM.

Method	Params. $\downarrow$ (M)	FLOPs $\downarrow$ (G)	Runtime $\downarrow$ (ms)	RMSE $\downarrow$ (mm)
NLSPN [ 8 ]	26.23	219.01	22.6	1033.02
MDANet [ 17 ]	3.04	86.27	134.7	1102.83
PENet [ 16 ]	131.92	156.16	329.8	949.98
DySPN [ 9 ]	26.80	233.55	55.4	1058.74
CFormer [ 10 ]	82.51	172.08	159.6	987.86
LRRU [ 26 ]	20.84	323.74	45.5	1200.87

TABLE VIII: Computational cost comparison with previous methods at 320

\times

240 resolution.

Fig. 13: Large-scale grid mapping experiments. (a) A long corridor. (b) A circular large-scale environment.

Fig. 14: Illustration of our quadrotor’s hardware and onboard algorithm with ToF depth completion model integrated.

Fig. 15: Long-range path planning experiments across three environments with increasing complexity.

Scene I: Single Wall
ToF Depth	Energy Cost $\downarrow$ ( $m^{2}/s^{5}$ )	Traj. Length $\downarrow$ ( $m$ )	Time $\downarrow$ ( $s$ )	Avg. Speed $\uparrow$ ( $m/s$ )
w/o DC	7.87	13.29	17.26	0.77
w/ DC	5.93	10.18	12.57	0.81
Scene II: Dead End
w/o DC	-	-	-	-
w/ DC	7.51	13.00	15.48	0.84

TABLE IX: Quantitative results of the path planning experiment across three scenes.

Depth Completion

Depth completion estimates dense, metrically grounded scene geometry from sparse depth measurements, often using aligned images and sensor data.

See all 27 papers

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors
Aug 5, 2026Hakyeong Kim, Ruicheng Wang, Chengtang Yao +2Metric Depth Estimation Depth Estimation
KAIST · USTC · Microsoft Research Asia
Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration
Aug 29, 2026Yansong Du, Yutong Deng, Yuting Zhou +7Depth Estimation Uncertainty Quantification
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518060, China
Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Mar 5, 2026Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4Depth Completion Prompt Tuning
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1

Aug 5, 2026J/K move · Enter open · S save

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors

Hakyeong Kim, Ruicheng Wang, Chengtang Yao, Jiaolong Yang+1

KAIST · USTC · Microsoft Research Asia

Metric Depth Estimation Depth Estimation Depth Completion

Open paper AskPDF arXiv

Direct Time-of-Flight (dToF) sensors provide highly accurate metric depth and are more robust than indirect ToF systems in challenging real-world conditions. However, their high manufacturing cost and limited photodiode array size produce depth maps that are extremely sparse, low-resolution, and noisy, making them unsuitable for VR/XR, robotics, and 3D perception tasks that require dense metric depth. Existing monocular and depth completion methods struggle to handle the unique sampling patterns and hardware artifacts of dToF devices, and their performance often deteriorates significantly under severe sparsity or noise. We present a generalizable framework for dense metric depth completion from sparse dToF measurements, capable of operating across diverse sensor types, sparsity levels, and noise conditions. Our model employs a depth-guided dual-branch Vision Transformer encoder that processes RGB images and sparse dToF measurements separately, while a masked joint attention module allows depth tokens to reliably guide image features without being overwritten by them. A lightweight decoder reconstructs dense metric depth efficiently, without diffusion-based or refinement-heavy post-processing. To address the scarcity of paired training data, we introduce a comprehensive dToF simulation pipeline that reproduces the characteristics of flash, sub-VGA flash, and rotating sensors, including hardware-induced degradation, irregular sparsity, and realistic noise distributions. Trained entirely on synthetic data, our model achieves strong zero-shot generalization across 6 datasets and 3 real dToF devices, outperforming state-of-the-art approaches in both accuracy and computational efficiency. This establishes a robust and practical solution for dense metric depth completion from sparse direct ToF sensors. Our code and models are open-sourced. See https://vclab.kaist.ac.kr/cvpr2026p3.

Abstract

Depth Completion

Explore similar work

Dense Metric Depth Completion from Sparse Direct Time-of-Flight Sensors

Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration

Any to Full: Prompting Depth Anything for Depth Completion in One Stage