Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

Tzu-Yuan Lin; Ho Jae Lee; Kevin Doherty; Yonghyeon Lee; Sangbae Kim

cs.CVApr 12, 2026

Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

Authors: Tzu-Yuan Lin, Ho Jae Lee, Kevin Doherty, Yonghyeon Lee, Sangbae Kim

Organizations: Massachusetts Institute of Technology, USA · Boston Dynamics, USA

Abstract

We present Point2Pose, a model-free method for causal 6D pose tracking of multiple rigid objects from monocular RGB-D video. Initialized only from sparse image points on the objects, our approach tracks multiple unseen objects without requiring object CAD models or category priors. Point2Pose leverages a 2D point tracker to obtain long-range correspondences, enabling instant recovery after complete occlusion. Simultaneously, the system incrementally reconstructs an online Truncated Signed Distance Function (TSDF) representation of the tracked targets. Alongside the method, we introduce a new multi-object tracking dataset comprising both simulation and real-world sequences, with motion-capture ground truth for evaluation. Experiments show that Point2Pose trades some single-object pose accuracy for broader model-free tracking capabilities, including multi-object tracking and recovery from complete occlusion. Project page: https://point2pose.github.io/.

Figures & tables

Figure 1: Point2Pose tracks multiple unknown objects through severe hand-object occlusion and recovers their poses when they reappear, without requiring object CAD models.

Figure 2: Overview of the Point2Pose framework. Starting from RGB-D frames and object masks, sparse object keypoints are tracked over time and used to register each frame to an object-centric map. The segmented depth observations are fused into TSDF maps for reconstruction and pose refinement, while pose-graph optimization maintains consistent trajectories across keyframes.

Panel of Figure 3 — Figure 3: Example from the YCBMultiTrack-Synthetic dataset: a two-object sequence (toy airplane and tomato soup can) with circular motion and mutual occlusions.

Figure 4: Texture analysis on HO3D pitcher sequences. (a) Comparison of a low-texture view ( AP12 ) and a view with stronger local texture ( AP14 ). White and cyan boxes show ground-truth and estimated poses. In AP12 , many high-error tracks occur on largely textureless surfaces. In contrast, the visible labels and cap in AP14 provide stronger local texture, leading to more reliable low-error tracks, resulting in better pose alignment. (b) Mean Sobel gradient magnitude is used as a proxy for visible texture. Sequences with weaker texture generally exhibit larger 2D point-tracking errors and lower ADD AUC, suggesting that reduced correspondence reliability degrades downstream pose tracking.

Metric	Setting	Method	AP10	AP11	AP12	AP13	AP14	MPM10	MPM11	MPM12	MPM13	MPM14	SB11	SB13	SM1	Mean
ADD-S (%) $\uparrow$	CAD	FoundationPose	96.33	95.86	96.29	96.21	96.35	97.80	97.66	98.30	98.21	97.40	97.35	98.12	Fail	97.16
	CAD-free	BundleSDF	95.82	96.07	96.96	96.20	97.17	90.94	96.29	96.16	59.05	97.18	97.06	97.69	96.88	93.34
	CAD-free	Ours	93.71	96.18	84.33	96.27	96.13	95.31	97.08	97.12	96.58	96.42	94.35	97.58	89.15	94.63
ADD (%) $\uparrow$	CAD	FoundationPose	91.58	90.50	85.80	89.87	92.83	95.40	95.58	96.75	96.62	94.44	94.92	96.54	Fail	93.40
	CAD-free	BundleSDF	89.69	91.01	94.54	92.80	94.81	78.78	91.69	91.33	33.71	94.16	93.73	95.25	94.23	87.36
	CAD-free	Ours	76.19	91.12	46.64	92.36	92.39	73.99	88.16	93.86	63.19	85.56	85.99	94.90	65.95	80.79

Table 1: Results on the HO3D dataset [ 11 ] . ADD-S/ADD AUC are percentages over the

0

–

0.1\,\mathrm{m}

threshold range. CD denotes Chamfer Distance for reconstruction evaluation. FoundationPose [ 32 ] is evaluated with provided CAD models. Bold numbers compare only between CAD-free methods.

Metric	Setting	Method	bleach		cracker		mustard		sugar		tomato	Mean
Metric	Setting	Method	v1	v2	v1	v2	v1	v2	v1	v2		Mean
ADD-S (%) $\uparrow$	CAD	FoundationPose	96.78	95.59	96.41	93.30	97.70	97.00	97.10	93.79	96.35	96.00
	CAD-free	BundleSDF	94.00	95.47	95.60	91.72	96.32	96.28	96.12	90.56	fail	94.51
	CAD-free	Ours	82.88	93.90	96.37	92.50	95.26	95.65	94.31	87.60	95.52	92.67
ADD (%) $\uparrow$	CAD	FoundationPose	92.39	92.48	93.31	88.45	95.41	94.83	94.65	89.24	91.52	92.48
	CAD-free	BundleSDF	87.71	90.99	91.73	84.06	90.44	93.08	90.10	84.41	fail	89.07

Table 2: Results on the YCBInEOAT dataset [ 30 ] . ADD-S/ADD AUC are percentages over the

0

–

0.1\,\mathrm{m}

threshold range. FoundationPose [ 32 ] is evaluated with provided CAD models. Bold numbers compare only between CAD-free methods.

Figure 5: Results on a fully occluded scene from YCBMultiTrack-Real . After the object becomes completely occluded and later re-enters the scene, our method successfully recovers the object pose, whereas BundleSDF [ 31 ] loses track and preserves a stale object orientation.

Table 3: Results on the YCBMultiTrack dataset. ADD-S/ADD AUC are reported as percentages over the

0

–

0.1\,\mathrm{m}

threshold range. FoundationPose [ 32 ] , denoted as FP, is evaluated with provided ground-truth object meshes, while BundleSDF [ 31 ] and Ours do not use object CAD models. Bold numbers compare only between methods without object CAD models.

Method	ADD-S (%) $\uparrow$	ADD (%) $\uparrow$
Proposed	95.07	82.76
No multi-hypothesis	89.81	65.52
No SDF refinement	94.40	77.37
No graph optimization	94.56	78.80

Table 4: Component-wise ablation on HO3D .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	ADD-S AUC (%) $\uparrow$		ADD AUC (%) $\uparrow$
Method	Mean	Std	Mean	Std
Proposed	95.07	3.74	82.76	15.18
Super Point Only	93.09	4.14	77.61	19.78
Uniform Sampling	91.47	5.92	72.31	17.89

Appendix

Table 5: Ablation results on keypoint sampling methods.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 187 papers

Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions
Jul 26, 2026Balázs Opra, Léo Ghafari, Thomas Stewart +1Visual Object Tracking Object Pose Estimation
Woven by Toyota, Inc. · University of Bonn, Germany. · University of Bonn, Center for Robotics, and the Lamarr Institute for Machine Learning and Artificial Intelligence.
GRC-Pose: Generation-Reconstruction Correspondence for Prior-Free 6D Object Pose Tracking
Sep 30, 2026Shiyang Liu, Weiquan Lin, Luping Xiao +4Object Pose Estimation
School of Automation, Beijing Institute of Technology · Zhongguancun Academy · Artificial Intelligence Academy, Xidian University +1
RRTrack: Robust and Recoverable Object 6D Pose Tracking for Dynamic Scenes
Jul 26, 2026Junyue Li, Ye Zheng, Yifan Chen +2Visual Object Tracking Object Pose Estimation
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China · College of Future Information Technology, Fudan University, Shanghai 200433, China

Jul 26, 2026J/K move · Enter open · S save

Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions

Balázs Opra, Léo Ghafari, Thomas Stewart, Cyrill Stachniss

Woven by Toyota, Inc. · University of Bonn, Germany. · University of Bonn, Center for Robotics, and the Lamarr Institute for Machine Learning and Artificial Intelligence.

Visual Object Tracking Object Pose Estimation

Open paper AskPDF arXiv

Real-time 6-DoF object pose tracking is essential for many robotics applications, and several approaches exist. Yet even today's approaches remain unreliable under temporary full occlusions and rapid object motions. Once tracking is lost, most methods struggle to detect the failure and recover automatically, often requiring manual re-initialization. In this paper, we address the problem of robust model-based 6-DoF tracking of unseen objects from RGB-D data, especially in scenarios with occlusion and fast motion. We propose a novel method that combines efficient learning-based keypoint matching with optimization-based alignment and introduces a novel failure detection and recovery module. Our system monitors pose reliability, detects tracking divergence or occlusions, and performs a global re-detection and pose estimation step that robustly verifies recovery candidates before resuming tracking. Our evaluation on standard tracking benchmarks and on a new dataset of occluded and fast-moving scenes shows that our method matches state-of-the-art accuracy on easy tracking sequences, maintains high tracking speed at 57.6 frames per second, and provides the most robust tracking performance under challenging conditions. Thus, we believe that our approach is a relevant step forward in robust 6-DoF object tracking from RGB-D data.

Abstract

Object Pose Estimation

Explore similar work

Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions

GRC-Pose: Generation-Reconstruction Correspondence for Prior-Free 6D Object Pose Tracking

RRTrack: Robust and Recoverable Object 6D Pose Tracking for Dynamic Scenes