We present Point2Pose, a model-free method for causal 6D pose tracking of multiple rigid objects from monocular RGB-D video. Initialized only from sparse image points on the objects, our approach tracks multiple unseen objects without requiring object CAD models or category priors. Point2Pose leverages a 2D point tracker to obtain long-range correspondences, enabling instant recovery after complete occlusion. Simultaneously, the system incrementally reconstructs an online Truncated Signed Distance Function (TSDF) representation of the tracked targets. Alongside the method, we introduce a new multi-object tracking dataset comprising both simulation and real-world sequences, with motion-capture ground truth for evaluation. Experiments show that Point2Pose trades some single-object pose accuracy for broader model-free tracking capabilities, including multi-object tracking and recovery from complete occlusion. Project page: https://point2pose.github.io/.
Figures & tables
Figure 1: Point2Pose tracks multiple unknown objects through severe hand-object occlusion and recovers their poses when they reappear, without requiring object CAD models.
Figure 2: Overview of the Point2Pose framework. Starting from RGB-D frames and object masks, sparse object keypoints are tracked over time and used to register each frame to an object-centric map. The segmented depth observations are fused into TSDF maps for reconstruction and pose refinement, while pose-graph optimization maintains consistent trajectories across keyframes.
Figure 3: Example from the YCBMultiTrack-Synthetic dataset: a two-object sequence (toy airplane and tomato soup can) with circular motion and mutual occlusions.
Figure 4: Texture analysis on HO3D pitcher sequences. (a) Comparison of a low-texture view ( AP12 ) and a view with stronger local texture ( AP14 ). White and cyan boxes show ground-truth and estimated poses. In AP12 , many high-error tracks occur on largely textureless surfaces. In contrast, the visible labels and cap in AP14 provide stronger local texture, leading to more reliable low-error tracks, resulting in better pose alignment. (b) Mean Sobel gradient magnitude is used as a proxy for visible texture. Sequences with weaker texture generally exhibit larger 2D point-tracking errors and lower ADD AUC, suggesting that reduced correspondence reliability degrades downstream pose tracking.
Metric
Setting
Method
AP10
AP11
AP12
AP13
AP14
MPM10
MPM11
MPM12
MPM13
MPM14
SB11
SB13
SM1
Mean
ADD-S (%) ↑
CAD
FoundationPose
96.33
95.86
96.29
96.21
96.35
97.80
97.66
98.30
98.21
97.40
97.35
98.12
Fail
97.16
CAD-free
BundleSDF
95.82
96.07
96.96
96.20
97.17
90.94
96.29
96.16
59.05
97.18
97.06
97.69
96.88
93.34
Ours
93.71
96.18
84.33
96.27
96.13
95.31
97.08
97.12
96.58
96.42
94.35
97.58
89.15
94.63
ADD (%) ↑
CAD
FoundationPose
91.58
90.50
85.80
89.87
92.83
95.40
95.58
96.75
96.62
94.44
94.92
96.54
Fail
93.40
CAD-free
BundleSDF
89.69
91.01
94.54
92.80
94.81
78.78
91.69
91.33
33.71
94.16
93.73
95.25
94.23
87.36
Ours
76.19
91.12
46.64
92.36
92.39
73.99
88.16
93.86
63.19
85.56
85.99
94.90
65.95
80.79
Table 1: Results on the HO3D dataset [ 11 ] . ADD-S/ADD AUC are percentages over the 0 – 0.1m threshold range. CD denotes Chamfer Distance for reconstruction evaluation. FoundationPose [ 32 ] is evaluated with provided CAD models. Bold numbers compare only between CAD-free methods.
Metric
Setting
Method
bleach
cracker
mustard
sugar
tomato
Mean
v1
v2
v1
v2
v1
v2
v1
v2
ADD-S (%) ↑
CAD
FoundationPose
96.78
95.59
96.41
93.30
97.70
97.00
97.10
93.79
96.35
96.00
CAD-free
BundleSDF
94.00
95.47
95.60
91.72
96.32
96.28
96.12
90.56
fail
94.51
Ours
82.88
93.90
96.37
92.50
95.26
95.65
94.31
87.60
95.52
92.67
ADD (%) ↑
CAD
FoundationPose
92.39
92.48
93.31
88.45
95.41
94.83
94.65
89.24
91.52
92.48
CAD-free
BundleSDF
87.71
90.99
91.73
84.06
90.44
93.08
90.10
84.41
fail
89.07
Table 2: Results on the YCBInEOAT dataset [ 30 ] . ADD-S/ADD AUC are percentages over the 0 – 0.1m threshold range. FoundationPose [ 32 ] is evaluated with provided CAD models. Bold numbers compare only between CAD-free methods.
Figure 5: Results on a fully occluded scene from YCBMultiTrack-Real . After the object becomes completely occluded and later re-enters the scene, our method successfully recovers the object pose, whereas BundleSDF [ 31 ] loses track and preserves a stale object orientation.
Table 3: Results on the YCBMultiTrack dataset. ADD-S/ADD AUC are reported as percentages over the 0 – 0.1m threshold range. FoundationPose [ 32 ] , denoted as FP, is evaluated with provided ground-truth object meshes, while BundleSDF [ 31 ] and Ours do not use object CAD models. Bold numbers compare only between methods without object CAD models.
Method
ADD-S (%) ↑
ADD (%) ↑
Proposed
95.07
82.76
No multi-hypothesis
89.81
65.52
No SDF refinement
94.40
77.37
No graph optimization
94.56
78.80
Table 4: Component-wise ablation on HO3D .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
ADD-S AUC (%) ↑
ADD AUC (%) ↑
Method
Mean
Std
Mean
Std
Proposed
95.07
3.74
82.76
15.18
Super Point Only
93.09
4.14
77.61
19.78
Uniform Sampling
91.47
5.92
72.31
17.89
Appendix
Table 5: Ablation results on keypoint sampling methods.
Real-time 6-DoF object pose tracking is essential for many robotics applications, and several approaches exist. Yet even today's approaches remain unreliable under temporary full occlusions and rapid object motions. Once tracking is lost, most methods struggle to detect the failure and recover automatically, often requiring manual re-initialization. In this paper, we address the problem of robust model-based 6-DoF tracking of unseen objects from RGB-D data, especially in scenarios with occlusion and fast motion. We propose a novel method that combines efficient learning-based keypoint matching with optimization-based alignment and introduces a novel failure detection and recovery module. Our system monitors pose reliability, detects tracking divergence or occlusions, and performs a global re-detection and pose estimation step that robustly verifies recovery candidates before resuming tracking. Our evaluation on standard tracking benchmarks and on a new dataset of occluded and fast-moving scenes shows that our method matches state-of-the-art accuracy on easy tracking sequences, maintains high tracking speed at 57.6 frames per second, and provides the most robust tracking performance under challenging conditions. Thus, we believe that our approach is a relevant step forward in robust 6-DoF object tracking from RGB-D data.
Balázs Opra, Léo Ghafari, Thomas Stewart +1
Woven by Toyota, Inc. · University of Bonn, Germany. · University of Bonn, Center for Robotics, and the Lamarr Institute for Machine Learning and Artificial Intelligence.
Prior-free 6D object pose tracking seeks to recover the trajectory of an unseen object from a single RGB video without object-specific CAD models, posed reference images, or pose annotations. Geometric foundation models provide complementary object-centric and scene-centric cues, yet SAM3D CAD is indexed by an arbitrary object-local surface parameterization, whereas reconstructed evidence is expressed in a sequence-specific world frame with partial surface coverage. To exploit this complementarity, we formulate tracking as generation-reconstruction correspondence and introduce GRC-Pose, a correspondence-based framework that combines learned correspondence prediction with robust pose estimation. Concretely, GeoCorr-Matcher estimates weighted object-scene correspondences and per-match uncertainty for each pose candidate. FGH-Solver integrates these matches through multiple robust geometric estimators and sequence-level posterior inference, while a posterior-gated memory retains only inlier-supported observations through occlusion and viewpoint change. Extensive evaluation shows that with SAM3D CAD, GRC-Pose achieves state-of-the-art Average Recall and motion retention on HOT3D, improving the latter by 58% over prior art. On classical benchmarks including YCBInEOAT and LINEMOD, it remains highly competitive.
Shiyang Liu, Weiquan Lin, Luping Xiao +4
School of Automation, Beijing Institute of Technology · Zhongguancun Academy · Artificial Intelligence Academy, Xidian University +1
Robust object 6D pose tracking is critical for robotic systems operating in dynamic and occluded scenes. Per-frame estimators are accurate but computationally expensive, while current trackers struggle with fast motion and complete occlusion due to their reliance on continuous visibility. To address these challenges, we present RRTrack, an efficient, recoverable object 6D pose tracker that enables robust tracking through fast motion and target disappearance--reappearance. RRTrack introduces a 2D--6D closed-loop tracking strategy that integrates memory-based video object segmentation (VOS) with 6D pose refinement. The 2D branch maintains target localization, and the 6D branch verifies geometric consistency before memory updates. In addition, a DINOv2-based dual-bank template matching module is developed to recover lost targets by jointly exploiting offline synthetic templates and online observation anchors while maintaining real-time efficiency. We also introduce a synthetic RGB-D benchmark comprising three robotic scenarios with fast motion and full occlusion. Experimental results on the synthetic benchmark demonstrate that RRTrack improves equal-subset mean ADD-S AR by 66.3% and ADD-S AUC by 65.7% over FoundationPose while achieving 55.2 FPS. Real-world experiments further validate the robustness of RRTrack under noisy sensing conditions. Project page: https://github.com/7kevin24/RRTrack
Junyue Li, Ye Zheng, Yifan Chen +2
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China · College of Future Information Technology, Fudan University, Shanghai 200433, China