Accurate 6-DoF object pose tracking is essential for robotic interaction, yet conventional monocular pipelines frequently suffer from catastrophic divergence under rapid object motion. We present RoPT, a robust tracking framework engineered to overcome this limitation by unifying visual perception, temporal kinematics, and global re-localization into a tightly coupled closed loop. Unlike existing open-loop trackers, RoPT delivers three critical capabilities that fundamentally alter system resilience. First, the visual perception component tracks object masks to robustly handle partial and severe occlusions. Second, a kinematic factor graph regularizes frame-to-frame updates, forecasts forward pose predictions, and drives a parallel multi-hypothesis verification scheme to bypass local minima. Finally, an active re-localization module transforms the pipeline into a closed-loop system, making it failure-aware and capable of deterministic self-recovery. We demonstrate the effectiveness of our framework across challenging benchmarks alongside real-world robotic demonstrations, achieving significantly higher tracking survivability under rapid motion.
Figures & tables
Figure 1: Tracking robustness evaluation on YCBInEOAT.
Figure 2: An Overview of the RoPT pipeline. The system integrates a mask-guided visual perception frontend with a continuous kinematic factor graph that regularizes updates and generates multi-hypothesis pose forecasts. To ensure robustness against rapid dynamics, the pipeline routes refined candidates through a pose scoring module. If catastrophic tracking loss occurs, the failure-aware self-recovery module synthesizes a virtual CAD rendering and establishes 3D-2D correspondences with the incoming RGB frame, seamlessly healing the trajectory.
Figure 3: Visualization of the global lighting optimization. It adapts the static rendering to match real-world environmental lighting.
Figure 4: Pose recovery via feature matching.
Mean 2.1 mm translation and 1.4∘ rotation per frame (Original Framerate).
Approach
cracker box
bleach cleanser
sugar box
tomato soup can
mustard bottle
All
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
TrackNet
87.36
90.92
92.19
95.73
89.75
92.82
92.12
95.57
98.63
98.76
91.34
94.28
RGBTrack
1.38
5.98
29.68
54.06
66.99
81.17
71.53
85.54
80.15
89.42
64.41
79.40
FPose
88.36
92.25
93.17
96.35
95.01
97.19
90.65
95.04
96.14
97.16
91.96
95.16
FPose++
88.26
91.95
92.93
96.02
94.09
96.69
90.47
94.99
96.24
97.22
91.70
94.94
Table 1: Quantitative robustness and accuracy comparison on the YCBInEOAT benchmark.
Method
TrackNet
RGBTrack
FoundationPose
FoundationPose++
RoPT
Runtime
30.1 ms (33.2 fps)
37.1 ms (27.0 fps)
23.3 ms (42.9 fps)
35.7 ms (28.0fps)
53.7 ms (18.6 fps)
Table 2: Quantitative tracking runtime comparison on the YCBInEOAT benchmark.
Setting
Original Framerate
2× Downsampling
4× Downsampling
8× Downsampling
16× Downsampling
Mean
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
Runtime
V0: FoundationPose (baseline)
91.97
95.16
91.94
95.16
65.81
71.56
56.74
61.10
37.97
41.02
68.89
72.80
23.3 ms
V1: V0 + Tracking Mask-Guided
69.91
89.85
70.18
89.29
71.45
89.87
77.50
89.39
72.21
89.95
72.25
89.67
38.6 ms
V2: V1 + Kinematic Factor Graph
91.96
95.15
91.95
95.17
91.35
95.06
91.68
95.15
81.76
91.06
89.74
94.32
44.1 ms
V3: V2 + Global Lighting Opt.
92.03
95.17
92.05
95.20
91.56
95.13
91.78
95.22
81.68
91.13
89.82
94.37
48.1 ms
V4: V3 + Self-Recovery (ours)
92.02
95.14
92.05
95.20
91.58
95.13
91.64
95.05
90.85
94.55
91.63
95.02
53.7 ms
Table 3: Ablation on the contributions of individual component on the YCBInEOAT benchmark.
Figure 5: Qualitative tracking comparison with state-of-the-art papers on the YCBInEOAT dataset.
Figure 6: Two real-world pose tracking experiments: Grasping and Tracking Benchmark (Supp.).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The relationship between the score function and the translation and rotation changes of the groundtruth. When the objects have rapid motions, it will be reflected in the score functions. We visualized the τs in red dash lines.
Video
48
49
50
51
52
53
54
55
56
57
58
59
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
A
A -S
FPose
21.31
41.36
71.47
82.01
80.91
88.98
96.01
97.67
86.59
91.07
53.46
69.46
72.75
75.99
56.25
58.69
96.22
97.93
72.55
76.75
53.85
60.63
22.28
33.04
FPose++
77.87
96.04
75.90
96.79
94.98
97.09
83.95
92.25
96.84
98.24
73.64
97.99
86.07
97.83
94.93
97.55
96.42
97.96
79.85
97.85
97.74
98.49
78.24
97.98
RoPT (ours)
92.76
96.95
95.71
97.49
89.55
91.08
97.03
98.15
96.63
98.32
78.05
98.00
96.96
98.12
96.43
98.15
96.37
98.01
91.56
97.91
97.84
98.60
96.76
98.13
Appendix
Table 4: Quantitative Comparisons on the YCB-Video dataset.
Figure 8: Qualitative comparions on YCB-Video dataset. FoundationPose often lose the object in tracking when camera has a rapid motion.
Peak Motion Speed
10cm/s
15cm/s
20cm/s
Average
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
ADD
ADD-S
RGBTrack
9.97
23.17
5.59
15.61
11.10
31.53
8.89
23.44
FPose
58.88
80.67
51.97
66.89
51.10
78.96
53.98
75.51
FPose++
61.01
79.39
52.65
75.93
53.98
82.12
55.88
79.14
RoPT (ours)
61.26
85.36
61.78
85.77
57.37
85.85
60.13
85.66
Appendix
Table 5: Quantitative Comparisons on the new Pose Tracking dataset.
Figure 9: This figure details qualitative results on the new pose tracking benchmark, where a robotic arm controls the object at real-world speeds. Existing methods frequently struggle to estimate correct translation and orientation.
Figure 10: Failure-Awareness and Pose Recovery Evaluation. Unlike existing open-loop methods that fail to detect unreliable estimates and cannot recover tracking post-occlusion, our method demonstrates both robust failure awareness and self-recovery. (a) Partial occlusion. (b) Approach: An obstacle approaches the target object. (c) Occlusion: The target object is fully occluded by the obstacle. Existing open-loop methods output erratic, unreliable poses, whereas our tracker correctly signals a tracking failure. (d) Pose Recovery: Upon the object’s reappearance, our method successfully re-establishes and recovers the accurate 6-DoF pose.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China · College of Future Information Technology, Fudan University, Shanghai 200433, China
Woven by Toyota, Inc. · University of Bonn, Germany. · University of Bonn, Center for Robotics, and the Lamarr Institute for Machine Learning and Artificial Intelligence.
Event-driven Perception for Robotics, Istituto Italiano di Tecnologia, Italy · Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Italy · University of Genoa, Genoa, Italy