Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong · EACV Research Center, Shenzhen Loop Area Institute · Xi’an Jiaotong University
Camera-to-robot pose estimation from a monocular image is significant for vision-based robot navigation and control. However, recovering 3D structural features from 2D observations remains a challenging task. Existing camera-to-robot pose estimation methods either rely on time-consuming iterative rendering or involve sophisticated 2D-3D registration which suffers from projection ambiguity, accumulated errors for distal joints and imbalanced learning for different joints. In this work, we propose RoPE-Flow, an end-to-end flow matching-based framework for camera-to-robot pose estimation. From the predicted 2D heatmaps, we propose a Keypoint-centered Feature Extractor to obtain positional features along with contextual representations from keypoints-centered multi-scale patches. Moreover, we introduce Graph-guided Condition Generator, which embeds a learnable adjacency map to generate condition for flow matching based on correlation among keypoints. Besides, we introduce Bone Length Loss and Ray-Depth Loss for topological consistency and scale stability, with joint-wise weighting to mitigate varied prediction difficulty across joints. The proposed method outperforms state-of-the-art methods for pose estimation across different types of robots with mutiple cameras. Besides, we establish a novel cross-scene benchmark to demonstrate the outstanding performance of RoPE-Flow on zero-shot cross-scene robot pose estimation. Meanwhile, animated visualizations are shown on the website: ropeflow.netlify.app
Figures & tables
Figure 1: Overview of our work. RoPE-Flow can reconstruct varied robot pose in diverse scenes with camera-frame 3D keypoints and joint angles estimation from monocular observation of diverse cameras even with human-robot interaction , occlusion and out-of-vision body .
Figure 2: Comparison with previous methods for camera-to-robot estimation (Top Row). Evidence for Varied Joint Difficulty is shown based on distribution of prediction errors along the joints and keypoints. Problem for end-to-end 3D keypoints generation: Projection Ambiguity , as well as potential solutions in this work is also visualized.
Figure 3: Pipeline of the proposed method.
Panda
Kuka
Baxter
Methods
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
Avg
DREAM-F*
79.5
81.3
68.9
24.4
76.1
61.9
/
/
/
/
DREAM-Q*
74.3
77.8
52.4
37.5
78.0
57.0
/
/
75.5
/
DREAM-H*
81.1
82.9
60.5
64.0
78.8
69.1
72.1
73.3
/
/
RoboPose †
79.7
82.9
70.4
77.6
74.3
70.4
73.2
80.2
32.7
71.3
HPE ‡
82.0
82.7
82.2
76.0
75.2
75.2
73.9
75.1
58.8
75.7
Table 1: Comparison of robot pose estimation on DREAM dataset using AUC on the ADD metric. *DREAM and its variants need ground-truth joint state parameters. ‡ Results with ground-truth bounding box. † Results with predicted bounding box. The Best , Second Best and Third Best results are marked by corresponding colors. Curves of ADD can be found in Appendix B.1 .
Panda
Kuka
Baxter
Methods
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
DREAM-F
785.8
552.0
> 1k
> 1k
> 1k
> 1k
/
/
/
DREAM-Q
569.0
359.0
> 1k
> 1k
27.2
> 1k
/
/
59.0
DREAM-H
518.8
325.6
56.5
> 1k
23.6
> 1k
> 1k
> 1k
/
RoboPose
21.5
7.9
24.4
14.0
23.1
1 9.4
33.5
1 9.2
15.5
RoboPose
21.4
32.2
34.3
22.3
26.0
30.1
89.0
34.2
87.9
Table 2: Comparison of Mean Average Distance (mm). The best results among all methods (including methods with given joint states ) are in bold, while the second best one is underlined. The Best and Second Best results among methods with no need of joint state parameters are marked by corresponding colors.
Panda
Kuka
Panda
Methods
Photo
DR
Photo
DR
AK
XK
RS
ORB
RoboPose
5.7
4.9
7.2
6.2
9.3
13.7
12.2
7.6
HPE
4.5
4.5
4.5
4.5
6.9
6.1
10.4
7.6
RoboPEPP
3.2
3.9
3.9
4.4
8.5
8.0
15.7
15.6
Ours
3.2
3.4
2.9
3.1
6.2
4.8
9.4
6.3
Table 3: Comparison of joint angle estimation results using average MAE (°) of joint angles. Detailed MAE on each joint can be found in Appendix B.3 . The Best and Second Best results are marked by corresponding colors.
Figure 4: Qualitative comparison on URDF-based reconstructed models. More qualitative results can be found in Appendix B.4 . Results on videos can be found in Supplementary Material .
Methods
mIoU
Dice
HPE
0.1167 ± 0.049
0.2056 ± 0.077
RoboPEPP
0.4456 ± 0.211
0.5836 ± 0.229
Ours
0.7328 ± 0.036
0.8453 ± 0.024
Table 4: Quantitative comparison on zero-shot cross-scene dataset.
Figure 5: Qualitative comparison for zero-shot cross-scene samples. At last row, we use green mask to denote the ground truth, while yellow mask represents overlapping area and red mask is the mask of reconstructed robot. More examples can be found in Appendix B.5 .
Panda
Kuka
Baxter
Ablations
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
Avg
Full Model
87.1
86.6
90.9
81.6
85.1
82.1
87.3
86.9
74.1
84.6
Keypoints-centered Feature Extraction
3 × 3 Patch
86.0
85.8
89.3
81.4
84.6
76.4
86.8
86.2
70.8
83.0
7 × 7 Patch
86.6
86.3
90.2
79.4
84.6
79.4
86.9
86.5
69.9
83.3
11 × 11 Patch
87.0
86.7
89.0
75.5
77.2
79.7
87.6
87.2
74.2
82.7
Table 5: Results of ablations on feature extraction, condition generation, flow matching and loss functions. The Best , Second Best and Third Best results are marked by corresponding colors.
Ablations
AK
XK
RS
ORB
Avg
Full Model
90.9
81.6
85.1
82.1
84.9
w/o LboneS2R
90.9
76.7
84.5
81.5
83.4
w/o Ldepth
82.5
70.1
76.5
62.0
72.8
Table 6: Results of ablations on Sim-to-Real Tuning.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Comparison of LabRoPE and DREAM.
Figure 7: Curves of ADD.
Figure 8: Qualitative Comparison of 3D Keypoints Prediction.
Franka
Photo
DR
Comp
J1
J2
J3
J4
J5
J6
Avg
J1
J2
J3
J4
J5
J6
Avg
Pose
7.7
3.5
4.3
3.4
7.3
8.1
5.7
6.1
2.7
3 .6
2.5
6.3
8.1
4.9
HPE
6.1
2.2
3.6
2.0
6.2
6.6
4 .5
6.2
2 .2
3.9
1 .9
5.9
6.6
4.5
PEPP
4 .4
1 .8
2.2
1 .8
4.4
4.8
3.2
4 .9
2.4
2.9
2.4
5 .2
5 .6
3 .9
Ours
3.2
1.7
2 .5
1.6
4 .9
5 .2
3.2
3.6
1.7
2.9
1.7
4.9
5.5
3.4
Appendix
Table 7: Comparison of mean MAE (°) on each joint of robot.
Figure 9: Supplementary qualitative results from DREAM dataset.
Figure 10: Supplementary qualitative results from LabRoPE dataset.
Figure 11: Qualitative results of sim-to-real finetuning with or without Bone Length Loss or Ray-depth Loss
Figure 12: Results from multi-scale patch embedding with single patch embedding.
Methods
mIoU
Dice
Full Model
0.7328 ± 0.036
0.8453 ± 0.024
3 × 3 Patch
0.3986 ± 0.066
0.5668 ± 0.067
7 × 7 Patch
0.4236 ± 0.067
0.5920 ± 0.067
11 × 11 Patch
0.3727 ± 0.064
0.5399 ± 0.068
No Graph
0.4064 ± 0.077
0.5736 ± 0.080
w/o Lbone
0.3929 ± 0.087
0.5576 ± 0.105
Appendix
Table 8: Ablation results of zero-shot pose estimation on LabRoPE dataset.
Figure 13: Qualitative results of ablation studies on LabRoPE dataset.
Flow Matching Steps
Data Splits
1
2
5
10
12
15
20
ADD AUC for 3D Keypoints Prediction
Panda DR
29.02
74.29
83.68
86.64
86.53
86.00
85.66
Panda Photo
29.12
74.47
84.08
87.09
86.99
86.43
86.08
Kuka DR
28.86
75.31
83.87
86.92
86.87
86.58
86.28
Kuka Photo
28.89
75.66
84.34
87.34
87.30
87.04
86.73
Appendix
Table 9: Results of 3D Keypoints Prediction and Joint Angle Prediction on different data splits from DREAM dataset.
Methods
Latency (ms)
FPS
Mem (MB)
Params (M)
GFLOPs
RoPE-Flow
36.5
27.4
4330
813.38
206.4
RoboPEPP
19.0
52.6
2446
334.90
132.0
HPE
35.7
28.0
379
79.62
35.0
Appendix
Table 10: Comparison of runtime and computational cost.