Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong · EACV Research Center, Shenzhen Loop Area Institute · Xi’an Jiaotong University
Camera-to-robot pose estimation from a monocular image is significant for vision-based robot navigation and control. However, recovering 3D structural features from 2D observations remains a challenging task. Existing camera-to-robot pose estimation methods either rely on time-consuming iterative rendering or involve sophisticated 2D-3D registration which suffers from projection ambiguity, accumulated errors for distal joints and imbalanced learning for different joints. In this work, we propose RoPE-Flow, an end-to-end flow matching-based framework for camera-to-robot pose estimation. From the predicted 2D heatmaps, we propose a Keypoint-centered Feature Extractor to obtain positional features along with contextual representations from keypoints-centered multi-scale patches. Moreover, we introduce Graph-guided Condition Generator, which embeds a learnable adjacency map to generate condition for flow matching based on correlation among keypoints. Besides, we introduce Bone Length Loss and Ray-Depth Loss for topological consistency and scale stability, with joint-wise weighting to mitigate varied prediction difficulty across joints. The proposed method outperforms state-of-the-art methods for pose estimation across different types of robots with mutiple cameras. Besides, we establish a novel cross-scene benchmark to demonstrate the outstanding performance of RoPE-Flow on zero-shot cross-scene robot pose estimation. Meanwhile, animated visualizations are shown on the website: ropeflow.netlify.app
Figures & tables
Figure 1: Overview of our work. RoPE-Flow can reconstruct varied robot pose in diverse scenes with camera-frame 3D keypoints and joint angles estimation from monocular observation of diverse cameras even with human-robot interaction , occlusion and out-of-vision body .
Figure 2: Comparison with previous methods for camera-to-robot estimation (Top Row). Evidence for Varied Joint Difficulty is shown based on distribution of prediction errors along the joints and keypoints. Problem for end-to-end 3D keypoints generation: Projection Ambiguity , as well as potential solutions in this work is also visualized.
Figure 3: Pipeline of the proposed method.
Panda
Kuka
Baxter
Methods
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
Avg
DREAM-F*
79.5
81.3
68.9
24.4
76.1
61.9
/
/
/
/
DREAM-Q*
74.3
77.8
52.4
37.5
78.0
57.0
/
/
75.5
/
DREAM-H*
81.1
82.9
60.5
64.0
78.8
69.1
72.1
73.3
/
/
RoboPose †
79.7
82.9
70.4
77.6
74.3
70.4
73.2
80.2
32.7
71.3
HPE ‡
82.0
82.7
82.2
76.0
75.2
75.2
73.9
75.1
58.8
75.7
Table 1: Comparison of robot pose estimation on DREAM dataset using AUC on the ADD metric. *DREAM and its variants need ground-truth joint state parameters. ‡ Results with ground-truth bounding box. † Results with predicted bounding box. The Best , Second Best and Third Best results are marked by corresponding colors. Curves of ADD can be found in Appendix B.1 .
Panda
Kuka
Baxter
Methods
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
DREAM-F
785.8
552.0
> 1k
> 1k
> 1k
> 1k
/
/
/
DREAM-Q
569.0
359.0
> 1k
> 1k
27.2
> 1k
/
/
59.0
DREAM-H
518.8
325.6
56.5
> 1k
23.6
> 1k
> 1k
> 1k
/
RoboPose
21.5
7.9
24.4
14.0
23.1
1 9.4
33.5
1 9.2
15.5
RoboPose
21.4
32.2
34.3
22.3
26.0
30.1
89.0
34.2
87.9
Table 2: Comparison of Mean Average Distance (mm). The best results among all methods (including methods with given joint states ) are in bold, while the second best one is underlined. The Best and Second Best results among methods with no need of joint state parameters are marked by corresponding colors.
Panda
Kuka
Panda
Methods
Photo
DR
Photo
DR
AK
XK
RS
ORB
RoboPose
5.7
4.9
7.2
6.2
9.3
13.7
12.2
7.6
HPE
4.5
4.5
4.5
4.5
6.9
6.1
10.4
7.6
RoboPEPP
3.2
3.9
3.9
4.4
8.5
8.0
15.7
15.6
Ours
3.2
3.4
2.9
3.1
6.2
4.8
9.4
6.3
Table 3: Comparison of joint angle estimation results using average MAE (°) of joint angles. Detailed MAE on each joint can be found in Appendix B.3 . The Best and Second Best results are marked by corresponding colors.
Figure 4: Qualitative comparison on URDF-based reconstructed models. More qualitative results can be found in Appendix B.4 . Results on videos can be found in Supplementary Material .
Methods
mIoU
Dice
HPE
0.1167 ± 0.049
0.2056 ± 0.077
RoboPEPP
0.4456 ± 0.211
0.5836 ± 0.229
Ours
0.7328 ± 0.036
0.8453 ± 0.024
Table 4: Quantitative comparison on zero-shot cross-scene dataset.
Figure 5: Qualitative comparison for zero-shot cross-scene samples. At last row, we use green mask to denote the ground truth, while yellow mask represents overlapping area and red mask is the mask of reconstructed robot. More examples can be found in Appendix B.5 .
Panda
Kuka
Baxter
Ablations
Photo
DR
AK
XK
RS
ORB
Photo
DR
DR
Avg
Full Model
87.1
86.6
90.9
81.6
85.1
82.1
87.3
86.9
74.1
84.6
Keypoints-centered Feature Extraction
3 × 3 Patch
86.0
85.8
89.3
81.4
84.6
76.4
86.8
86.2
70.8
83.0
7 × 7 Patch
86.6
86.3
90.2
79.4
84.6
79.4
86.9
86.5
69.9
83.3
11 × 11 Patch
87.0
86.7
89.0
75.5
77.2
79.7
87.6
87.2
74.2
82.7
Table 5: Results of ablations on feature extraction, condition generation, flow matching and loss functions. The Best , Second Best and Third Best results are marked by corresponding colors.
Ablations
AK
XK
RS
ORB
Avg
Full Model
90.9
81.6
85.1
82.1
84.9
w/o LboneS2R
90.9
76.7
84.5
81.5
83.4
w/o Ldepth
82.5
70.1
76.5
62.0
72.8
Table 6: Results of ablations on Sim-to-Real Tuning.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Comparison of LabRoPE and DREAM.
Figure 7: Curves of ADD.
Figure 8: Qualitative Comparison of 3D Keypoints Prediction.
Franka
Photo
DR
Comp
J1
J2
J3
J4
J5
J6
Avg
J1
J2
J3
J4
J5
J6
Avg
Pose
7.7
3.5
4.3
3.4
7.3
8.1
5.7
6.1
2.7
3 .6
2.5
6.3
8.1
4.9
HPE
6.1
2.2
3.6
2.0
6.2
6.6
4 .5
6.2
2 .2
3.9
1 .9
5.9
6.6
4.5
PEPP
4 .4
1 .8
2.2
1 .8
4.4
4.8
3.2
4 .9
2.4
2.9
2.4
5 .2
5 .6
3 .9
Ours
3.2
1.7
2 .5
1.6
4 .9
5 .2
3.2
3.6
1.7
2.9
1.7
4.9
5.5
3.4
Appendix
Table 7: Comparison of mean MAE (°) on each joint of robot.
Figure 9: Supplementary qualitative results from DREAM dataset.
Figure 10: Supplementary qualitative results from LabRoPE dataset.
Figure 11: Qualitative results of sim-to-real finetuning with or without Bone Length Loss or Ray-depth Loss
Figure 12: Results from multi-scale patch embedding with single patch embedding.
Methods
mIoU
Dice
Full Model
0.7328 ± 0.036
0.8453 ± 0.024
3 × 3 Patch
0.3986 ± 0.066
0.5668 ± 0.067
7 × 7 Patch
0.4236 ± 0.067
0.5920 ± 0.067
11 × 11 Patch
0.3727 ± 0.064
0.5399 ± 0.068
No Graph
0.4064 ± 0.077
0.5736 ± 0.080
w/o Lbone
0.3929 ± 0.087
0.5576 ± 0.105
Appendix
Table 8: Ablation results of zero-shot pose estimation on LabRoPE dataset.
Figure 13: Qualitative results of ablation studies on LabRoPE dataset.
Flow Matching Steps
Data Splits
1
2
5
10
12
15
20
ADD AUC for 3D Keypoints Prediction
Panda DR
29.02
74.29
83.68
86.64
86.53
86.00
85.66
Panda Photo
29.12
74.47
84.08
87.09
86.99
86.43
86.08
Kuka DR
28.86
75.31
83.87
86.92
86.87
86.58
86.28
Kuka Photo
28.89
75.66
84.34
87.34
87.30
87.04
86.73
Appendix
Table 9: Results of 3D Keypoints Prediction and Joint Angle Prediction on different data splits from DREAM dataset.
Methods
Latency (ms)
FPS
Mem (MB)
Params (M)
GFLOPs
RoPE-Flow
36.5
27.4
4330
813.38
206.4
RoboPEPP
19.0
52.6
2446
334.90
132.0
HPE
35.7
28.0
379
79.62
35.0
Appendix
Table 10: Comparison of runtime and computational cost.
Conditional flow matching has enabled a step forward in object 6D pose estimation, achieving state-of-the-art performance by progressively denoising and registering object representations to observed scenes. Existing methods require training task-specific encoders supervised on object-scene overlap and rely on trivial feature fusion strategies to resolve pose ambiguities. We present FunFlow6D, a novel flow matching-based formulation that leverages features from geometric and appearance foundation models for pose estimation, eliminating the need for task-specific encoder training. We also introduce a cross attention-based fusion mechanism that dynamically combines geometric and appearance features to provide richer conditioning for the flow matching module. Experiments on four datasets from the BOP benchmark show that FunFlow6D outperforms the previous state of the art while reducing supervision requirements and memory overhead. Extensive ablations validate the contribution of each proposed component. Project website: https://tev-fbk.github.io/FunFlow6D/.
Amir Hamza, Davide Boscaini, Fabio Poiesi
Fondazione Bruno Kessler Trento, Italy · University of Trento Trento, Italy
Monocular visual odometry (VO) is a fundamental computer vision problem with applications in autonomous navigation, augmented reality and more. While deep learning-based methods have recently shown superior accuracy compared to traditional geometric pipelines, particularly in environments where handcrafted features struggle due to poor structure or lighting conditions, most rely on deterministic regression, which lacks the uncertainty awareness required for robust applications. We propose PoseFM, the first framework to reformulate monocular frame-to-frame VO as a generative task using Flow Matching (FM). By leveraging FM, we model camera motion as a distribution rather than a point estimate, learning to transform noise into realistic pose predictions via continuous-time ODEs. This approach provides a principled mechanism for uncertainty estimation and enables robust motion inference under challenging visual conditions. In our evaluations, PoseFM achieves strong performance on TartanAir, KITTI and TUM-RGBD benchmarks, achieving the lowest absolute trajectory error (ATE) on some of the trajectories and overall being competitive with the best frame-to-frame monocular VO methods. Code and model checkpoints will be made available at https://github.com/helsinki-sda-group/posefm.
Dominik Kuczkowski, Laura Ruotsalainen
Department of Computer Science University of Helsinki Finland
Existing methods for instance-level 6D pose estimation typically rely on neural networks that either directly regress the pose in SE(3) or estimate it indirectly via local feature matching. The former struggle with object symmetries, while the latter fail in the absence of distinctive local features. To overcome these limitations, we propose a novel formulation of 6D pose estimation as a conditional flow matching problem in R3. We introduce Flose, a generative method that infers object poses via a denoising process conditioned on local features. While prior approaches based on conditional flow matching perform denoising solely based on geometric guidance, Flose integrates appearance-based semantic features to mitigate ambiguities caused by object symmetries. We further incorporate RANSAC-based registration to handle outliers. We validate Flose on five datasets from the established BOP benchmark. Flose outperforms prior methods with an average improvement of +4.5 Average Recall. Project Website : https://tev-fbk.github.io/Flose/
Amir Hamza, Davide Boscaini, Weihang Li +2
Fondazione Bruno Kessler · University of Trento · Technical University of Munich +1