This paper studies category-level RGB-D object pose estimation, which recovers an object's 3D rotation and translation without instance-specific CAD models or reference views. This task is highly ambiguous due to intra-class variations, symmetries, and partial observations. Current solutions struggle to balance reliability and efficiency: methods based on intermediate structures are prone to error propagation, while diffusion models suffer from high inference latency and complex post-hoc candidate extraction. To address this, we present F2G-Pose, a real-time direct, single-pass framework that predicts category-level pose, metric size, and camera-frame completed shape from a single RGB-D observation in one feed-forward pass. F2G-Pose lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and integrates object-level structure with a Mixture-of-Experts Transformer. Joint size prediction and camera-frame shape completion encourage metric consistency through dense geometric supervision. With its task-specific network trained only on pose-annotated synthetic SOPE data and its frozen RADIO backbone externally pretrained, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE while running at 28 FPS for pose-only inference, and further demonstrates strong camera-frame geometry on HouseCat6D and real-object transfer on HANDAL. Analyses of latency, robustness, shape completion, and GenPose++ candidate behavior further demonstrate the efficiency and reliability of F2G-Pose, with real-world manipulation experiments demonstrating deployment potential under segmented RGB-D inputs. The project page, code, and pretrained models are made available at https://jrryzh.github.io/F2G-Pose/.
Figures & tables
Figure 1: F2G-Pose Overview. Given a cropped RGB image and segmented partial point cloud, F2G-Pose extracts dense RADIOv2.5 features [ 13 ] and lifts them onto 3D points by projection. DGCNN tokenization produces fused RGB-D object tokens, and a MoE Transformer encodes a global object representation. Two parallel decoder branches directly predict (i) 6D pose and metric size and (ii) camera-frame completed shape through coarse-to-fine refinement. The pipeline produces a pose-size-shape estimate in one feed-forward pass.
Dataset
Method
AUC ↑
VUS ↑
Mean error ↓
FPS ↑
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
rot( ∘ )
trans(cm)
ROPE
NOCS [ 42 ]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
–
–
5
SGPA [ 1 ]
10.5
2.0
0.0
4.3
6.7
9.3
15.0
60.2
2.81
6
IST-Net [ 26 ]
28.7
10.6
0.5
2.0
3.4
5.3
8.8
78.4
3.40
35
HS-Pose [ 55 ]
31.6
13.6
1.1
3.5
5.3
8.4
12.7
63.3
3.02
50
GenPose [ 51 ]
–
–
–
6.6
9.6
13.1
19.3
42.2
2.05
6
Table 1: Category-level pose-size estimation on ROPE and SOPE under the shared SOPE/ROPE category taxonomy. ‘–’ indicates unavailable or non-hardware-normalized metrics. Joint-RegDiff and RFMPose are report-only comparisons from their papers. * indicates our pose-only inference mode (shape head disabled); the full model runs at 23.7 FPS including shape completion.
Figure 2: Qualitative results on ROPE. We show the input RGB image, ground-truth pose, poses from GenPose++ and F2G-Pose, and the predicted camera-frame shape.
Analysis
Rotation Error (degrees)
Mean ↓
Median ↓
>45∘↓
≤5∘↑
All candidates
39.3
8.2
26.9%
34.0%
Best-of-50 oracle
9.0
1.78
6.1%
79.3%
Final pipeline
30.7
5.2
19.1%
48.3%
Table 2: GenPose++ candidate diagnostics on ROPE. The sampled set can contain accurate hypotheses, but deployed accuracy depends on candidate extraction.
Figure 3: Pareto analysis of GenPose++ candidate budget on ROPE.
Table 6Table 7
Category
ROPE
HouseCat6D
AUC IoU25 ↑
AUC IoU75 ↑
VUS 5∘2cm↑
Rot. Err. ↓
Trans. Err. ↓
CD ↓
Transparent
43.5
4.7
10.4
25.3
1.19
6.2
Reflective
42.6
4.5
9.8
25.2
1.25
10.1
Average
46.5
5.8
11.3
27.5
1.19
7.1
Table 7: Performance on reflective and transparent materials across ROPE and HouseCat6D.
Data
Method
AUC ↑
IoU25
IoU50
IoU75
ROPE
FoundationPose (1 ref)
35.0
18.0
3.1
Any6D (1 ref)
37.2
19.4
3.5
Any6D (0 ref)
22.5
8.1
0.3
Ours
46.5
27.8
5.8
HANDAL
Any6D (0 ref)
14.5
3.8
0.0
Table 8: Cross-dataset ROPE/HANDAL evaluation.
Figure 10
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Examples of controlled RGB and depth perturbations.
Perturbation
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
Δ AUC / Δ VUS (%)
Clean
46.5
27.8
5.8
11.3
15.9
21.8
31.1
+0.0 / +0.0
Light
46.3
27.7
5.7
11.3
15.9
21.7
30.8
-0.5 / -0.6
Moderate
46.1
27.4
5.6
11.2
15.7
21.5
30.5
-1.3 / -1.6
Heavy
44.7
26.4
5.4
10.6
15.2
20.5
29.1
-4.7 / -6.4
Appendix
Table 10: Controlled robustness under photometric perturbations on ROPE.
Perturbation
α
β
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
Δ AUC / Δ VUS (%)
Clean
0.000
0.000
46.5
27.8
5.8
11.3
15.9
21.8
31.1
+0.0 / +0.0
Low
0.001
0.000
46.3
27.7
5.7
11.2
15.9
21.9
31.2
-0.5 / +0.1
Moderate
0.002
0.003
43.3
24.4
4.5
10.1
14.9
20.3
29.5
-10.4 / -7.2
Heavy
0.003
0.006
39.0
20.0
3.3
8.5
13.2
17.6
26.9
-23.4 / -20.0
Appendix
Table 11: Controlled robustness under heteroscedastic depth noise on ROPE.
Symmetry type
Mean rotation error ↓
Candidates ≤5∘↑
Continuous
24.0
49%
Quarter-turn
19.0
33%
Half-turn
45.0
25%
None
57.0
21%
Appendix
Table 12: GenPose++ candidate quality by symmetry on ROPE. Values are computed over sampled candidates before the final candidate extraction step.
Figure 6: Failure cases caused by corrupted or incomplete depth on reflective object regions.
Figure 7: Failure cases caused by symmetric, geometrically regular, or small objects with ambiguous orientation cues.
Setting
AUC IoU25
AUC IoU50
AUC IoU75
VUS 5 ∘ 5cm
VUS 10 ∘ 2cm
Main components
Full
46.5
27.8
5.8
15.9
21.8
w/o RGB
34.5
16.2
2.2
6.5
13.2
w/o MoE
41.0
24.0
3.9
8.7
17.8
w/o Shape Sup.
39.5
23.2
3.5
8.1
16.5
MoE routing and capacity
Appendix
Table 13: Supplementary pose-size ablations on ROPE. We keep the metric definitions consistent with the main tables; “–” indicates a metric that is not applicable or was not measured for that variant.
Figure 8: Additional qualitative examples from ROPE.
Figure 9: Reflective and transparent object examples from HouseCat6D. These cases illustrate material-induced depth corruption and the resulting camera-frame shape predictions.
Figure 10: Qualitative results on HANDAL. We compare ground-truth 3D boxes with predictions from Any6D and F2G-Pose.
Pose/
VUS ↑
Mean error ↓
Canon.
Group
Model
size loss
IoU50
5 ∘ 2cm
10 ∘ 5cm
Rot.
Trans.
CD ↓
A
F2G-Pose
Yes
41.5
18.5
38.5
17.8
1.10
34.1
B
F2G-Pose
No
–
–
–
–
–
36.7
C
AdaPoinTr
No
–
–
–
–
–
37.2
D
AdaPoinTr
Yes
38.5
16.5
35.0
19.5
1.25
35.5
Appendix
Table 14: Matched RGB-D completion on SOPE. All variants predict normalized canonical shapes. Canonical CD-L1 is shown ×103 ; rotation and translation errors are in degrees and cm. “–” denotes no evaluated pose output.
Mask
IoU50 AUC
IoU75 AUC
VUS 5 ∘ 5cm
Rot. ( ∘ )
Trans. (cm)
GT
27.8
5.8
15.9
27.5
1.19
SAM
27.2
5.6
15.2
28.1
1.21
Erosion, IoU 90%
26.6
5.3
14.8
28.3
1.24
Erosion, IoU 80%
23.8
4.1
13.7
28.8
1.32
Dilation, IoU 90%
27.5
5.3
15.8
27.6
1.20
Dilation, IoU 80%
25.8
4.0
15.5
27.8
1.21
Appendix
Table 15: Sensitivity of pose estimation to segmentation masks on ROPE. Rotation and translation columns report mean errors.
Mask
Mask IoU
Shape CD ↓
GT
100%
5.42
Dilation
90%
5.45
Dilation
80%
5.48
Erosion
90%
5.57
Erosion
80%
5.80
Appendix
Table 16: Sensitivity of SOPE camera-frame shape completion to segmentation masks. Symmetric CD-L1 is shown ×103 .
HANDAL category
SOPE category
Instances
IoU25
IoU50
IoU75
Claw Hammer
hammer
19
35.8
11.5
0.1
Adjustable Wrench
spanner
13
12.1
1.0
0.0
Combination Wrench
spanner
12
11.8
0.9
0.0
Ladle
spoon
12
29.4
3.9
0.0
Measuring Cup
cup
12
45.2
17.0
0.3
Mug
mug
12
68.3
35.4
1.6
Appendix
Table 17: HANDAL per-category IoU AUC for the seven directly matched categories. Counts refer to instances. The last row is the unweighted macro-average over seven HANDAL categories.