This paper studies category-level RGB-D object pose estimation, which recovers an object's 3D rotation and translation without instance-specific CAD models or reference views. This task is highly ambiguous due to intra-class variations, symmetries, and partial observations. Current solutions struggle to balance reliability and efficiency: methods based on intermediate structures are prone to error propagation, while diffusion models suffer from high inference latency and complex post-hoc candidate extraction. To address this, we present F2G-Pose, a real-time direct, single-pass framework that predicts category-level pose, metric size, and camera-frame completed shape from a single RGB-D observation in one feed-forward pass. F2G-Pose lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and integrates object-level structure with a Mixture-of-Experts Transformer. Joint size prediction and camera-frame shape completion encourage metric consistency through dense geometric supervision. With its task-specific network trained only on pose-annotated synthetic SOPE data and its frozen RADIO backbone externally pretrained, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE while running at 28 FPS for pose-only inference, and further demonstrates strong camera-frame geometry on HouseCat6D and real-object transfer on HANDAL. Analyses of latency, robustness, shape completion, and GenPose++ candidate behavior further demonstrate the efficiency and reliability of F2G-Pose, with real-world manipulation experiments demonstrating deployment potential under segmented RGB-D inputs. The project page, code, and pretrained models are made available at https://jrryzh.github.io/F2G-Pose/.
Figures & tables
Figure 1: F2G-Pose Overview. Given a cropped RGB image and segmented partial point cloud, F2G-Pose extracts dense RADIOv2.5 features [ 13 ] and lifts them onto 3D points by projection. DGCNN tokenization produces fused RGB-D object tokens, and a MoE Transformer encodes a global object representation. Two parallel decoder branches directly predict (i) 6D pose and metric size and (ii) camera-frame completed shape through coarse-to-fine refinement. The pipeline produces a pose-size-shape estimate in one feed-forward pass.
Dataset
Method
AUC ↑
VUS ↑
Mean error ↓
FPS ↑
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
rot( ∘ )
trans(cm)
ROPE
NOCS [ 42 ]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
–
–
5
SGPA [ 1 ]
10.5
2.0
0.0
4.3
6.7
9.3
15.0
60.2
2.81
6
IST-Net [ 26 ]
28.7
10.6
0.5
2.0
3.4
5.3
8.8
78.4
3.40
35
HS-Pose [ 55 ]
31.6
13.6
1.1
3.5
5.3
8.4
12.7
63.3
3.02
50
GenPose [ 51 ]
–
–
–
6.6
9.6
13.1
19.3
42.2
2.05
6
Table 1: Category-level pose-size estimation on ROPE and SOPE under the shared SOPE/ROPE category taxonomy. ‘–’ indicates unavailable or non-hardware-normalized metrics. Joint-RegDiff and RFMPose are report-only comparisons from their papers. * indicates our pose-only inference mode (shape head disabled); the full model runs at 23.7 FPS including shape completion.
Figure 2: Qualitative results on ROPE. We show the input RGB image, ground-truth pose, poses from GenPose++ and F2G-Pose, and the predicted camera-frame shape.
Analysis
Rotation Error (degrees)
Mean ↓
Median ↓
>45∘↓
≤5∘↑
All candidates
39.3
8.2
26.9%
34.0%
Best-of-50 oracle
9.0
1.78
6.1%
79.3%
Final pipeline
30.7
5.2
19.1%
48.3%
Table 2: GenPose++ candidate diagnostics on ROPE. The sampled set can contain accurate hypotheses, but deployed accuracy depends on candidate extraction.
Figure 3: Pareto analysis of GenPose++ candidate budget on ROPE.
Table 6Table 7
Category
ROPE
HouseCat6D
AUC IoU25 ↑
AUC IoU75 ↑
VUS 5∘2cm↑
Rot. Err. ↓
Trans. Err. ↓
CD ↓
Transparent
43.5
4.7
10.4
25.3
1.19
6.2
Reflective
42.6
4.5
9.8
25.2
1.25
10.1
Average
46.5
5.8
11.3
27.5
1.19
7.1
Table 7: Performance on reflective and transparent materials across ROPE and HouseCat6D.
Data
Method
AUC ↑
IoU25
IoU50
IoU75
ROPE
FoundationPose (1 ref)
35.0
18.0
3.1
Any6D (1 ref)
37.2
19.4
3.5
Any6D (0 ref)
22.5
8.1
0.3
Ours
46.5
27.8
5.8
HANDAL
Any6D (0 ref)
14.5
3.8
0.0
Table 8: Cross-dataset ROPE/HANDAL evaluation.
Figure 10
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Examples of controlled RGB and depth perturbations.
Perturbation
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
Δ AUC / Δ VUS (%)
Clean
46.5
27.8
5.8
11.3
15.9
21.8
31.1
+0.0 / +0.0
Light
46.3
27.7
5.7
11.3
15.9
21.7
30.8
-0.5 / -0.6
Moderate
46.1
27.4
5.6
11.2
15.7
21.5
30.5
-1.3 / -1.6
Heavy
44.7
26.4
5.4
10.6
15.2
20.5
29.1
-4.7 / -6.4
Appendix
Table 10: Controlled robustness under photometric perturbations on ROPE.
Perturbation
α
β
IoU25
IoU50
IoU75
5 ∘ 2cm
5 ∘ 5cm
10 ∘ 2cm
10 ∘ 5cm
Δ AUC / Δ VUS (%)
Clean
0.000
0.000
46.5
27.8
5.8
11.3
15.9
21.8
31.1
+0.0 / +0.0
Low
0.001
0.000
46.3
27.7
5.7
11.2
15.9
21.9
31.2
-0.5 / +0.1
Moderate
0.002
0.003
43.3
24.4
4.5
10.1
14.9
20.3
29.5
-10.4 / -7.2
Heavy
0.003
0.006
39.0
20.0
3.3
8.5
13.2
17.6
26.9
-23.4 / -20.0
Appendix
Table 11: Controlled robustness under heteroscedastic depth noise on ROPE.
Symmetry type
Mean rotation error ↓
Candidates ≤5∘↑
Continuous
24.0
49%
Quarter-turn
19.0
33%
Half-turn
45.0
25%
None
57.0
21%
Appendix
Table 12: GenPose++ candidate quality by symmetry on ROPE. Values are computed over sampled candidates before the final candidate extraction step.
Figure 6: Failure cases caused by corrupted or incomplete depth on reflective object regions.
Figure 7: Failure cases caused by symmetric, geometrically regular, or small objects with ambiguous orientation cues.
Setting
AUC IoU25
AUC IoU50
AUC IoU75
VUS 5 ∘ 5cm
VUS 10 ∘ 2cm
Main components
Full
46.5
27.8
5.8
15.9
21.8
w/o RGB
34.5
16.2
2.2
6.5
13.2
w/o MoE
41.0
24.0
3.9
8.7
17.8
w/o Shape Sup.
39.5
23.2
3.5
8.1
16.5
MoE routing and capacity
Appendix
Table 13: Supplementary pose-size ablations on ROPE. We keep the metric definitions consistent with the main tables; “–” indicates a metric that is not applicable or was not measured for that variant.
Figure 8: Additional qualitative examples from ROPE.
Figure 9: Reflective and transparent object examples from HouseCat6D. These cases illustrate material-induced depth corruption and the resulting camera-frame shape predictions.
Figure 10: Qualitative results on HANDAL. We compare ground-truth 3D boxes with predictions from Any6D and F2G-Pose.
Pose/
VUS ↑
Mean error ↓
Canon.
Group
Model
size loss
IoU50
5 ∘ 2cm
10 ∘ 5cm
Rot.
Trans.
CD ↓
A
F2G-Pose
Yes
41.5
18.5
38.5
17.8
1.10
34.1
B
F2G-Pose
No
–
–
–
–
–
36.7
C
AdaPoinTr
No
–
–
–
–
–
37.2
D
AdaPoinTr
Yes
38.5
16.5
35.0
19.5
1.25
35.5
Appendix
Table 14: Matched RGB-D completion on SOPE. All variants predict normalized canonical shapes. Canonical CD-L1 is shown ×103 ; rotation and translation errors are in degrees and cm. “–” denotes no evaluated pose output.
Mask
IoU50 AUC
IoU75 AUC
VUS 5 ∘ 5cm
Rot. ( ∘ )
Trans. (cm)
GT
27.8
5.8
15.9
27.5
1.19
SAM
27.2
5.6
15.2
28.1
1.21
Erosion, IoU 90%
26.6
5.3
14.8
28.3
1.24
Erosion, IoU 80%
23.8
4.1
13.7
28.8
1.32
Dilation, IoU 90%
27.5
5.3
15.8
27.6
1.20
Dilation, IoU 80%
25.8
4.0
15.5
27.8
1.21
Appendix
Table 15: Sensitivity of pose estimation to segmentation masks on ROPE. Rotation and translation columns report mean errors.
Mask
Mask IoU
Shape CD ↓
GT
100%
5.42
Dilation
90%
5.45
Dilation
80%
5.48
Erosion
90%
5.57
Erosion
80%
5.80
Appendix
Table 16: Sensitivity of SOPE camera-frame shape completion to segmentation masks. Symmetric CD-L1 is shown ×103 .
HANDAL category
SOPE category
Instances
IoU25
IoU50
IoU75
Claw Hammer
hammer
19
35.8
11.5
0.1
Adjustable Wrench
spanner
13
12.1
1.0
0.0
Combination Wrench
spanner
12
11.8
0.9
0.0
Ladle
spoon
12
29.4
3.9
0.0
Measuring Cup
cup
12
45.2
17.0
0.3
Mug
mug
12
68.3
35.4
1.6
Appendix
Table 17: HANDAL per-category IoU AUC for the seven directly matched categories. Counts refer to instances. The last row is the unweighted macro-average over seven HANDAL categories.
Category-level object pose estimation predicts the rotation, translation, and metric size of unseen instances within known categories. Many accurate RGB-D methods rely on external instance segmentation and crop-based pose estimation, introducing separate stages and object-dependent processing costs that hinder real-time inference. To bring accurate pose estimation into real time, we present \ours{}, an end-to-end trainable query-based RGB-D set predictor. It jointly detects and segments objects and estimates their \mbox{9-DoF} poses without explicit CAD-derived shape priors or a separately trained instance segmentor. Shared image and scene encoding avoids repeated per-object crop encoding. A query-conditioned geometry pathway associates observed 3D points and RGB features with object queries and incorporates shared scene context. Object-centric refinement uses the resulting point descriptors to update an explicit pose state through pose-conditioned cross-attention and recurrent residual corrections. On NOCS, \ours{} substantially improves on published RGB-D joint detection and pose estimation results. It achieves competitive performance compared with two-stage methods under all-object evaluation on REAL275 and HouseCat6D, while enabling real-time full-frame pose estimation at 31.8 FPS on an RTX~A6000. Project page: https://yopo-series.github.io/RYOPO-project-page/.
Hakjin Lee, Junghoon Seo, Jaehoon Sim
PIT IN Co., Anyang-si, Gyeonggi-do, Republic of Korea.
Object pose estimation is a fundamental problem in 3D vision. Although recent state-of-the-art approaches achieve strong performance, generalization to novel categories and unseen scenes remains challenging. We propose UniPose9D, a unified model for category-agnostic 9D object pose estimation: given an instance mask/ROI and either an RGB-D observation or an RGB image with predicted depth, the model estimates rotation, translation, and metric size without category labels, CAD models, mean-shape priors, or reference views. Specifically, UniPose9D samples point pairs from the observed object geometry and uses DINOv2 and PointNet features to predict NOCS coordinates for each pair. To improve accuracy, we introduce a point-pair-based RANSAC N-hop Kabsch-Umeyama algorithm with an adaptive threshold. We further employ flow matching to address symmetric ambiguities and construct a large-scale training set by curating and aligning pose annotations from existing public datasets. Experiments across eight datasets show that a single unified model achieves competitive performance on standard benchmarks while generalizing to unseen objects, unseen categories, and in-the-wild scenarios. Our code and model are available at https://github.com/qq456cvb/UniPose9D.
Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object pose and size, and a canonical Semantic Gaussian Field. Rather than treating reconstruction as a detached auxiliary task, LEGAU uses the Gaussian field as a category-conditioned structural prior that participates in multimodal feature fusion and provides global guidance for local pose reasoning. Conditioned on a categorical text embedding, LEGAU processes RGB-D observations through a transformer-based fusion module that integrates visual, geometric, and category-level cues, decoding the NOCS map, pose and size information and the Gaussian-based object representation. Extensive experiments on synthetic and real-world benchmarks show that this coupled pose-shape formulation achieves strong performance in a single-model multi-category setting, with up to 22% on SOPE and competitive transfer to real-world data. These results highlight the benefit of jointly learning canonical correspondence, object shape, and pose alignment within a unified representation.
Hongli Xu, Zhaowei Lu, Junwen Huang +5
Technical University of Munich Munich, Germany · Munich Center for Machine Learning Munich, Germany · Siemens AG Munich, Germany +1