RGB-based category-level 9D pose estimation remains highly challenging because a monocular system must generalize across unseen object instances while jointly inferring 3D translation, 3D rotation, and 3D size. Unlike instance-level pose estimation, the category-level setting requires the network to learn both what is shared across objects within a category and how individual instances vary in structure. This challenge is amplified in the RGB-only setting, where pose cues, category-level commonality, and instance-specific variation are entangled in appearance without direct geometric constraints, making it difficult for existing frameworks to learn a representation that is truly aligned with the category-level estimation objective. To address this problem, we propose SchemaPose, a schema-guided framework for RGB-based category-level pose estimation. SchemaPose equips the estimation pipeline with an explicit parametric schema for modeling category-level regularities together with structured instance variation, and uses the inferred schema variables to guide pose-related prediction in an end-to-end query-based framework. The same formulation also supports template instantiation, synthetic annotation, and pose learning within a unified framework. Experiments on synthetic and real data show that SchemaPose achieves state-of-the-art performance among RGB-based methods for category-level 9D pose estimation.
Figures & tables
Figure 1: Overview of the parametric category schema, compact representation, and data generation pipeline. (a) Parametric Category Schema. For category c , the schema comprises discrete part choices dc and continuous geometric parameters xc , which define a valid object structure via semantic part templates. (b) Compact Representation for Learning. A category-specific compression function Ψc maps the full schema variable zc=(dc,xc) to a compact vector gc , preserving the dominant pose-relevant structural factors for network training. (c) Schema-Driven Data Generation. The schema variable zc is analytically instantiated into an object model Mc via Φc . These models are diversified through parameter perturbation and rendered within synthetic scenes to generate training images with aligned ground-truth pose labels.
Figure 2: Relationship between object schema and pose. Objects are parameterized through schema variables, which provide the structural basis for expressing and interpreting 9D pose.
Figure 3: Overview of SchemaPose and Dual-Stage Schema Guided Mechanism. (a) SchemaPose follows an end-to-end query-based pipeline. Given an RGB image, the backbone and Transformer encoder produce encoder memory Qenc , from which top-K proposals are selected to initialize decoder queries. The Transformer decoder refines these queries and produces decoder features Qdec . (b) The Schema-Guided Pose Head (SGPH) follows the same design at the proposal and decoder stages. Given an object feature q , SGPH first predicts the object category c and bounding box b , then infers a unified schema vector g . A category-specific mask mc selects the effective schema representation g~ , which is fed back as structural guidance for pose-related prediction. During training, detection, schema, and pose predictions are supervised at both stages, while inference uses the final pose output (ox,oy),z,R,s from the last decoder layer.
Method
CAMERA25
REAL275
IoU 50
IoU 75
10cm
10∘
10∘ 10cm
IoU 50
IoU 75
10cm
10∘
10∘ 10cm
MSOS [ 18 ]
32.4
5.1
29.7
60.8
19.2
23.4
3.0
39.5
29.2
9.6
OLD-Net [ 17 ]
32.1
5.4
30.1
74.0
23.4
25.4
1.9
38.9
37.0
9.8
DMSR [ 16 ]
34.6
6.5
32.3
81.4
27.4
28.3
6.1
37.3
59.5
23.6
LaPose [ 13 ]
–
–
–
–
–
17.5
2.6
44.4
–
30.5
GIVEPose [ 12 ]
–
–
–
–
–
20.1
–
45.9
–
34.2
Table 1: Comparison with state-of-the-art RGB-based methods on CAMERA25 and REAL275. The best result in each column is shown in bold .
Figure 4: Qualitative comparison with YOPO on REAL275. Green boxes indicate ground truth and red boxes indicate predictions. Light-blue arrows mark instances where YOPO fails while SchemaPose produces more accurate pose estimates.
Table 6Table 7
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Additional examples of SchemaData . The top rows show rendered RGB training images generated by compositional scene rendering with diverse object layouts, viewpoints, lighting conditions, materials, textures, and backgrounds. The bottom row shows schema-generated object instances from different categories. These objects are obtained by varying discrete template choices and continuous geometric parameters within category-consistent ranges. Each rendered instance is associated with aligned pose labels and a compact schema vector inherited from its generation parameters.
Branch
Input
Output dim.
Classification
object feature
6
2D box
object feature
4
Schema
feature + box
60
Projected center
feature + box + schema
2
Depth
feature + box + schema
1
Rotation
feature + box + schema
36
Appendix
Table 6: Branch configuration of the Schema-Guided Pose Head.
Category
Continuous dim.
Discrete dim.
Total dim.
Bottle
11
3
14
Bowl
5
4
9
Camera
6
2
8
Can
7
2
9
Laptop
7
1
8
Mug
8
4
12
Appendix
Table 7: Compact schema dimensions for each category.
Loss term
Weight
Classification focal loss
1.0
BBox ℓ1 loss
5.0
BBox GIoU loss
2.0
Projected center ℓ1 loss
5.0
Depth ℓ2 loss
50.0
Rotation geodesic loss
5.0
Appendix
Table 8: Loss weights used for training SchemaPose.
Matching cost term
Weight
Focal classification cost
2.0
BBox ℓ1 cost
5.0
GIoU cost
2.0
Translation cost
5.0
Rotation cost
2.0
Appendix
Table 9: Hungarian matching cost weights.
Method
IoU 50
IoU 75
10cm
10∘
10∘ 10cm
SchemaPose
86.5 ± 1.0
25.8 ± 1.5
90.8 ± 0.3
74.9 ± 1.0
67.5 ± 1.3
Appendix
Table 10: Statistical variation of SchemaPose on REAL275 across five random seeds. We report mean ± standard deviation.
Object pose estimation is a fundamental problem in 3D vision. Although recent state-of-the-art approaches achieve strong performance, generalization to novel categories and unseen scenes remains challenging. We propose UniPose9D, a unified model for category-agnostic 9D object pose estimation: given an instance mask/ROI and either an RGB-D observation or an RGB image with predicted depth, the model estimates rotation, translation, and metric size without category labels, CAD models, mean-shape priors, or reference views. Specifically, UniPose9D samples point pairs from the observed object geometry and uses DINOv2 and PointNet features to predict NOCS coordinates for each pair. To improve accuracy, we introduce a point-pair-based RANSAC N-hop Kabsch-Umeyama algorithm with an adaptive threshold. We further employ flow matching to address symmetric ambiguities and construct a large-scale training set by curating and aligning pose annotations from existing public datasets. Experiments across eight datasets show that a single unified model achieves competitive performance on standard benchmarks while generalizing to unseen objects, unseen categories, and in-the-wild scenarios. Our code and model are available at https://github.com/qq456cvb/UniPose9D.
In the pursuit of robust and generalizable category-level object pose estimation, most existing methods adopt parametric formulations that learn effective representations from data, yet they primarily encode category-level patterns into fixed shape priors or static parameter weights, which limits their scalability to highly diverse instances. In this paper, we rethink category-level pose estimation from a memory-centric perspective and present MemPose, a memory-augmented framework that explicitly incorporates category-level geometric memory into the pose estimation pipeline. We introduce an external memory buffer that stores and dynamically updates structural representations from previously observed instances, enabling the model to leverage accumulated experience to support current perception. Extensive experiments on four challenging benchmarks (REAL275, CAMERA25, Housecat6D and Wild6D) demonstrate the superiority of our proposed method over previous state-of-the-art approaches.
Xiao Lin, Minghao Zhu, Yun Peng +4
Tongji University, Shanghai, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems
Category-level object pose estimation predicts the rotation, translation, and metric size of unseen instances within known categories. Many accurate RGB-D methods rely on external instance segmentation and crop-based pose estimation, introducing separate stages and object-dependent processing costs that hinder real-time inference. To bring accurate pose estimation into real time, we present \ours{}, an end-to-end trainable query-based RGB-D set predictor. It jointly detects and segments objects and estimates their \mbox{9-DoF} poses without explicit CAD-derived shape priors or a separately trained instance segmentor. Shared image and scene encoding avoids repeated per-object crop encoding. A query-conditioned geometry pathway associates observed 3D points and RGB features with object queries and incorporates shared scene context. Object-centric refinement uses the resulting point descriptors to update an explicit pose state through pose-conditioned cross-attention and recurrent residual corrections. On NOCS, \ours{} substantially improves on published RGB-D joint detection and pose estimation results. It achieves competitive performance compared with two-stage methods under all-object evaluation on REAL275 and HouseCat6D, while enabling real-time full-frame pose estimation at 31.8 FPS on an RTX~A6000. Project page: https://yopo-series.github.io/RYOPO-project-page/.
Hakjin Lee, Junghoon Seo, Jaehoon Sim
PIT IN Co., Anyang-si, Gyeonggi-do, Republic of Korea.