RYOPO: Bringing End-to-End Category-Level Object Pose Estimation into Real Time

Hakjin Lee; Junghoon Seo; Jaehoon Sim

cs.CVOct 2, 2026

RYOPO: Bringing End-to-End Category-Level Object Pose Estimation into Real Time

Authors: Hakjin Lee, Junghoon Seo, Jaehoon Sim

Organizations: PIT IN Co., Anyang-si, Gyeonggi-do, Republic of Korea.

Abstract

Category-level object pose estimation predicts the rotation, translation, and metric size of unseen instances within known categories. Many accurate RGB-D methods rely on external instance segmentation and crop-based pose estimation, introducing separate stages and object-dependent processing costs that hinder real-time inference. To bring accurate pose estimation into real time, we present \ours{}, an end-to-end trainable query-based RGB-D set predictor. It jointly detects and segments objects and estimates their \mbox{9-DoF} poses without explicit CAD-derived shape priors or a separately trained instance segmentor. Shared image and scene encoding avoids repeated per-object crop encoding. A query-conditioned geometry pathway associates observed 3D points and RGB features with object queries and incorporates shared scene context. Object-centric refinement uses the resulting point descriptors to update an explicit pose state through pose-conditioned cross-attention and recurrent residual corrections. On NOCS, \ours{} substantially improves on published RGB-D joint detection and pose estimation results. It achieves competitive performance compared with two-stage methods under all-object evaluation on REAL275 and HouseCat6D, while enabling real-time full-frame pose estimation at $31.8$ FPS on an RTX~A6000. Project page: https://yopo-series.github.io/RYOPO-project-page/.

Figures & tables

Fig. 1: REAL275 all-object

10^{\circ}5

cm AP versus mean pose-output latency from Table VI on RTX A6000.

Fig. 2: End-to-end RGB-D set prediction in RYOPO. (A) query-conditioned geometry and (B) geometry feedback support (C) object-centric pose refinement. Unfilled blocks form the DETR-style scaffold. Point colors come from RGB; marker size and opacity encode support, and hollow markers indicate low support. Output visualization: predicted cuboids (green), annotations (white).

Method	CAD/prior	TTO	IoU 50	$5^{\circ}$ 5 cm	$10^{\circ}$ 5 cm	IoU 50	$5^{\circ}$ 5 cm	$10^{\circ}$ 5 cm	FPS
			CAMERA25			REAL275
CenterSnap [ 6 ]	✓	✗	92.3	63.0	79.5	80.2	27.2	58.8	28.9
CenterSnap-R [ 6 ]	✓	✓	92.5	66.2	81.3	80.2	29.1	64.3	–
ShAPO [ 7 ]	✓	✓	93.5	66.6	81.9	79.0	48.8	66.8	0.03 ‡
FSD [ 16 ]	✓	✗	–	–	–	77.4	28.1	61.5	–
SceneGrasp [ 37 ]	✓	✗	–	–	–	76.2	18.4	37.5	–

TABLE I: Joint RGB-D detection and pose estimation on NOCS. CAD/prior: instance 3D models or explicit shape priors used during training or inference; TTO: test-time optimization. ‡ short-run TTO FPS estimate.

Method	IoU 50	IoU ${}_{75}^{*}$	$5^{\circ}$ 2 cm	$5^{\circ}$ 5 cm	$10^{\circ}$ 2 cm	$10^{\circ}$ 5 cm	IoU 50	IoU ${}_{75}^{*}$	$5^{\circ}$ 2 cm	$5^{\circ}$ 5 cm	$10^{\circ}$ 2 cm	$10^{\circ}$ 5 cm	FPS
	CAMERA25						REAL275
DPDN [ 13 ]	93.9	70.5	64.1	70.7	75.3	84.8	83.4	54.4	38.1	42.2	58.7	65.3	8.8 †
HS-Pose [ 15 ]	82.1	51.3	62.2	68.9	67.4	75.9	82.8	41.3	38.5	46.6	57.3	70.1	10.4 †
VI-Net [ 14 ]	91.4	77.2	70.2	77.0	74.5	82.0	83.1	48.4	40.7	47.1	58.7	68.3	8.7 †
AG-Pose (DINO) [ 2 ]	94.1	80.1	75.6	79.5	82.7	87.9	84.1	61.2	46.4	52.7	63.2	71.0	7.5 †
CleanPose [ 3 ]	94.1	79.6	76.5	80.0	83.4	88.1	84.1	62.6	50.4	55.1	65.7	72.3	7.5 †

TABLE II: NOCS all-object comparison with per-object pose pipelines. Pose AP includes all annotated instances. ∗ CATRE IoU; † detector-plus-pose FPS estimate at

K=8

Pose AP: IoU-matched instances
Method	masks	IoU 25	IoU 50	$5^{\circ}$ 2 cm	$5^{\circ}$ 5 cm	$10^{\circ}$ 2 cm	$10^{\circ}$ 5 cm
VI-Net [ 14 ]	GT	80.7	56.4	8.4	10.3	20.5	29.1
AG-Pose (DINO) [ 2 ]	GT	88.1	76.9	21.3	22.1	51.3	54.3
CleanPose [ 3 ]	GT	89.2	79.8	22.4	24.1	51.6	56.5
Pose AP: all annotated instances
AG-Pose (DINO) [ 2 ]	GT	87.9	76.5	21.5	22.9	50.6	54.3

TABLE III: HouseCat6D. Shaded rows and GT denote ground-truth masks.

				AP (%) $\uparrow$
Variant	(A)	(B)	(C)	$\mathrm{IoU}_{75}^{*}$	$5^{\circ}$ 5 cm	$10^{\circ}$ 5 cm
RGB-D component configurations
Base RGB-D predictor	–	–	–	27.8	32.9	69.0
Pooled descriptors + MLP	$\checkmark$	–	–	29.0	36.2	65.4
w/o pose refinement	$\checkmark$	$\checkmark$	–	23.0	44.2	63.3
w/o query feedback	$\checkmark$	–	$\checkmark$	49.9	37.6	70.6

TABLE IV: Component configurations and early-fusion alternatives on REAL275. ∗ CATRE IoU.

(A) Query-conditioned geometry
Variant	$\mathrm{IoU}_{75}^{*}$	$5^{\circ}$ 5 cm	$10^{\circ}$ 5 cm
RYOPO	52.7	51.1	73.6
w/o mask weighting in pooling	48.2	53.2	74.9
w/o depth consensus	49.6	50.6	71.0
(B) Geometry feedback
Scene CNN replaced by MLP	52.8	36.9	72.1

TABLE V: Within-component controls on REAL275 (AP, %). ∗ CATRE IoU.

Fig. 3: Instance recovery, pose alignment, and nearby-frame consistency on REAL275 and HouseCat6D. Each column compares the same crop across methods. The mug views share a crop and illustrate pose accuracy and consistency across nearby observations; $t$ , $t+3$ , and $t+11$ denote source-frame offsets. All predictions are shown without yaw alignment at a $0.3$ confidence threshold. GT/predicted cuboids are white/green. Canonical axes are RGB: dashed with hollow tips for GT, solid with filled tips for predictions; symmetric objects show only the observable up axis.

Method	$D$	$K{=}1$	$K{=}4$	$K{=}8$	$K{=}8$
		Cropped pipeline $P(K)$			Total
CleanPose	26.1	72.7	79.7	106.5	132.7
AG-Pose	26.1	71.0	82.9	107.3	133.4
HS-Pose	26.1	20.1	43.8	70.3	96.5
VI-Net	26.1	76.8	83.2	88.4	114.6
DPDN	26.1	54.8	69.0	88.1	114.2

TABLE VI: Pose-output latency and supplied-object scaling (ms/frame).

K

: supplied candidates;

P(K)

: cropped pose pipeline;

D

: Mask R-CNN .

CAMERA25
Variant	IoU 50	IoU 75	IoU ${}_{75}^{*}$	$5^{\circ}$ 2 cm	$5^{\circ}$ 5 cm	$10^{\circ}$ 2 cm	$10^{\circ}$ 5 cm	FPS †
RYOPO	98.6	96.7	81.5	79.8	85.8	86.0	92.5	–
Mask-free	98.4	96.3	78.5	77.0	85.1	83.4	92.0	–
REAL275
RYOPO	93.0	84.5	52.7	38.7	51.1	57.7	73.6	31.8
Mask-free	91.9	81.3	52.0	36.7	48.5	56.8	71.9	51.4

TABLE S1: RYOPO and its mask-free variant. All-object AP (%); ∗ CATRE IoU, other IoU independent; † deployed FPS on REAL275.

Fig. S1: Mask-free query-conditioned geometry. Box-aligned samples use depth validity

v_{i}

without mask confidence

m_{i}

. Depth-consensus pooling, descriptor construction, geometry feedback (B) , and pose refinement (C) are retained.

Fig. S2: RYOPO and mask-free pose predictions on the same REAL275 and HouseCat6D object crops. Green cuboids show object extent; red, green, and blue axes indicate the canonical X, Y, and Z directions.

Cls. weight	Box AP 50	IoU ${}_{75}^{*}$	$5^{\circ}$ 5 cm	$10^{\circ}$ 5 cm
0.5	83.1	24.2	24.6	42.6
1.0	86.9	26.9	21.9	42.2
2.0	85.2	20.8	21.8	40.2
3.0	85.8	20.1	14.6	38.8
5.0	90.8	20.8	19.1	39.4

TABLE S2: Classification-loss weight on HouseCat6D with mask-free RYOPO. Scores in %; ∗ CATRE IoU; pose AP uses all objects.

Fig. S3: Live mask-free hex-bolt predictions during object placement, partial hand occlusion, and camera viewpoint changes. Green cuboids and colored axes are predictions. Times refer to the source recording.

Fig. S4: Live predictions on the training toy and two unseen instances. Top: front-facing views. Bottom: rear views (left, center) and an oblique side view (right). The training instance is partially hand-occluded.

Explore similar work

UniPose9D: Universal Category-Agnostic Object Pose Estimation
Jul 10, 2026Yang You, Yi Du, Cole Harrison +1Object Pose Estimation
Stanford University · Amazon
Symmetry-Aware 9D Pose Estimation with Sim(3)-Consistent Feature and Spherical Inception Convolution
Jun 1, 2026Panfei Cheng, Hongshan Yu, Wenrui Chen +3Object Pose Estimation Convolutional Neural Networks
National Engineering Research Center for Robot Visual Perception and Control, School of Robotics and Artificial Intelligence, Hunan University, Changsha 410012, China
MemPose: Category-level Object Pose Estimation with Memory
Jul 6, 2026Xiao Lin, Minghao Zhu, Yun Peng +4Memory-Augmented Neural Networks Object Pose Estimation
Tongji University, Shanghai, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems

Jul 10, 2026, cs.CVJ/K move · Enter open · S save

UniPose9D: Universal Category-Agnostic Object Pose Estimation

Yang You, Yi Du, Cole Harrison, Leonidas Guibas

Stanford University · Amazon

Object Pose Estimation

Open paper AskPDF arXiv

Object pose estimation is a fundamental problem in 3D vision. Although recent state-of-the-art approaches achieve strong performance, generalization to novel categories and unseen scenes remains challenging. We propose UniPose9D, a unified model for category-agnostic 9D object pose estimation: given an instance mask/ROI and either an RGB-D observation or an RGB image with predicted depth, the model estimates rotation, translation, and metric size without category labels, CAD models, mean-shape priors, or reference views. Specifically, UniPose9D samples point pairs from the observed object geometry and uses DINOv2 and PointNet features to predict NOCS coordinates for each pair. To improve accuracy, we introduce a point-pair-based RANSAC N-hop Kabsch-Umeyama algorithm with an adaptive threshold. We further employ flow matching to address symmetric ambiguities and construct a large-scale training set by curating and aligning pose annotations from existing public datasets. Experiments across eight datasets show that a single unified model achieves competitive performance on standard benchmarks while generalizing to unseen objects, unseen categories, and in-the-wild scenarios. Our code and model are available at https://github.com/qq456cvb/UniPose9D.

Abstract

Explore similar work

UniPose9D: Universal Category-Agnostic Object Pose Estimation

Symmetry-Aware 9D Pose Estimation with Sim(3)-Consistent Feature and Spherical Inception Convolution

MemPose: Category-level Object Pose Estimation with Memory