PnP-Transformer: A Neural Pose Solver for Zero-Shot 6D Object Pose Estimation

Emirkan Burak Yilmaz; Yakup Genc

cs.CVOct 19, 2026

PnP-Transformer: A Neural Pose Solver for Zero-Shot 6D Object Pose Estimation

Authors: Emirkan Burak Yilmaz, Yakup Genc

Organizations: Computer Engineering Dept., Gebze Technical University, Kocaeli, Türkiye

Abstract

We present PnP-Transformer, a neural pose solver designed as an iterative-free, differentiable alternative to classical PnP-RANSAC algorithms. PnP-Transformer regresses the 6D object pose by processing predicted semi-dense 2D-3D correspondence fields alongside confidence scores, while mapping geometric inputs into a continuous latent space. To evaluate its practical utility, we integrate PnP-Transformer into a zero-shot 6D pose estimation pipeline, where the correspondence module and pose solver are trained separately. PnP-Transformer achieves a 41.2% mean AR score with low latency (0.46 s/obj) on seven core BOP benchmark datasets, surpassing all prior baselines (such as RayPose with 39.1% AR) but one (Co-op with 58.4% AR). When the pose solver is replaced with EPnP-RANSAC (similar to Co-op) while keeping the correspondence module unchanged, the RANSAC alternative outperforms PnP-Transformer on six of the seven datasets by reaching 52.2% mean AR at a cost of an additional 60 ms per object, revealing an accuracy-latency trade-off. Code and pretrained models are available at https://github.com/ebylmz/pnp-transformer.

Figures & tables

Figure 1: Proposed architecture. (a) Method Overview: During an offline onboarding stage, RGB-D templates are rendered from the 3D CAD model and processed through a ViT encoder to acquire the template features. At inference, a query image crop is similarly processed using the same encoder to obtain query features. A transformer decoder and following heads then predict the patch-level match matrix $\mathbf{M}$ and displacement offset $\mathbf{O}$ across all query-template pairs. The template with the highest overall match confidence is selected, and its corresponding 2D–3D correspondence field with associated point-wise confidences $(u,X,c)$ is fed into PnP-Transformer to regress the relative transformation $\Delta=(\mathbf{R},\mathbf{t})\in\mathrm{SE}(3)$ while producing the latent representation $\mathbf{z}$ of the recovered transformation. (b) PnP-Transformer Module: PnP-Transformer consists of three primary sub-modules: PnP-Encoder that processes input correspondences into a geometric latent vector $\mathbf{z}$ , Pose-Encoder used during training to map ground-truth target poses $(\mathbf{R}_{\text{gt}},\mathbf{t}_{\text{gt}})$ into target embeddings $\bar{\mathbf{z}}$ , and Pose-Decoder that regresses final relative 6D pose parameters from $\mathbf{z}$ .

Method	RANSAC	LM-O	T-LESS	TUD-L	IC-BIN	ITODD	HB	YCB-V	Mean	Time(s)
MegaPose [ 17 ]	$\times$	22.9	17.7	25.8	15.2	10.8	25.1	28.1	20.8	15.46
GenFlow [ 21 ]	$\times$	25.0	21.5	30.0	16.8	15.4	28.3	27.7	23.5	3.84
GigaPose [ 24 ]	✓	29.9	27.3	30.2	23.1	18.8	34.8	29.0	27.6	0.38
FoundPose [ 27 ]	✓	39.7	33.8	46.9	23.9	20.4	50.8	45.2	37.3	1.69
RayPose [ 14 ]	$\times$	42.1	36.9	48.3	21.8	–	–	46.2	39.1	–
Ours	$\times$	45.6	24.0	46.3	35.5	19.1	63.2	54.4	41.2	0.46

Table 1: Main results on BOP benchmark datasets. Comparison of our method against RGB-only coarse pose estimation baselines by reporting the Average Recall (AR) scores on the seven core BOP [ 25 ] datasets.

Figure 2: Representative pose estimation results on BOP datasets. Predicted ( red ) and ground-truth ( green ) object contours overlaid on the input images. Rows 1 and 2 represent success cases for PnP-Transformer and PnP-RANSAC, respectively, while Rows 3 and 4 show their corresponding failure cases.

Configuration	LM-O	T-LESS	TUD-L	IC-BIN	ITODD	HB	YCB-V	Mean AR
w/o $\mathcal{L}_{\text{lat}}$ (no Pose-Encoder)	43.0	18.4	45.9	34.6	18.7	62.1	52.1	39.3
w/ $\mathcal{L}_{\text{lat}}$ (Full Model)	45.6	24.0	46.3	35.5	19.1	63.2	54.4	41.2

Table 2: Ablation of the Pose-Encoder and latent alignment loss

\mathcal{L}_{\text{lat}}

. Evaluating the Average Recall (AR) on the seven core BOP datasets with and without latent alignment supervision during training.

Figure 3: Visualization of the learned pose latent space. (a) Trajectory Tracing : Background points represent camera poses sampled across an icosphere, color-coded by azimuth angle. The mapped trajectory of a continuous $360^{\circ}$ camera orbit around the world Y-axis forms a closed loop in the latent space. (b) Latent Additivity : A relative displacement vector $\Delta\mathbf{z}=\mathbf{z}_{B}-\mathbf{z}_{A}$ extracted from a source pose pair (blue) is applied to neighboring starting points $\mathbf{z}_{\text{start}}$ . The resulting transferred shifts (red arrows) demonstrate similar displacements.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 184 papers

TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers
Aug 15, 2026Paul Julius Kühn, Duc Anh Nguyen, Saptarshi Neil Sinha +2Object Pose Estimation Edge Inference
Fraunhofer IGD 64283 Darmstadt, Germany · TU Darmstadt, 64289 Darmstadt, Germany · TU Delft 2628 CD Delft, Netherlands
Back to the Feature: Zero-Shot 6DoF Pose Estimation via Dense Local Features
Sep 6, 2026Ali Rafiaei, Michael GreenspanObject Pose Estimation Local Feature Matching
Dept. Electrical & Computer Engineering, Ingenuity Labs Queen’s University Kingston, ON, Canada
Generative 6D Pose Estimation via Conditional Flow Matching
Feb 23, 2026Amir Hamza, Davide Boscaini, Weihang Li +2Object Pose Estimation Conditional Flow Matching
Fondazione Bruno Kessler · University of Trento · Technical University of Munich +1

Aug 15, 2026J/K move · Enter open · S save

TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers

Paul Julius Kühn, Duc Anh Nguyen, Saptarshi Neil Sinha, Michael Weinmann+1

Fraunhofer IGD 64283 Darmstadt, Germany · TU Darmstadt, 64289 Darmstadt, Germany · TU Delft 2628 CD Delft, Netherlands

Object Pose Estimation Edge Inference Detection Transformer

Open paper AskPDF arXiv

Real-time 6DoF object pose estimation on resource-constrained hardware remains challenging, as accurate correspondence-based and refinement pipelines typically rely on non-differentiable PnP/RANSAC stages or costly iterative refinement, while recent foundation-model-based approaches incur inference costs that are prohibitive for edge deployment. We present TinyDETR-Pose, a lightweight, end-to-end, single-stage framework that jointly detects objects and regresses their full 6D pose in a single forward pass. Built on the efficient LW-DETR architecture, TinyDETR-Pose formulates detection and pose estimation as a set-prediction problem and attaches dedicated MLP heads for rotation, monocular depth, and projected object center regression to each decoder query, eliminating the need for PnP, NMS (non-maximum suppression), or iterative pose refinement. Object symmetries are handled through a ADD-S loss applied uniformly to all objects, without the need for object-specific loss schedules or separate geodesic/ADD supervision. In addition, predictions are assigned to ground truth using a symmetry-safe Hungarian matcher based on class and 2D spatial cues, yielding stable assignment under symmetry and depth ambiguity. On YCB-V, TinyDETR-Pose achieves a comparable ADD-S AUC of 85.9, while requiring up to 72.7% fewer parameters than other DETR-based single-stage pose-estimation approaches. Due to its compact design, TinyDETR-Pose runs in real time and achieves an inference latency of only ~4.5 ms per frame on an NVIDIA Jetson Nano using TensorRT, demonstrating that accurate end-to-end transformer-based 6D pose estimation can be made practical for edge deployment.

Abstract

Object Pose Estimation

Explore similar work

TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers

Back to the Feature: Zero-Shot 6DoF Pose Estimation via Dense Local Features

Generative 6D Pose Estimation via Conditional Flow Matching