Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation

Nassim Ali Ousalah; Peyman Rostami; Vincent Gaudillière; Emmanuel Koumandakis; Anis Kacem; Enjie Ghorbel; Djamila Aouada

cs.CVMar 20, 2026

Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation

Authors: Nassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière, Emmanuel Koumandakis, Anis Kacem, Enjie Ghorbel, Djamila Aouada

Organizations: CVI2, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg⋆ · Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France† · Infinite Orbits⋊ · Cristal Laboratory, ENSI, University of Manouba‡

Abstract

In this paper, we address the problem of 6-DoF object pose estimation from a single RGB image. Indirect methods that typically predict intermediate 2D keypoints, followed by a Perspective-n-Point solver, have shown great performance. Direct approaches, which regress the pose in an end-to-end manner, are usually computationally more efficient but less accurate. However, direct pose regression heads rely on globally pooled features, ignoring spatial second-order statistics despite their informativeness in pose prediction. They also predict, in most cases, discontinuous pose representations that lack robustness. Herein, we therefore propose a covariance-pooled representation that encodes convolutional feature distributions as a symmetric positive definite (SPD) matrix. Moreover, we propose a novel pose encoding in the form of an SPD matrix via its Cholesky decomposition. Pose is then regressed in an end-to-end manner with a manifold-aware network head, taking into account the Riemannian geometry of SPD matrices. Experiments and ablations consistently demonstrate the relevance of second-order pooling and continuous representations for direct pose regression, including under partial occlusion.

Figures & tables

Figure 1: Pose–representation alignment . Pairs of images 2 2 2 Training images from the 002_master_chef_can class of the YCB-V [ 87 ] dataset. We use an EfficientNet-B6 [ 73 ] pretrained on ImageNet [ 15 ] with no prior training on pose estimation. are sorted according to the geodesic distances in $\mathrm{SE}(3)$ between their corresponding ground-truth poses (x-axis). The x-axis is then partitioned into bins. For each bin, we plot: (i) the average Log–Euclidean distance between spatial covariances of image pairs within this bin (red) and (ii) the average Cosine distance between flattened features of image pairs within the bin (blue). Covariance distances increase with pose distance, whereas flattened feature distances remain nearly constant, indicating that spatial covariance is more correlated with the pose.

Figure 2: Overview of the proposed pose estimation framework (best viewed in color). (A) General overview. A CNN backbone extracts feature maps $\mathbf{F}\!\in\!\mathbb{R}^{C\times H\times W}$ . Spatial covariance pooling over the $H{\times}W$ grid gives $\boldsymbol{{\hat{\Sigma}}}\!\in\!\mathcal{S}^{H{\times}W}_{++}$ which is fed to the pose head. (B) Learning the pose. After $L$ geometry-aware layers, the compact SPD matrix $\mathbf{\mathbf{}}{\mathbf{\Sigma}_{L}}\!\in\!\mathcal{S}^{4}_{++}$ is decoded by a differentiable Cholesky layer into $(\mathbf{u},\mathbf{v};\mathbf{t})$ ; $(\mathbf{u},\mathbf{v})$ are mapped to $\mathcal{SO}(3)$ and the loss is computed on $\mathcal{SO}(3)\times\mathbb{R}^{3}$ . (C) SPD learning layers. Each layer applies a BiMap followed by eigenvalue rectification $\operatorname{ReEig}_{\varepsilon}$ [ 32 ] preserving strict positive definiteness.

Figure 3: Spatial Covariance Pooling. Extracted features are flattened and centered and $2^{\text{nd}}$ order statistics are constructed pixel-wise. Every entry of the covariance matrix describes how two spatial locations of the given input co-variate.

Method		Ape	Bvise.	Cam	Can	Cat	Drill	Duck	Eggbox	Glue	Holep.	Iron	Lamp	Phone	Avg. $\uparrow$
Indirect	PVNet [ 66 ]	43.62	99.90	86.86	95.47	79.34	96.43	52.58	99.15	95.66	81.92	98.88	99.33	92.41	86.27
	CheckerPose [ 49 ]	—	—	—	—	—	—	—	—	—	—	—	—	—	97.1
	VAPO [ 48 ]	—	—	—	—	—	—	—	—	—	—	—	—	—	97.1
End-to-End	Pix2Pose [ 59 ]	58.1	91.0	60.9	84.4	65.0	76.3	43.8	96.8	79.4	74.8	83.4	82.0	45.0	72.4
	Self6D++ [ 78 ]	76.0	91.6	97.1	99.8	85.6	98.8	56.5	91.0	92.2	35.4	99.5	97.4	91.8	85.6
	DeepIM [ 45 ]	77.0	97.5	93.5	96.5	82.1	95.0	77.7	97.1	99.4	52.8	98.3	97.5	87.7	88.6

Table 1: LM results in terms of ADD(-S). Best result in bold ; second best underlined . Cov2Pose (Ours) outperforms all non-P

n

P methods and slightly surpasses P

n

P-based ones.“—” indicates values not reported in the original paper. Classes in italics are symmetric.

Method		Ape	Can	Cat	Drill	Duck	Eggb.	Glu.	Holep.	Avg. $\uparrow$
Indirect	ZebraPose [ 72 ]	57.9	95.0	60.6	94.8	64.5	70.9	88.7	83.0	76.9
	CheckerPose [ 49 ]	58.3	95.7	62.3	93.7	69.9	70.0	86.4	83.8	77.5
	VAPO [ 48 ]	—	—	—	—	—	—	—	—	78.02
	6D-Diff [ 88 ]	60.6	97.9	63.2	96.6	67.2	73.5	92.0	85.5	79.6
End-to-End	PoseCNN [ 87 ]	9.6	45.2	0.93	41.4	19.6	22.0	38.5	22.1	24.9
	Pix2Pose [ 59 ]	—	—	—	—	—	—	—	—	36.3

Table 2: LM-O results in terms of ADD(-S). Group-wise best in bold, group-wise second underlined. Cov2Pose (Ours) outperforms all No-PnP methods and is slightly behind ZebraPose [ 72 ] with only a

0.1\%

difference.

Method		ADD(-S)	AUC of ADD-S	AUC of ADD(-S)
Indirect	PVNet [ 66 ]	–	–	73.4
	ZebraPose [ 72 ]	80.5	90.1	85.3
	CheckerPose [ 49 ]	81.4	91.3	86.4
	VAPO [ 48 ]	84.9	92.3	87.9
	6D-Diff [ 88 ]	83.8	91.5	87.0
End-to-End	Pix2Pose [ 59 ]	45.7	—	—

Table 3: YCB-V results in terms of ADD(-S), AUC of ADD(-S)/ADD(-S). The higher the better. Group-wise best in bold, group-wise second underlined.

Rep.	SPD size	Cont. on $\mathcal{SO}(3)$	Avg. ADD(-S) $\uparrow$
Euler angles	$3{\times}3$	✗	70.9
6D+GS (Ours)	$4{\times}4$	✓	76.8

Table 4: Comparison with an alternative rotation parameterization. Under identical settings, we replace our 6D + differentiable GS rotation with an Euler-angle parameterization encoded as a

3{\times}3

SPD. The 6D+GS choice results in higher performance.

Variant	(A)	(B)	(C)	Cov2Pose
Avg. ADD(-S) $\uparrow$	31.0	70.9	72.3	76.8

Table 5: Comparison between Ours , (A) Euclidean regression head, (B) Channel-wise covariance, and (C) Training on log tangent space of SPD manifold.

Figure 4: Qualitative results. Green 3D boxes denote GT poses, whereas blue 3D boxes denote predictions. Cov2Pose predicts accurate poses in cluttered scenes and under partial occlusion.

Method		Time	ADD-(S)
Method		(ms)	LM [ 26 ]	LM-O [ 7 ]	YCB-V [ 87 ]
Indirect	PVNet [ 66 ]	44.9	86.27	—	—
	ZebraPose [ 72 ]	119.3	—	76.9	80.5
	CheckerPose [ 49 ]	116.7	97.1	77.5	81.4
	VAPO [ 48 ]	69.8	97.1	78.02	84.9
End-to-End	Pix2Pose [ 59 ]	123	72.4	36.3	45.7

Table 6: Comparison of inference time and ADD-(S) across datasets. We measured inference time of the open-source baselines on the same hardware as ours.

Variant	(A)	(B)	(C)	Cov2Pose
Ape	27.2	56.1	62.6	64.0
Can	40.8	93.8	94.4	96.4
Cat	15.7	70.0	66.1	74.1
Driller	26.7	88.1	91.9	96.3
Duck	16.7	49.3	48.1	55.8
Eggbox	47.5	48.0	53.6	57.9

Table S1 : Per-object results on LM-O . We compare our full method with ablated variants of Cov2Pose to isolate the contribution of each component of the SPD head. This table is the detailed per-object version of Table 5 .

Figure S1 : Variant (A) . The spatial covariance matrix $\boldsymbol{\hat{\Sigma}}$ is vectorized and fed to a two-layer MLP that regresses pose purely in Euclidean space.

Figure S2 : Variant (C) . We remove the Cholesky-based pose decoder and train on the log-tangent space of the SPD manifold. $\mathbf{Z}_{L}$ and $\mathbf{Z}_{\text{gt}}$ are mapped via $\operatorname{LogEig}$ to the log–tangent space. For clarity, a single tangent space $T_{\mathcal{S}_{++}^{4}}$ is depicted.

Figure S3 : Additional Pose–representation alignment. As in Figure 1 , image pairs are sorted by ground-truth $\text{SE}(3)$ distance. We plot Log–Euclidean distances between spatial covariances and, for comparison, cosine and Euclidean distances between flattened features. Covariance distances grow with distances in $\text{SE}(3)$ , while flattened-feature distances remain nearly constant, reinforcing the hypothesis that spatial covariances are more correlated with true pose.

$H{\times}W$	CovPool	SPD Head
$H{\times}W$	(ms)	(ms)
$8^{2}$	0.38	7.8
$\mathbf{17^{2}}$	0.50	23.8
$25^{2}$	0.61	68.6

Table S2: CovPool layer and SPD head latencies across different spatial resolutions.

Class (Ape; LM-O)	ADD $\uparrow$
ViT+MLP	33.3
ViT+Cov+SPD head	40.6

Table S3 : Pilot comparison on Ape (LM-O) with a ViT_b_16 backbone. Results reported in ADD.

Method	CAD?	Egg.	Glu.
Method	CAD?	ADD-S $\uparrow$	ADD-S $\uparrow$
Cov2Pose	$\times$	57.9	93.6
Cov2Pose + $\mathcal{L}_{\text{ADD}\text{-}\text{S}}$	✓	55.0	84.8
Cov2Pose + $\mathcal{L}_{\mathbf{R}\text{,sym}}$ [ 79 ]	✓	57.8	93.8

Table S4 : Pilot study on symmetric LM-O objects using point-based and symmetry-aware losses.

Method	K. College	Hospital	Shop Facade	St. Mary	Avg. $\downarrow$
PoseNet [ 40 ]	1.92, 5.40 ∘	2.31, 5.38 ∘	1.46, 8.08 ∘	2.65, 8.48 ∘	2.08, 6.83 ∘
BayesianPN [ 38 ]	1.74, 4.06 ∘	2.57, 5.14 ∘	1.25, 7.54 ∘	2.11, 8.38 ∘	1.91, 6.28 ∘
GPoseNet [ 39 ]	1.61, 2.29 ∘	2.62, 3.89 ∘	1.14, 5.73 ∘	2.93, 6.46 ∘	2.07, 4.59 ∘
Ours	1.57 , 1.81 ∘	2.08 , 2.35 ∘	1.11 , 5.33 ∘	2.06 , 5.00 ∘	1.70 , 3.56 ∘

Table S5: APR results on Cambridge Landmarks [ 40 ] . Median translation [m] and rotation [ ∘ ] errors per scene and their average (Lower is better). Cov2Pose achieves the best performance across all scenes and on average.

Method	PoseCNN [ 87 ]			Single-Stage [ 30 ] †			DeepIM [ 45 ]			GDR-Net [ 79 ] †			Ours
	ADD(-S)	AUC of	AUC of	ADD(-S)	AUC of	AUC of	ADD(-S)	AUC of	AUC of	ADD(-S)	AUC of	AUC of	ADD(-S)	AUC of	AUC of
	ADD(-S)	ADD-S	ADD(-S)	ADD(-S)	ADD-S	ADD(-S)	ADD(-S)	ADD-S	ADD(-S)	ADD(-S)	ADD-S	ADD(-S)	ADD(-S)	ADD-S	ADD(-S)
002_master_chef_can	—	84.0	50.9	—	—	—	—	—	—	41.5	96.3	65.2	61.0	94.7	63.1
003_cracker_box	—	76.9	51.7	—	—	—	—	—	—	83.2	97.0	88.8	86.7	91.2	79.8
004_sugar_box	—	84.3	68.6	—	—	—	—	—	—	91.5	98.9	95.0	94.5	94.7	89.8
005_tomato_soup_can	—	80.9	66.0	—	—	—	—	—	—	65.9	96.5	91.9	73.8	95.6	89.7

Table S6 : YCB-V per-object results in terms of ADD(-S) and AUC-S/AUC(-S).“—” indicates values not reported in the original paper. Classes denoted with ∗ are symmetric.

		YOLOv8s-pose [ 6 ]	† SPNv2 [ 60 ]	Ours
	Latency (ms)	—	80.25	60.46
Synthetic	$E_{T}$ [m]	—	—	0.184
	$E_{R}$ [ ∘ ]	—	—	3.462
	$E_{\text{pose}}$	—	—	0.089
Lightbox	$E_{T}$ [m]	0.758	0.368	0.300
	$E_{R}$ [ ∘ ]	18.00	20.258	15.096

Table S7 : Results on SPEED+. Performance in terms of Translation

E_{T}

, Rotation

E_{R}

and Pose

E_{\text{pose}}

errors as defined in [ 60 ] . The inference time of SPNv2 is measured on the same device and at the same input resolution (768

\times

512) as ours.

Figure S4 : Qualitative results on SPEED+. Examples of accurate pose predictions from Cov2Pose on the synthetic (1 ${}^{\text{st}}$ row) and lightbox (2 ${}^{\text{nd}}$ row) subsets of SPEED+ [ 62 ] . Green bounding boxes show predicted poses, while blue bounding boxes indicate ground-truth poses.

Riemannian Neural Networks

See all 10 papers

Generative 6D Pose Estimation via Conditional Flow Matching
Feb 23, 2026Amir Hamza, Davide Boscaini, Weihang Li +2Object Pose Estimation Conditional Flow Matching
Fondazione Bruno Kessler · University of Trento · Technical University of Munich +1
PnP-Transformer: A Neural Pose Solver for Zero-Shot 6D Object Pose Estimation
Oct 19, 2026Emirkan Burak Yilmaz, Yakup GencObject Pose Estimation
Computer Engineering Dept., Gebze Technical University, Kocaeli, Türkiye
MAPRPose: Mask-Aware Proposal and Amodal Refinement for Multi-Object 6D Pose Estimation
Apr 22, 2026Yang Luo, Yan Gong, Yongsheng Gao +2Object Pose Estimation Amodal Segmentation
aState Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin, 150001, China · bSchool of Civil Engineering, Harbin Institute of Technology, Harbin, 150001, China

Feb 23, 2026J/K move · Enter open · S save

Generative 6D Pose Estimation via Conditional Flow Matching

Amir Hamza, Davide Boscaini, Weihang Li, Benjamin Busam+1

Fondazione Bruno Kessler · University of Trento · Technical University of Munich+1

Object Pose Estimation Conditional Flow Matching

Open paper AskPDF arXiv

Existing methods for instance-level 6D pose estimation typically rely on neural networks that either directly regress the pose in

\mathrm{SE}(3)

or estimate it indirectly via local feature matching. The former struggle with object symmetries, while the latter fail in the absence of distinctive local features. To overcome these limitations, we propose a novel formulation of 6D pose estimation as a conditional flow matching problem in

\mathbb{R}^3

. We introduce Flose, a generative method that infers object poses via a denoising process conditioned on local features. While prior approaches based on conditional flow matching perform denoising solely based on geometric guidance, Flose integrates appearance-based semantic features to mitigate ambiguities caused by object symmetries. We further incorporate RANSAC-based registration to handle outliers. We validate Flose on five datasets from the established BOP benchmark. Flose outperforms prior methods with an average improvement of +4.5 Average Recall. Project Website : https://tev-fbk.github.io/Flose/

Abstract

Riemannian Neural Networks

Explore similar work

Generative 6D Pose Estimation via Conditional Flow Matching

PnP-Transformer: A Neural Pose Solver for Zero-Shot 6D Object Pose Estimation

MAPRPose: Mask-Aware Proposal and Amodal Refinement for Multi-Object 6D Pose Estimation