cs.CVApr 8, 2026
SaveExploring 6D Object Pose Estimation with Deformation
Organizations: State Key Laboratory of ISN, Xidian University · MagicLeap
Abstract
We present DeSOPE, a large-scale dataset for 6DoF deformed objects. Most 6D object pose methods assume rigid or articulated objects, an assumption that fails in practice as objects deviate from their canonical shapes due to wear, impact, or deformation. To model this, we introduce the DeSOPE dataset, which features high-fidelity 3D scans of 26 common object categories, each captured in one canonical state and three deformed configurations, with accurate 3D registration to the canonical mesh. Additionally, it features an RGB-D dataset with 133K frames across diverse scenarios and 665K pose annotations produced via a semi-automatic pipeline. We begin by annotating 2D masks for each instance, then compute initial poses using an object pose method, refine them through an object-level SLAM system, and finally perform manual verification to produce the final annotations. We evaluate several object pose methods and find that performance drops sharply with increasing deformation, suggesting that robust handling of such deformations is critical for practical applications.
Figures & tables
Figure 2: Comparison of 6D object pose datasets. The first row shows two examples from an instance-level dataset [ 4 ] , where each object instance is associated with its own 3D model, under the assumption that the object is perfectly rigid and does not deform over time. The second row depicts a category-level dataset [ 7 ] , in which multiple instances of the same category share a single 3D model (rightmost). The third row shows the canonical instance from the proposed DeSOPE dataset along with three deformed versions, while the fourth row presents their corresponding 3D meshes. We visualize meshes as black-bordered images on a white background in this figure.
| Type | Categories | Instances | Images | |
|---|---|---|---|---|
| LINEMOD [ 4 ] | Instance | - | 15 | 19K |
| YCBV [ 7 ] | Instance | - | 21 | 80K |
| PhoCaL [ 46 ] | Instance | 8 | 60 | 3K |
| Wild6D [ 17 ] | Instance | 5 | 162 | 10K |
| Objectron [ 1 ] | Instance | 9 | 17K | 4M |
| CO3D [ 37 ] | Instance | 5 | 19K | 1.5M |
Table 1: Comparison of 6D object pose datasets. Most instance-level datasets focus on individual instances, assuming objects are perfectly rigid and treating each instance independently. Category-level datasets, on the other hand, provide a single canonical mesh representing all instances within a semantic category. The proposed DeSOPE dataset captures multiple deformation states of the same instance, offering accurate 3D registration between the deformed state and the canonical form, along with precise 6D object pose annotations in images captured across diverse scenes.
Figure 3: Overview of the dataset generation framework. The framework consists of four main steps: Object Scanning , which acquires the canonical mesh of objects along with multiple deformed states of the same instance using a high-precision 3D scanner; Model Alignment , beginning with coarse manual alignment and followed by flow-driven 3D registration [ 20 ] ; Video Capture , which records RGB-D videos of objects across diverse scenes with a stereo camera; and Pose Annotation , which performs initial object labeling and iteratively refines poses using implicit neural networks to obtain accurate annotations.
Figure 4: Example of 3D model alignment. We estimate the optimal registration between each deformed mesh and its corresponding canonical mesh through a two-stage process. We begin with a manual alignment to obtain a coarse initialization, providing a reasonable starting point for further refinement. This initial estimate is then refined based on dense 2D matching computed from six orthogonal viewpoints. The resulting error map visualizes the pixel-wise differences between the canonical mesh and the aligned deformed mesh when projected from the same viewpoints, both before and after refinement.
Figure 5: Statistical analysis of the DeSOPE dataset. We summarize the different properties of our dataset using histograms and report percentages (%) on the y-axis across all subplots. (a) Distribution of camera pose angles (x-axis: rotation angle in degrees), illustrating the coverage of pitch, roll, and yaw across all annotated frames. (b) Distribution of object-to-camera distances (x-axis: distance in cm), with values concentrated around 50-60 cm. (c) Distribution of physical dimensions for 104 object instances across 26 categories (x-axis: size in cm), demonstrating the diversity in object sizes. (d) Distribution of deformation severity (x-axis: deformation magnitude in cm) across three levels: mild for Deformed 1, moderate for Deformed 2, and severe for Deformed 3. The deformation magnitude is computed as the average point-wise 3D distance between corresponding vertices after aligning the deformed and undeformed meshes.
Figure 6: Effect of pose refinement. We visualize the predicted pose by overlaying the rendered textured mesh onto the input image according to the estimated object pose. The initial pose exhibits noticeable misalignment; after applying our pose refinement strategy, the rendered mesh aligns much more accurately with the input image.
Figure 7: Example of captured images and pose annotations. The green boundary contours represent pose projections onto the 2D plane using the corresponding mesh, as obtained by the annotation algorithm proposed in this paper. The dataset contains images captured in cluttered scenes under both human-manipulated and non-manipulated conditions.
| Deformed 1 | Deformed 2 | Deformed 3 | |||
|---|---|---|---|---|---|
| Init. | +Refine | Init. | +Refine | Init. | +Refine |
| 0.473 | 0.261 | 0.938 | 0.440 | 1.177 | 1.010 |
| 0.657 | 0.191 | 1.050 | 0.511 | 1.306 | 1.253 |
| 1.922 | 1.129 | 2.399 | 1.358 | 1.291 | 1.091 |
| 0.731 | 0.297 | 0.967 | 0.577 | 0.980 | 0.843 |
| 0.855 | 0.498 | 1.098 | 0.576 | 0.672 | 0.690 |
Table 2: Results of 3D model alignment. “Init.” denotes the matching error obtained from manual alignment between deformed and undeformed objects, while “+Refine” indicates the error after refinement. The six middle rows report the alignment errors for each of the six projected views, and the final row summarizes the overall alignment error. The error is reported in centimeters after RANSAC outlier removal with a threshold of 3 cm.
| SCFlow2 | FoundationPose | GenPose | |
|---|---|---|---|
| Canonical | 0.82 | 0.78 | 0.67 |
| Deformed 1 | 0.67 | 0.58 | 0.56 |
| Deformed 2 | 0.43 | 0.38 | 0.36 |
| Deformed 3 | 0.23 | 0.24 | 0.31 |
| Canonical | 0.77 | 0.72 | 0.61 |
| Deformed 1 | 0.64 | 0.54 | 0.53 |
Table 3: Evaluation of state-of-the-art methods on DeSOPE. We report results in two groups: the first is evaluated on all the images without human manipulation, and the second on images with human manipulation. We report the Average Recall (AR) for three representative methods: SCFlow2 [ 47 ] , FoundationPose [ 48 ] , and GenPose [ 53 ] . Although these methods generalize well, their performance degrades significantly when the meshes in the images do not match the canonical mesh assumed by the model. The more severe the deformation, the worse their performance.
Figure 8: State-of-the-art methods on DeSOPE. Most methods achieve strong performance on images with canonical meshes (first row). However, their accuracy degrades significantly when the meshes undergo deformations that deviate from the canonical configuration (second row), as they assume the target in the image still conforms to the canonical mesh, which is not the case. Pose estimation results are projected onto the 2D plane using the corresponding mesh. Color code: green—Ground Truth; red—GenPose; pink—FoundationPose; cyan—SCFlow2.
Figure 9: Performance analysis. All plots present Average Recall (AR) across four mesh sets (Canonical and Deformed 1–3) for three methods: SCFlow2, FoundationPose (FPose), and GenPose. Key observations: (1) performance decreases as deformation severity increases across all settings; (2) greater occlusion leads to lower performance; (3) scenes with human manipulation consistently perform worse due to complex motion and occlusions during interaction.
More in this topic
Object Pose Estimation
Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.
See all 187 papers