Perceiving and reconstructing objects from images are critical for real-to-sim transfer tasks, which are widely used in the robotics community. Existing methods rely on multiple submodules such as detection, segmentation, shape reconstruction, and pose estimation to complete the pipeline. However, such modular pipelines suffer from inefficiency and cumulative error, as each stage operates on only partial or locally refined information while discarding global context. To address these limitations, we propose UniPR, the first end-to-end object-level real-to-sim perception and reconstruction framework. Operating directly on a single stereo image pair, UniPR leverages geometric constraints to resolve the scale ambiguity. We introduce Pose-Aware Shape Representation to eliminate the need for per-category canonical definitions and to bridge the gap between reconstruction and pose estimation tasks. Furthermore, we construct a large-vocabulary stereo dataset, LVS6D, comprising over 6,300 objects, to facilitate large-scale research in this area. Extensive experiments demonstrate that UniPR reconstructs all objects in a scene in parallel within a single forward pass, achieving significant efficiency gains and preserves true physical proportions across diverse object types, highlighting its potential for practical robotic applications.
Figures & tables
Figure 3: Overview of Our Proposed UniPR. We present UniPR, a single-forward network capable of simultaneously processing multiple unknown objects. Taking stereo image pairs as input, UniPR first encodes the scene into Tri-Plane View features that comprehensively capture spatial and geometric information. Within the transformer decoder, object queries are employed to extract instance-specific features from these TPV embeddings, enabling the network to reason about multiple objects in parallel. The resulting object embeddings are then fed into specialized prediction heads to infer each object’s semantic label, 3D position, physical scale, and pose-aware shape representation.
Method
Requirement
CD ↓
CD@Top10 ↓
F-Score ↑
SPE ↓
Inference Time (s)
2D
Seg
Pose
Single
Whole Image
Trellis [ 33 ]
✓
✓
✓
0.1096
0.0220
0.334
0.475
8.62
43.08
Hunyuan2.1 [ 10 ]
✓
✓
✓
0.0644
0.0072
0.553
0.320
74.16
370.78
Ours
✗
✗
✗
0.0083
0.0026
0.883
0.109
0.63
0.63
Table 1: Quantitative comparison on reconstruction. Both Trellis and HunYuan2.1 require perfect 2D bounding boxes, segmentation masks, and poses as inputs, whereas our method operates in an end-to-end manner directly from stereo images. Here, CD denotes the Chamfer Distance, SPE refers to the Shape Proportion Error, and inference time is measured in seconds per object and per scene.
Method
Easy
Medium
Hard
AP ↑
APE ↓
ACD ↓
AP ↑
APE ↓
ACD ↓
AP ↑
APE ↓
ACD ↓
Coders [ 39 ]
0.102
2.200
2.348
0.112
2.144
3.084
0.070
2.230
10.146
Coders with PASR
0.495
1.332
0.850
0.572
1.522
1.225
0.483
1.711
1.816
Ours
0.702
0.885
0.413
0.764
1.096
0.761
0.752
1.248
1.224
Ours-48
0.759
0.762
0.311
0.824
0.951
0.604
0.748
1.101
1.035
Table 2: Quantitative comparisons on LVS6D dataset. Here, AP refers to the Average Precision of 3DIoU with a threshold of 50%. APE denotes the Average Position Error, and ACD represents the Average Chamfer distance. The results demonstrate that UniPR outperforms Coders across all dataset subsets, with particularly strong improvements observed in the Hard subset with high intra-class variations.
Figure 4: Qualitative shape reconstruction results compared with image-to-3D models. The results demonstrate the accurate preservation of shape proportions achieved by our proposed UniPR across various objects in the LVS6D dataset.
Figure 5: Qualitative pose-aware shape reconstruction results on LVS6D dataset. The results highlight the key role of PASR in simplifying rotation prediction, as it eliminates the ambiguity caused by different canonical definitions for categories with similar geometry.
Table 3: Comparison on public stereo datasets. The results highlight the effectiveness of the proposed PASR and its capability to decode accurate 6D object poses.