Learning Positive-Incentive Point Sampling in Neural Implicit Fields for Object Pose Estimation
Authors: Yifei Shi, Boyan Wan, Xin Xu, Kai Xu
Organizations: College of Intelligence Science and Technology, National University of Defense Technology, China · College of Computer Science, National University of Defense Technology, China
Learning neural implicit fields of 3D shapes is a rapidly emerging field that enables shape representation at arbitrary resolutions. Due to the flexibility, neural implicit fields have succeeded in many research areas, including shape reconstruction, novel view image synthesis, and more recently, object pose estimation. Neural implicit fields enable learning dense correspondences between the camera space and the object's canonical space-including unobserved regions in camera space-significantly boosting object pose estimation performance in challenging scenarios like highly occluded objects and novel shapes. Despite progress, predicting canonical coordinates for unobserved camera-space regions remains challenging due to the lack of direct observational signals. This necessitates heavy reliance on the model's generalization ability, resulting in high uncertainty. Consequently, densely sampling points across the entire camera space may yield inaccurate estimations that hinder the learning process and compromise performance. To alleviate this problem, we propose a method combining an SO(3)-equivariant convolutional implicit network and a positive-incentive point sampling (PIPS) strategy. The SO(3)-equivariant convolutional implicit network estimates point-level attributes with SO(3)-equivariance at arbitrary query locations, demonstrating superior performance compared to most existing baselines. The PIPS strategy dynamically determines sampling locations based on the input, thereby boosting the network's accuracy and training efficiency. Our method outperforms the state-of-the-art on three pose estimation datasets. Notably, it demonstrates significant improvements in challenging scenarios, such as objects captured with unseen pose, high occlusion, novel geometry, and severe noise.
Figures & tables
Fig. 1: We propose PIPS , a data-driven approach to dynamically determine where to sample to boost the network training, achieving better performance with training on fewer sampling points, compared to (a) the random sampling baseline. PIPS consists of two components: (b) positive-incentive point sampling with high estimation certainty ( PIPS-C ) and (c) positive-incentive point sampling with high geometric stability ( PIPS-S ).
Fig. 2: (a) Point-wise canonical coordinate estimation on as few as three points (in red) is sufficient for determining all the 6-DoFs of object pose. (b) Extra voters with inaccurate point-level estimations (in blue) would degrade the performance.
Fig. 3: The quantitative comparisons of the proposed PIPS-C and PIPS-S to the baseline of random sampling. We see our method reduce the number of sample points and the training time while achieving better performance in object pose estimation. The experiment is conducted on the NOCS-REAL275 dataset.
Fig. 4: Overview of the proposed method. First, an SO(3)-equivariant convolutional implicit network with dense point sampling (the teacher model) is optimized to generate the pseudo ground-truth. Second, the PIPS-C and PIPS-S estimation networks (the student model) are trained based on the generated pseudo ground-truth. Third, an SO(3)-equivariant convolutional implicit network is trained with the sample points estimated by the PIPS estimation network.
Fig. 5: (a) By rotating the 3D graph convolution kernel via a regular icosahedron rotation group, we generate a set of convolutional kernels. (b) Point cloud convolutions with the rotated convolutional kernels make the generated features SO(3)-invariant. By generating vector neurons by the SO(3)-invariant point feature and multiplying the vector neurons with the rotation matrix Ra corresponding to the qa∈Q with the highest activation, the feature becomes SO(3)-equivariant.
Fig. 6: Architecture of the SO(3)-equivariant convolutional implicit network. Left: Multiple SO(3)-equivariant 3D graph convolution layers and point downsampling layers are adopted to aggregate multiple-scale point features. The RGB features are extracted by DINOv2. Right: The aggregate features are propagated to any 3D sample point via feature propagation layers, estimating the point-wise object canonical coordinate.
Fig. 7: The network architecture of the PIPS estimation network. The network takes a point cloud as input, embeds it into a high-dimensional feature space with an encoder, and generates the PIPS-C sample points with a decoder. A gating unit and a Gumbel-Softmax module are applied to generate the PIPS-S sample points. Multiple loss functions are adopted to train the network.
Fig. 8: A 2D example of PIPS-S selection. The dots are the input to the PIPS-S estimation component. The dots with an ellipse are the selected PIPS-S sample points, where the ellipse represents its anisotropic variances. (a) The pointset is unstable w.r.t. the alignment because it has a high variance along the y-axis direction, meaning that the locations along the y-axis are not well-constrained. (b) The pointset is stable w.r.t. the alignment since no DoF shows high variance for the points within the set, indicating the locations are well-constrained.
Fig. 9: The statistical comparisons between the training set and the test set of ShapeNet-C.
Dataset
Avg. instances per category
Avg. Chamfer distance to mean shape (mm)
Avg. occlusion (%)
NOCS-REAL275 [ 67 ]
4
2.7
46
NOCS-CAMERA25 [ 67 ]
31
2.4
44
Wild6D [ 77 ]
360
2.2
42
HouseCat6D [ 25 ]
19
2.8
49
ShapeNet-C
2000
3.1
53
TABLE I: The comparisons of ShapeNet-C to existing datasets.
Fig. 10: The qualitative comparisons on the NOCS-REAL275 dataset. Our method could accurately estimate the pose on all the tested objects, while the baselines cannot.
Methods
Input
Data source
Shape Prior
IoU75 ∗ ↑
5∘ 2cm ↑
5∘ 5cm ↑
10∘ 2cm ↑
10∘ 5cm ↑
SGPA [ 8 ]
RGB-D
Syn.+Real
✓
0.37
0.36
0.40
0.61
0.71
DPDN [ 35 ]
RGB-D
Syn.+Real
✓
0.54
0.46
0.51
0.70
0.78
HS-Pose [ 79 ]
D
Real
×
0.39
0.45
0.55
0.69
0.84
GenPose [ 78 ]
D
Real
×
0.50
0.52
0.61
0.72
0.84
VI-Net [ 36 ]
RGB-D
Real
×
0.48
0.50
0.58
0.71
0.82
SecondPose [ 9 ]
RGB-D
Syn.+Real
×
0.50
0.56
0.64
0.75
0.86
TABLE II: Quantitative comparisons on the NOCS-REAL275 dataset. ‘*’ denotes the IoU metrics computed as in [ 40 ] .
Methods
Input
Data source
Rotation
Translation
Mean( ∘ ) ↓
Median( ∘ ) ↓
5∘↑
Mean(cm) ↓
Median(cm) ↓
5∘5 cm ↑
NOCS [ 67 ]
RGB
Syn.
53.15
27.68
0.34
6.87
5.03
0.29
GPV-Pose [ 12 ]
D
Syn.
50.95
20.62
0.38
6.31
4.82
0.33
GenPose [ 78 ]
D
Syn.
48.29
11.81
0.41
5.76
4.29
0.37
VI-Net [ 36 ]
RGB-D
Syn.
46.45
9.81
0.47
5.55
3.89
0.42
SecondPose [ 9 ]
RGB-D
Syn.
45.77
7.87
0.49
5.42
3.91
0.45
TABLE III: Quantitative comparisons on the ShapeNet-C dataset.
Fig. 11: The qualitative comparisons on the ShapeNet-C dataset. The grey points denote the input points. The colored points represent the canonical shapes transformed with the estimated poses.
Methods
Data type
Refinement
AR
time
CDPNv2 [ 34 ]
RGB
×
62.4
0.98
NCF [ 18 ]
RGB
×
63.2
7.17
ZebraPose [ 59 ]
RGB
×
72.1
0.25
CosyPose [ 30 ]
RGB-D
✓
71.4
13.74
GDRNPP [ 41 ]
RGB-D
×
71.3
0.28
GDRNPP [ 41 ] (GPose)
RGB-D
✓
80.5
4.58
TABLE IV: Quantitative results on the LineMOD-O dataset. The best results are in bold, and the second-best results are underlined.
Fig. 12: The qualitative comparisons on the LineMOD-O dataset. The boundary of the shape transformed by the estimated pose is shown in the figure. Please pay attention to the highlighted objects.
Fig. 13: Visualization of the PIPS sample points. The results show that: 1) PIPS-C sample points are dense, distributed around the object surface, possibly around the unseen surface region, such as the occluded chair in (b); 2) instead of uniformly distributing around the object surface, the sampling rule varies on different shapes, such as the laptop in (a); 3) PIPS-S sample points are sparse, covering the crucial regions of the input shape; 4) adding noise to the input points will not substantially change the distribution of PIPS-S sample points, such as the two cases in last row of (b), where the sampled points are speared near the wings and tail of the airplane despite the noise. The results are produced with only the point cloud as input, without the RGB image features.
Methods
Median( ∘ ) ↓
Median(cm) ↓
5∘ 5cm ↑
w/o PIPS
5.92
2.78
0.51
w/o PIPS-C
5.34
2.61
0.53
w/o PIPS-S
4.23
2.49
0.57
w/o anisotropic variance
4.51
2.89
0.55
w/o SO(3)-equivariance
5.18
3.02
0.49
w/o RGB
3.48
2.67
0.59
TABLE V: Ablation study of our method on ShapeNet-C.
Fig. 14: Robustness evaluation. The quantitative comparisons of our method to a baseline method on the four challenging subsets of the NOCS-CAMERA25 dataset. Our method is more robust on unseen poses, novel shapes, high occlusion, and severe noise.
Voxel size
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
4
5.42
3.15
0.56
0.49
8
3.16
2.13
0.68
0.64
16
3.28
2.08
0.68
0.62
32
3.71
2.34
0.63
0.58
TABLE VI: Effect of voxel size h in PIPS-C estimation.
Target sparsity
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
0.3
3.42
2.65
0.63
0.56
0.2
3.28
2.38
0.68
0.61
0.1
3.16
2.13
0.66
0.62
0.05
3.71
2.84
0.59
0.52
TABLE VII: Effect of target sparsity ρ in PIPS-S estimation.
Threshold
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
0.9
4.32
3.24
0.57
0.49
0.7
3.89
2.95
0.61
0.54
0.5
3.16
2.13
0.66
0.62
0.3
3.58
2.75
0.63
0.56
TABLE VIII: Effect of threshold ω in positive sample selection of the PIPS estimation network.
Fig. 15: (a) The training plot of the generalizing to shape reconstruction experiment. (b) Examples of the reconstructed shapes during the network training.
Object pose estimation is a fundamental problem in 3D vision. Although recent state-of-the-art approaches achieve strong performance, generalization to novel categories and unseen scenes remains challenging. We propose UniPose9D, a unified model for category-agnostic 9D object pose estimation: given an instance mask/ROI and either an RGB-D observation or an RGB image with predicted depth, the model estimates rotation, translation, and metric size without category labels, CAD models, mean-shape priors, or reference views. Specifically, UniPose9D samples point pairs from the observed object geometry and uses DINOv2 and PointNet features to predict NOCS coordinates for each pair. To improve accuracy, we introduce a point-pair-based RANSAC N-hop Kabsch-Umeyama algorithm with an adaptive threshold. We further employ flow matching to address symmetric ambiguities and construct a large-scale training set by curating and aligning pose annotations from existing public datasets. Experiments across eight datasets show that a single unified model achieves competitive performance on standard benchmarks while generalizing to unseen objects, unseen categories, and in-the-wild scenarios. Our code and model are available at https://github.com/qq456cvb/UniPose9D.
Learning model-free object pose estimation for unseen instances remains a fundamental challenge in 3D vision. Existing methods typically fall into two disjoint paradigms: category-level approaches predict absolute poses in a canonical space but rely on predefined taxonomies, while relative pose methods estimate cross-view transformations but cannot recover single-view absolute pose. In this work, we propose Object Pose Transformer (\ours{}), a unified feed-forward framework that bridges these paradigms through task factorization within a single model. \ours{} jointly predicts depth, point maps, camera parameters, and normalized object coordinates (NOCS) from RGB inputs, enabling both category-level absolute SA(3) pose and unseen-object relative SE(3) pose. Our approach leverages contrastive object-centric latent embeddings for canonicalization without requiring semantic labels at inference time, and uses point maps as a camera-space representation to enable multi-view relative geometric reasoning. Through cross-frame feature interaction and shared object embeddings, our model leverages relative geometric consistency across views to improve absolute pose estimation, reducing ambiguity in single-view predictions. Furthermore, \ours{} is camera-agnostic, learning camera intrinsics on-the-fly and supporting optional depth input for metric-scale recovery, while remaining fully functional in RGB-only settings. Extensive experiments on diverse benchmarks (NOCS, HouseCat6D, Omni6DPose, Toyota-Light) demonstrate state-of-the-art performance in both absolute and relative pose estimation tasks within a single unified architecture.
Weihang Li, Lorenzo Garattoni, Fabien Despinoy +2
Technical University of Munich · Munich Center for Machine Learning · Toyota Motor Europe
Current object pose estimation research remains predominantly model-centric, focusing on architectural innovations and post-processing refinements. This paper introduces a data-centric optimization by proposing a novel, physically grounded rotation representation through principal axes alignment. Our method aligns the object's coordinate system with its inherent geometric axes, derived from inertial properties, yielding three key advantages: Inherent Stability-leveraging the energy-minimizing property of principal axes provides a robust representation that is less sensitive to noise and occlusions; Symmetry-Aware Canonicalization-explicitly resolving rotational ambiguities for symmetric objects at the data level, which fundamentally eliminates label confusion during network training; and Framework Agnosticism-the optimization is applied purely at the dataset level, ensuring plug-and-play compatibility with existing networks without any architectural modification. We validate the framework across diverse category-level and instance-level models. Extensive experiments demonstrate consistent and significant accuracy improvements, while preserving the integrity of the baseline network. This work establishes a new, geometry-driven direction for enhancing pose estimation, circumventing the need for complex network redesign.