Learning Positive-Incentive Point Sampling in Neural Implicit Fields for Object Pose Estimation
Authors: Yifei Shi, Boyan Wan, Xin Xu, Kai Xu
Organizations: College of Intelligence Science and Technology, National University of Defense Technology, China · College of Computer Science, National University of Defense Technology, China
Learning neural implicit fields of 3D shapes is a rapidly emerging field that enables shape representation at arbitrary resolutions. Due to the flexibility, neural implicit fields have succeeded in many research areas, including shape reconstruction, novel view image synthesis, and more recently, object pose estimation. Neural implicit fields enable learning dense correspondences between the camera space and the object's canonical space-including unobserved regions in camera space-significantly boosting object pose estimation performance in challenging scenarios like highly occluded objects and novel shapes. Despite progress, predicting canonical coordinates for unobserved camera-space regions remains challenging due to the lack of direct observational signals. This necessitates heavy reliance on the model's generalization ability, resulting in high uncertainty. Consequently, densely sampling points across the entire camera space may yield inaccurate estimations that hinder the learning process and compromise performance. To alleviate this problem, we propose a method combining an SO(3)-equivariant convolutional implicit network and a positive-incentive point sampling (PIPS) strategy. The SO(3)-equivariant convolutional implicit network estimates point-level attributes with SO(3)-equivariance at arbitrary query locations, demonstrating superior performance compared to most existing baselines. The PIPS strategy dynamically determines sampling locations based on the input, thereby boosting the network's accuracy and training efficiency. Our method outperforms the state-of-the-art on three pose estimation datasets. Notably, it demonstrates significant improvements in challenging scenarios, such as objects captured with unseen pose, high occlusion, novel geometry, and severe noise.
Figures & tables
Fig. 1: We propose PIPS , a data-driven approach to dynamically determine where to sample to boost the network training, achieving better performance with training on fewer sampling points, compared to (a) the random sampling baseline. PIPS consists of two components: (b) positive-incentive point sampling with high estimation certainty ( PIPS-C ) and (c) positive-incentive point sampling with high geometric stability ( PIPS-S ).
Fig. 2: (a) Point-wise canonical coordinate estimation on as few as three points (in red) is sufficient for determining all the 6-DoFs of object pose. (b) Extra voters with inaccurate point-level estimations (in blue) would degrade the performance.
Fig. 3: The quantitative comparisons of the proposed PIPS-C and PIPS-S to the baseline of random sampling. We see our method reduce the number of sample points and the training time while achieving better performance in object pose estimation. The experiment is conducted on the NOCS-REAL275 dataset.
Fig. 4: Overview of the proposed method. First, an SO(3)-equivariant convolutional implicit network with dense point sampling (the teacher model) is optimized to generate the pseudo ground-truth. Second, the PIPS-C and PIPS-S estimation networks (the student model) are trained based on the generated pseudo ground-truth. Third, an SO(3)-equivariant convolutional implicit network is trained with the sample points estimated by the PIPS estimation network.
Fig. 5: (a) By rotating the 3D graph convolution kernel via a regular icosahedron rotation group, we generate a set of convolutional kernels. (b) Point cloud convolutions with the rotated convolutional kernels make the generated features SO(3)-invariant. By generating vector neurons by the SO(3)-invariant point feature and multiplying the vector neurons with the rotation matrix Ra corresponding to the qa∈Q with the highest activation, the feature becomes SO(3)-equivariant.
Fig. 6: Architecture of the SO(3)-equivariant convolutional implicit network. Left: Multiple SO(3)-equivariant 3D graph convolution layers and point downsampling layers are adopted to aggregate multiple-scale point features. The RGB features are extracted by DINOv2. Right: The aggregate features are propagated to any 3D sample point via feature propagation layers, estimating the point-wise object canonical coordinate.
Fig. 7: The network architecture of the PIPS estimation network. The network takes a point cloud as input, embeds it into a high-dimensional feature space with an encoder, and generates the PIPS-C sample points with a decoder. A gating unit and a Gumbel-Softmax module are applied to generate the PIPS-S sample points. Multiple loss functions are adopted to train the network.
Fig. 8: A 2D example of PIPS-S selection. The dots are the input to the PIPS-S estimation component. The dots with an ellipse are the selected PIPS-S sample points, where the ellipse represents its anisotropic variances. (a) The pointset is unstable w.r.t. the alignment because it has a high variance along the y-axis direction, meaning that the locations along the y-axis are not well-constrained. (b) The pointset is stable w.r.t. the alignment since no DoF shows high variance for the points within the set, indicating the locations are well-constrained.
Fig. 9: The statistical comparisons between the training set and the test set of ShapeNet-C.
Dataset
Avg. instances per category
Avg. Chamfer distance to mean shape (mm)
Avg. occlusion (%)
NOCS-REAL275 [ 67 ]
4
2.7
46
NOCS-CAMERA25 [ 67 ]
31
2.4
44
Wild6D [ 77 ]
360
2.2
42
HouseCat6D [ 25 ]
19
2.8
49
ShapeNet-C
2000
3.1
53
TABLE I: The comparisons of ShapeNet-C to existing datasets.
Fig. 10: The qualitative comparisons on the NOCS-REAL275 dataset. Our method could accurately estimate the pose on all the tested objects, while the baselines cannot.
Methods
Input
Data source
Shape Prior
IoU75 ∗ ↑
5∘ 2cm ↑
5∘ 5cm ↑
10∘ 2cm ↑
10∘ 5cm ↑
SGPA [ 8 ]
RGB-D
Syn.+Real
✓
0.37
0.36
0.40
0.61
0.71
DPDN [ 35 ]
RGB-D
Syn.+Real
✓
0.54
0.46
0.51
0.70
0.78
HS-Pose [ 79 ]
D
Real
×
0.39
0.45
0.55
0.69
0.84
GenPose [ 78 ]
D
Real
×
0.50
0.52
0.61
0.72
0.84
VI-Net [ 36 ]
RGB-D
Real
×
0.48
0.50
0.58
0.71
0.82
SecondPose [ 9 ]
RGB-D
Syn.+Real
×
0.50
0.56
0.64
0.75
0.86
TABLE II: Quantitative comparisons on the NOCS-REAL275 dataset. ‘*’ denotes the IoU metrics computed as in [ 40 ] .
Methods
Input
Data source
Rotation
Translation
Mean( ∘ ) ↓
Median( ∘ ) ↓
5∘↑
Mean(cm) ↓
Median(cm) ↓
5∘5 cm ↑
NOCS [ 67 ]
RGB
Syn.
53.15
27.68
0.34
6.87
5.03
0.29
GPV-Pose [ 12 ]
D
Syn.
50.95
20.62
0.38
6.31
4.82
0.33
GenPose [ 78 ]
D
Syn.
48.29
11.81
0.41
5.76
4.29
0.37
VI-Net [ 36 ]
RGB-D
Syn.
46.45
9.81
0.47
5.55
3.89
0.42
SecondPose [ 9 ]
RGB-D
Syn.
45.77
7.87
0.49
5.42
3.91
0.45
TABLE III: Quantitative comparisons on the ShapeNet-C dataset.
Fig. 11: The qualitative comparisons on the ShapeNet-C dataset. The grey points denote the input points. The colored points represent the canonical shapes transformed with the estimated poses.
Methods
Data type
Refinement
AR
time
CDPNv2 [ 34 ]
RGB
×
62.4
0.98
NCF [ 18 ]
RGB
×
63.2
7.17
ZebraPose [ 59 ]
RGB
×
72.1
0.25
CosyPose [ 30 ]
RGB-D
✓
71.4
13.74
GDRNPP [ 41 ]
RGB-D
×
71.3
0.28
GDRNPP [ 41 ] (GPose)
RGB-D
✓
80.5
4.58
TABLE IV: Quantitative results on the LineMOD-O dataset. The best results are in bold, and the second-best results are underlined.
Fig. 12: The qualitative comparisons on the LineMOD-O dataset. The boundary of the shape transformed by the estimated pose is shown in the figure. Please pay attention to the highlighted objects.
Fig. 13: Visualization of the PIPS sample points. The results show that: 1) PIPS-C sample points are dense, distributed around the object surface, possibly around the unseen surface region, such as the occluded chair in (b); 2) instead of uniformly distributing around the object surface, the sampling rule varies on different shapes, such as the laptop in (a); 3) PIPS-S sample points are sparse, covering the crucial regions of the input shape; 4) adding noise to the input points will not substantially change the distribution of PIPS-S sample points, such as the two cases in last row of (b), where the sampled points are speared near the wings and tail of the airplane despite the noise. The results are produced with only the point cloud as input, without the RGB image features.
Methods
Median( ∘ ) ↓
Median(cm) ↓
5∘ 5cm ↑
w/o PIPS
5.92
2.78
0.51
w/o PIPS-C
5.34
2.61
0.53
w/o PIPS-S
4.23
2.49
0.57
w/o anisotropic variance
4.51
2.89
0.55
w/o SO(3)-equivariance
5.18
3.02
0.49
w/o RGB
3.48
2.67
0.59
TABLE V: Ablation study of our method on ShapeNet-C.
Fig. 14: Robustness evaluation. The quantitative comparisons of our method to a baseline method on the four challenging subsets of the NOCS-CAMERA25 dataset. Our method is more robust on unseen poses, novel shapes, high occlusion, and severe noise.
Voxel size
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
4
5.42
3.15
0.56
0.49
8
3.16
2.13
0.68
0.64
16
3.28
2.08
0.68
0.62
32
3.71
2.34
0.63
0.58
TABLE VI: Effect of voxel size h in PIPS-C estimation.
Target sparsity
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
0.3
3.42
2.65
0.63
0.56
0.2
3.28
2.38
0.68
0.61
0.1
3.16
2.13
0.66
0.62
0.05
3.71
2.84
0.59
0.52
TABLE VII: Effect of target sparsity ρ in PIPS-S estimation.
Threshold
Median( ∘ ) ↓
Median(cm) ↓
5∘↑
5∘ 5cm ↑
0.9
4.32
3.24
0.57
0.49
0.7
3.89
2.95
0.61
0.54
0.5
3.16
2.13
0.66
0.62
0.3
3.58
2.75
0.63
0.56
TABLE VIII: Effect of threshold ω in positive sample selection of the PIPS estimation network.
Fig. 15: (a) The training plot of the generalizing to shape reconstruction experiment. (b) Examples of the reconstructed shapes during the network training.