Correspondence-Free Camera Pose Estimation by Matching Moments
Organizations: School of Computing, The Australian National University, Canberra
Abstract
The blind perspective--point (PnP) problem is to estimate the pose of a calibrated camera from 2D and 3D point sets with unknown correspondences. We show that, for complete, noiseless observations, fourteen permutation-invariant image moments up to order four identify a fixed camera pose for almost every 3D point set containing at least twenty points. Our learning-free method, MomentPnP, matches these moments between the observed 2D points and the projected 3D points to initialize a camera pose without correspondences and refines the pose and correspondence weights using expectation--maximization. Across the test sets of three benchmarks, our method reduces median rotation error by -- relative to the best learned baseline, both with and without outliers. We recover accurate poses even with as many outliers as inliers. On dense ETH3D scans, pose accuracy improves as the point sets grow, even as individual correspondences become harder to resolve.
Figures & tables
| MegaDepth [ 46 ] | ModelNet40 [ 79 ] | Data61 [ 56 ] | |||||||||||
| no outliers | with outliers | no outliers | with outliers | no outliers | with outliers | ||||||||
| Method | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Time (s) |
| Learning-free baselines | |||||||||||||
| P3P-RANSAC [ 18 ] | 131 / 8,482 | 0.0 | 127 / 7,091 | 0.0 | 138 / 161 | 0.0 | 124 / 123 | 0.0 | 132 / 5,536 | 0.0 | 137 / 4,734 | 0.0 | 30 |
| SoftPOSIT [ 11 ] | 0.063 / 0.31 | 86.0 | 9.47 / 36 | 34.0 | 0.27 / 0.26 | 80.0 | 30 / 14 | 0.0 | 22 / 67 | 41.0 | 7.63 / 29 | 37.0 | 1.0–30 |
| GOSMA [ 6 ] | 135 / 342 | 0.0 | 78 / 297 | 0.0 | 49 / 75 | 0.0 | 43 / 56 | 0.0 | 121 / 528 | 0.0 | 128 / 494 | 0.0 | 0.25–2.9 |
| No outliers | With outliers | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | C | P | S | E | All | C | P | S | E | All |
| Deep Blind PnP | 33 | 92 | 69 | 20 | 48 | 4 | 69 | 50 | 13 | 33 |
| BPnPNet | 0 | 19 | 36 | 0 | 12 | 0 | 0 | 3 | 0 | 1 |
| GoMatch | 15 | 78 | 17 | 1 | 23 | 8 | 62 | 6 | 3 | 17 |
| CFPE (reproduced) | 81 | 92 | 100 | 100 | 95 | 0 | 0 | 0 | 0 | 0 |
| MomentPnP (ours) | 100 | 100 | 100 | 100 | 100 | 100 | 96 | 100 | 87 | 95 |
| Method | Finished | Best ( ∘ ) | Median ( ∘ ) | Time (s) | |
|---|---|---|---|---|---|
| GOPAC zero | 12/30 | 0/30 | 5.4 | 122 | 2,562 |
| GOPAC annealed | 12/30 | 0/30 | 4.2 | 132 | 3,015 |
| GOSMA | 30/30 | 0/30 | 1.7 | 156 | 68.8 |
| MomentPnP (ours) | 30/30 | 30/30 | 0.0006 | 0.083 | 1.92 |
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| Inputs, pose & correspondences ( Sec. 3 ) | |
| Numbers of observations, of model points and of correspondences, so that is the number of inliers in each set. | |
| 2D point set of observations in pixels, and 3D point set, or 3D model, of model points in model coordinates. | |
| Known intrinsics, projection to pixels, and projection to calibrated image coordinates. | |
| Pose with rotation and translation , true pose and estimated pose. | |
| Model point in camera coordinates, its projection , and the camera center in model coordinates. | |
| Orientation pairs | Rank | Bound | |
|---|---|---|---|
| , | |||
| , | |||
| , | |||
| , rotation by or | |||
| , rotation by | |||
| , any other rotation |
| Stage | Settings |
|---|---|
| Inputs | Outlier-free: . With outliers: , and EM needs at least three model points in front of the camera with finite image projections. |
| Outlier-free search | We use rotations, refined to in the second stage, and keep at most or initial poses separated by . Moment matching uses 30 iterations, keeping at most 96 or 192 fits separated by . |
| Fourier moment matching | rotations scored with and , keeping 48 initial poses separated by . Five fitting stages use , , , and , and fit the best 48, 20, 20, 8 and 8 poses for 20, 12, 12, 10 and 10 iterations. Padding and taper width . The image average uses midpoints of , and uses an midpoint lattice of . The second stage starts from the outlier fractions 0, 0.15, 0.30, 0.45 and 0.60. |
| Other poses with outliers | rotations. Translations come from centroids and spreads corrected for observed outlier fractions 0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4 and 0.5 and model outlier fractions 0, 0.2 and 0.4, keeping the best score per rotation, or from cross-correlation at seven scales from 0.5 to 2 times the ratio of the spreads. Each translation estimate keeps 2,048 initial poses separated by , and after 16 GPU EM iterations the 256 of highest likelihood continue. |
| GPU EM | At most 120 iterations. Without outliers, initial noise and ceiling . With outliers, initial noise and ceiling . The batch stops when its fit of highest likelihood has converged, counting with outliers only fits with , and the other fits stop early. Fits whose estimated noise reaches times the ceiling have not found the observations and do not go to CPU refinement. |
| CPU refinement | We refine at most three candidates per stage, selected by the ranking rules of Step 3. Without outliers, fits stopped early receive at most 60 CPU EM iterations, and converged fits or fits at the iteration limit go to the acceptance criteria without further EM. With outliers, we first run 30 CPU EM iterations. If the fit has not converged and its association level is at most , we run at most 60 further iterations. |
| Set | Method | 0 | |||
|---|---|---|---|---|---|
| MegaDepth [ 46 ] | Deep Blind PnP | 240 | 119 | 149 | 133 |
| BPnPNet | 223 | 180 | 180 | 174 | |
| GoMatch | 21 | 28 | 29 | 39 | |
| MomentPnP (ours) | 31 | 206 | 259 | 394 | |
| ModelNet40 [ 79 ] | Deep Blind PnP | 330 | 27 | 34 | 46 |
| BPnPNet | 159 | 95 | 97 | 100 |
| MegaDepth [ 46 ] | ModelNet40 [ 79 ] | Data61 [ 56 ] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | 0 | 0 | 0 | |||||||||
| Rotation error below (%) | ||||||||||||
| BPnPNet [ 5 ] | ||||||||||||
| RANSAC | 50.1 | 13.0 | 6.3 | 1.0 | 5.4 | 0.0 | 0.0 | 0.0 | 51.1 | 5.7 | 1.0 | 0.3 |
| declarative | 19.2 | 0.0 | 0.0 | 0.3 | 1.9 | 0.3 | 0.0 | 0.0 | 9.6 | 1.3 | 1.7 | 0.3 |
| RANSAC+LM | 53.4 | 16.0 | 7.0 | 1.7 | 6.0 | 0.0 | 0.0 | 0.0 | 55.4 | 6.0 | 1.0 | 0.7 |
| Rotation error ( ∘ ) | Translation error (%) | Both below | ||
|---|---|---|---|---|
| Set | Method | Q1 / median / Q3 | Q1 / median / Q3 | and 1% (%) |
| MegaDepth [ 46 ] | MomentPnP (ours) | 0.000 / 0.009 / 0.031 | 0.005 / 0.013 / 0.035 | 99.0 |
| Deep Blind PnP (posest top- ) | 0.018 / 0.045 / 0.096 | 0.065 / 0.155 / 0.398 | 89.4 | |
| BPnPNet (RANSAC+LM) | 0.277 / 0.853 / 5.530 | 1.571 / 5.402 / 26.087 | 16.9 | |
| GoMatch (released) | 0.059 / 0.136 / 0.348 | 0.228 / 0.590 / 1.880 | 62.0 | |
| ModelNet40 [ 79 ] | MomentPnP (ours) | 0.101 / 0.155 / 0.250 | 0.046 / 0.096 / 0.171 | 96.5 |
| Frames below (%) | Acceptance (frames) | Time (ms) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Split | Outliers | Frames | accepted | accepted, | declined, | no pose | median | mean | ||
| MegaDepth [ 46 ] , test | 0 | 10,795 | 99.9 | 100.0 | 10,788 | 0 | 7 | 0 | 31 | 58 |
| 10,795 | 96.8 | 97.4 | 9,559 | 1 | 921 | 7 | 219 | 597 | ||
| ModelNet40 [ 79 ] , test | 0 | 2,466 | 97.0 | 98.7 | 2,366 | 13 | 60 | 0 | 55 | 256 |
| Data61 [ 56 ] , test | 0 | 2,640 | 100.0 | 100.0 | 2,640 | 0 | 0 | 0 | 48 | 52 |
| 2,640 | 98.9 | 99.1 | 2,601 | 0 | 11 | 1 | 261 | 391 | ||
| Dataset | Direct EM | Degree 2 | Degree 3 | Degree 4 | |
|---|---|---|---|---|---|
| MegaDepth | 100 | 0 / 68 | 0 / 57 | 53 / 70 | 59 / 70 |
| ModelNet40 | 100 | 0 / 39 | 2 / 26 | 56 / 78 | 54 / 66 |
| Data61 | 100 | 0 / 64 | 1 / 48 | 69 / 75 | 78 / 83 |
| Synthetic | 30 | 0 / 12 | 0 / 15 | 27 / 27 | 24 / 25 |
| Method | Initial bound | Rot. | Rot. | Optimality flag | Queue limit | Stopped | Median time (s) | Largest time (s) |
|---|---|---|---|---|---|---|---|---|
| GOPAC | Zero | 0/30 | 0/30 | 12/30 | 0/30 | 18/30 | 2,562 | 4,088 |
| GOPAC | Annealed | 0/30 | 1/30 | 12/30 | 0/30 | 18/30 | 3,015 | 4,770 |
| GOSMA | — | 0/30 | 1/30 | 0/30 | 30/30 | 0/30 | 68.8 | 293 |
| MomentPnP (ours) | — | 30/30 | 30/30 | — | — | 0/30 | 1.92 | 2.58 |
| Method | Rot. ( ∘ ) | Time (s) | Stop | ||
|---|---|---|---|---|---|
| 8 | GOPAC zero | 1.838 | 1/3 | 18.749 | 0/3 |
| 8 | GOPAC annealed | 2.469 | 1/3 | 16.263 | 0/3 |
| 8 | GOSMA | 165.053 | 0/3 | 30.471 | 3/3 |
| 8 | MomentPnP (ours) | 3/3 | 0.100 | 0/3 | |
| 16 | GOPAC zero | 1.866 | 0/3 | 2.589 | 0/3 |
| 16 | GOPAC annealed | 1.866 | 0/3 | 1.234 | 0/3 |
| Deep Blind PnP | BPnPNet | GoMatch | CFPE (reproduced) | MomentPnP (ours) | |||||||||||||
| Dataset | Images/scenes | Error | Time | Error | Time | Error | Time | Error | Time | Error | Time | ||||||
| Without outliers | |||||||||||||||||
| Cambridge Landmarks [ 40 ] | 27/4 | 326–2,120 | 33 | 88.80 | 351 | 0 | 122.44 | 147 | 15 | 103.63 | 37 | 81 | 0.539 | 585 | 100 | 0.125 | 85 |
| Phototourism [ 69 , 37 ] | 36/3 | 469–2,607 | 92 | 0.07 | 377 | 19 | 6.80 | 135 | 78 | 0.23 | 58 | 92 | 0.200 | 941 | 100 | 0.006 | 67 |
| 7-Scenes [ 68 ] | 36/3 | 128–582 | 69 | 0.53 | 303 | 36 | 21.41 | 103 | 17 | 7.36 | 29 | 100 | 0.360 | 1,029 | 100 | 0.134 | 25 |
| ETH3D [ 65 ] | 70/6 | 202–5,010 | 20 | 94.48 | 355 | 0 | 108.32 | 111 | 1 | 113.42 | 72 | 100 | 0.015 | 956 | 100 | 0.004 | 66 |
| Within ( ) | Recorded failures ( ) | Time per image ( ) | |||||
|---|---|---|---|---|---|---|---|
| Method | Images | % | Returned/attempted | OOM | Other exceptions | Median (ms) | Maximum (s) |
| Deep Blind PnP | 49 | 29.0 | 169/169 | 0 | 0 | 366 | 0.60 |
| BPnPNet | 1 | 0.6 | 169/169 | 0 | 0 | 108 | 0.22 |
| GoMatch | 26 | 15.4 | 160/169 | 9 | 0 | 55 | 0.60 |
| CFPE (reproduced) | 0 | 0.0 | 169/169 | 0 | 0 | 1,976 | 11.27 |
| MomentPnP (ours) | 147 | 87.0 | 167/169 | 0 | 2 | 547 | 57.15 |
| Facade | Courtyard | Office | |||||||
| Points | Time | Error | Labels | Time | Error | Labels | Time | Error | Labels |
| Every model point has a 2D point | |||||||||
| 1,000 | 0.09 | 1.5 | 100 | 0.14 | 1.9 | 100 | 0.14 | 2.1 | 100 |
| 4,000 | 0.28 | 0.62 | 100 | 0.46 | 1.2 | 100 | 0.39 | 1.2 | 100 |
| 16,000 | 0.97 | 0.30 | 99 | 2.4 | 0.43 | 100 | 1.9 | 0.81 | 99 |
| 64,000 | 6.6 | 0.21 | 98 | 22.7 | 0.22 | 98 | 18.6 | 0.21 | 98 |
| View | Rotation ( ∘ ) | Rel. translation | Labels (%) |
|---|---|---|---|
| Facade 1 | 78.0767 | ||
| Courtyard 3 | 77.4822 | ||
| Office 1 | 77.6801 |
| Rotation ( deg) | Relative translation ( ) | 1M points | ||||||
|---|---|---|---|---|---|---|---|---|
| View | 64k | 256k | 1M | 64k | 256k | 1M | Solve (s) | Labels (%) |
| Courtyard 1 | 2.2351 | 2.1025 | Invalid | 2.3367 | 2.9767 | – | – | – |
| Courtyard 2 | 1.2694 | 1.0114 | Timeout | 1.3939 | 1.1487 | – | – | – |
| Courtyard 3 | 2.8570 | 1.2198 | 0.2497 | 2.4586 | 1.7281 | 0.4510 | 688.7 | 77.48 |
| Office 1 | 2.0979 | 2.4092 | 0.4824 | 8.0690 | 5.6997 | 0.3121 | 985.1 | 77.68 |
| Office 2 | 2.6994 | 0.9101 | 1.1428 | 7.5577 | 3.8684 | 3.7290 | 1243.7 | 76.32 |
| Recovered poses | ||||
|---|---|---|---|---|
| Points | Deep Blind PnP | BPnPNet | GoMatch | MomentPnP (ours) |
| 1,000 | 2/9 | 0/9 | 0/9 | 9/9 |
| 4,000 | 0/9 | 0/9 | 0/9 | 9/9 |
| 16,000 | 0/9 | 0/9 | OOM | 9/9 |
| 64,000 | OOM | OOM | OOM | 9/9 |
| 256,000 | OOM | OOM | OOM | 9/9 |
| Recovery | MomentPnP (ours) | ||||||
|---|---|---|---|---|---|---|---|
| Scene | Points | Ratio | DBP | BPnP | GoM | Tuning | Test |
| Kicker | 15,897 | 2.7 | 0/3 | 0/3 | OOM | 3/3 | 3/3 |
| Office | 3,461 | 3.4 | 2/3 | 0/3 | 0/3 | 3/3 | 3/3 |
| Facade | 85,096 | 4.8 | OOM | 0/3 | OOM | 3/3 | 3/3 |
| Terrace | 10,676 | 5.3 | 0/3 | 0/3 | OOM | 3/3 | 3/3 |
| Pipes | 2,473 | 5.4 | 0/3 | 0/3 | 0/3 | 3/3 | 3/3 |
| Affine initialization (ours) | Full MomentPnP (ours) | |||||
|---|---|---|---|---|---|---|
| Depth condition | Error ( ∘ ) | Time (ms) | Error ( ∘ ) | Time (ms) | ||
| Spread | 1.04 | 20.88 | 51 | 8/10 | 0.374 | 108 |
| Spread | 1.22 | 15.43 | 51 | 10/10 | 0.227 | 109 |
| Spread | 1.85 | 12.88 | 51 | 10/10 | 0.089 | 108 |
| Spread | 3.95 | 156.55 | 50 | 10/10 | 0.062 | 110 |
| Spread | 18.04 | 161.26 | 51 | 10/10 | 0.051 | 109 |
| MegaDepth | ModelNet40 | Data61 | |
| GPU search (ours) (%) | 99.9 | 97.0 | 100.0 |
| CPU search only (ours) (%) | 99.8 | 96.8 | 99.9 |
| CPU/GPU runtime ratio | 4.7 | 6.4 | 5.0 |
| Ambiguous pose examined (frames) | 154 | 715 | 0 |
| replaces the pose, GPU search | 0 | 2 | 0 |
| replaces the pose, CPU search | 15 | 47 | 0 |
| Random selection | Directional deletion | |||
| 24 | 29 | 5 | 1 | 0 |
| 32 | 27 | 4 | 0 | 0 |
| 48 | 26 | 5 | 0 | 0 |
| 96 | 0 | 0 | 0 | 0 |
| Observations | Thresholds | Raw | Symmetry |
|---|---|---|---|
| All SIFT detections | , 1% extent | 2/6 | — |
| RGB/CAD edges | , 5 cm | 6/20 | 8/20 |
| New objects | , 5 cm | 3/16 | 5/16 |
| Motion-capture centroids | , 5 cm | 6/8 | — |
| Infrared LEDs (relative) | , 5 cm | 19/31 | — |
| Query | 2D points | Locations | Model points | Rot. ( ∘ ) | Center (%) | Time (s) | Recovered | Accepted |
|---|---|---|---|---|---|---|---|---|
| Office 9 | 335 | 287 | 3,272 | 70.58 | 54.546 | 3600.1 | no | no |
| Office 18 | 1,327 | 1,134 | 3,272 | 0.11 | 0.054 | 3604.2 | yes | no |
| Courtyard 13 | 13,123 | 11,525 | 42,714 | 103.08 | 15.811 | 3618.6 | no | no |
| Courtyard 26 | 11,079 | 9,399 | 42,714 | 93.82 | 7.460 | 3600.2 | no | no |
| Facade 26 | 9,432 | 8,205 | 96,909 | 0.0027 | 0.001 | 870.6 | yes | yes |
| Facade 51 | 12,525 | 10,517 | 96,909 | 177.34 | 21.350 | 3600.2 | no | no |
| Raw pose | Model symmetry | ||||
| Object | View | Rot. | Center | Rot. | Center |
| 1 | 72 | 159.94 | 1169.4 | 159.88 | 1169.7 |
| 1 | 360 | 17.81 | 188.4 | 14.70 | 167.0 |
| 5 | 72 | 128.37 | 1141.4 | 128.37 | 1141.4 |
| 5 | 360 | 1.52 | 17.4 | 1.52 | 17.4 |
| 9 | 72 | 179.65 | 353.4 | 3.33 | 35.8 |
| Motion capture | Infrared LEDs | |
|---|---|---|
| Accurate comparisons | 6/8 | 19/31 |
| Accepted pose outputs | 5/8 | 25/32 |
| Inaccurate accepted comparisons | 1/8 | 5/31 |
| Extraction failures | 0/8 | 7/32 |
| Median rotation error ( ∘ ) | 0.324 | 1.063 |
| Median position error (mm) | 20.9 | 25.9 |
Camera Pose Estimation
Camera pose estimation recovers a camera’s position and orientation from images, visual features, scene geometry, and sometimes sensor measurements.
See all 267 papers