Kolibri: Color Finds the Place, Refinement Sets the Pose for RGB-D Localization in Colored Point Clouds
Organizations: Chair of Cognitive Science, ETH Zürich
Abstract
An agent or robot in a scanned building must locate its camera in rooms that no model saw during training. Point cloud registration methods can align each RGB-D frame with the building's colored point cloud, accurately but at a high cost. We observe that geometric refinement brings nearby initial poses to the same final pose, so the learned component only needs to find the place. Kolibri does this with a graph network of 55,712 parameters that describes 10 cm cubes of points, called cells, by their color, local shape and neighbors. With the same refinement, it matches GeDi on 96 unseen ScanNet rooms (81.4% against 80.3% within 5 cm and 5°) and on a held-out test of 10 further rooms, in 82 ms per frame on an RTX 2080 Ti. The saved time allows for search without a room label. In one point cloud of 100 rooms, Kolibri loses 1.3 points against knowing the room, while retrieving the room first loses 56.0. In a hospital emergency department scene from a real scan, it localizes 86.5% of the frames rendered from the scan in 384 ms each, where GeDi needs 91 minutes per frame on cubes of the scan for a similar recall. The method needs colors that agree between the frame and the point cloud. On gray frames and across cameras with different color responses, point cloud registration methods keep the lead.
Figures & tables
| Set | Frames | Rooms | Point cloud | Role |
| Room given ( Tabs. 2 and 3 ) and rooms joined into one point cloud ( Sec. 4.4 ) | ||||
| ScanNet-10 | 871 | 10 | same recording | dev.: design, ablation rooms |
| ScanNet-10-S2 | 683 | 4 | first session | development |
| 7-Scenes | 1,700 | 7 | other recordings | development, ablation |
| 12-Scenes | 575 | 12 | other recordings | development |
| ScanNet-96 | 7,986 | 96 | same recording | test |
| ScanNet-96 (7,986 frames) | ScanNet-96-S2 (5,133 frames) | |||||||
|---|---|---|---|---|---|---|---|---|
| Fast | Precise | Fast | Precise | |||||
| [0pt][2]Kolibri | 81.4 1.8 | 85.3 1.6 | 65.1 3.4 | 66.5 3.4 | ||||
| [0pt][2] + color pass | 81.6 1.8 | 85.5 1.6 | 65.5 3.4 | 67.0 3.3 | ||||
| BUFFER-X | 69.4 2.8 | ▲ | 72.4 2.8 | ▲ | 61.5 3.7 | ▲ | 62.7 3.6 | ▲ |
| BUFFER-X-Lite | 70.4 2.7 | ▲ | 73.3 2.7 | ▲ | 61.9 3.7 | ▲ | 63.0 3.6 | ▲ |
| GeDi | 80.3 1.9 | ≈▲ | 84.2 1.6 | ≈▲ | 65.5 3.3 | ≈ | 67.1 3.3 | ≈ |
| Test scenes | Per frame | ScanNet-10 871 frames | 7-Scenes 1,700 frames | 12-Scenes 575 frames | ScanNet-10-S2 683 frames | ||||||||||||||||
| Method | Parameters | in training | median ms | p95 ms | GB | 5 cm | 0.5 m | 5 cm | 0.5 m | 5 cm | 0.5 m | 5 cm | 0.5 m | ||||||||
| [0pt][2]Kolibri Fast | 55.7k | none | 81.7 | 251.1 | 1.44 | 75.7 6.5 | 94.9 2.5 | 93.0 2.5 | 98.5 0.9 | 97.9 3.1 | 99.7 0.6 | 73.3 7.4 | 95.3 3.3 | ||||||||
| [0pt][2]Kolibri Precise | 55.7k | none | 116.4 | 417.4 | 1.44 | 79.2 5.9 | 95.1 2.5 | 93.1 2.7 | 98.6 0.9 | 98.6 2.0 | 99.7 0.6 | 75.9 7.3 | 95.4 3.4 | ||||||||
| BUFFER-X [ 39 ] | 0.91M | 7S, 12S | 1,416 | 1,922 | 6.21 | 66.4 6.7 | ▲ | 91.3 3.8 | ▲ | 72.7 7.8 | ▲ | 96.2 2.5 | ▲ | 82.1 7.8 | ▲ | 91.8 5.0 | ▲ | 66.9 8.6 | ▲ | 95.2 2.9 | ∘ |
| + refinement | 1,444 | 1,948 | 6.21 | 74.7 6.9 | ≈ | 92.9 3.6 | ∘ | 92.2 3.6 | ≈ | 97.5 2.3 | ∘ | 91.8 4.7 | ▲ | 93.4 4.4 | ▲ | 73.8 7.3 | ≈ | 97.1 2.0 | ∘ | ||
| BUFFER-X-Lite [ 39 ] | 0.91M | 7S, 12S | 1,280 | 1,896 | 6.21 | 63.9 6.7 | ▲ | 90.7 4.2 | ▲ | 66.6 7.3 | ▲ | 96.4 2.5 | ▲ | 81.6 7.9 | ▲ | 91.7 5.6 | ▲ | 62.1 9.6 | ▲ | 95.0 2.9 | ∘ |
| Fast | Precise | ||||
| Params | ScanNet | 7-Scenes | ScanNet | 7-Scenes | |
| [0pt][2]Full network | 55.7k | 78.5 | 93.0 | 80.1 | 93.1 |
| One change to the network, retrained | |||||
| No message passing | 5.5k | −14.6 | −24.4 | −13.4 | −22.7 |
| [−20.9, −8.5] | [−32.5, −16.7] | [−18.9, −8.4] | [−31.1, −15.2] | ||
| Geometry only | 55.3k | −26.2 | −72.7 | −26.2 | −72.0 |
| One color reading | Color pass | |||||||
|---|---|---|---|---|---|---|---|---|
| Fast | Precise | Fast | Precise | |||||
| Time per frame, median (ms) | 81.7 | 116.4 | 187.8 | 278.0 | ||||
| Time per frame, p95 (ms) | 251.1 | 417.4 | 677.3 | 1,030 | ||||
| Peak GPU memory (GB) | 1.44 | 1.44 | 1.55 | 1.55 | ||||
| Recall at 5 cm, clean frames | ||||||||
| ScanNet-10 | 75.7 | 6.5 | 79.2 | 5.9 | 76.6 | 6.5 | 79.7 | 5.6 |
| Method | Search | Frames | 5 cm | 0.5 m | s/frame |
|---|---|---|---|---|---|
| [0pt][2]Kolibri Fast | whole scan | 440 | 86.5 0.7 | 90.6 0.5 | 0.38 |
| [0pt][2] | 30 | 84.4 | 90.0 | ||
| [0pt][2]Kolibri Precise | whole scan | 440 | 86.4 0.5 | 91.1 0.5 | 0.62 |
| [0pt][2] | 30 | 86.7 | 91.1 | ||
| GeDi | whole scan | 30 | 26.7 | 30.0 | 531 |
| GeDi | 350k keypoints, 15 cm | 30 | 43.3 | 46.7 | 121 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Test scenes | Per frame | ScanNet-10 871 frames | 7-Scenes 1,700 frames | 12-Scenes 575 frames | ScanNet-10-S2 683 frames | ||||||||||||||||
| Method | Parameters | in training | median ms | p95 ms | GB | 5 cm | 0.5 m | 5 cm | 0.5 m | 5 cm | 0.5 m | 5 cm | 0.5 m | ||||||||
| [0pt][2]Kolibri Fast | 55.7k | none | 81.7 | 251.1 | 1.44 | 75.7 6.5 | 94.9 2.5 | 93.0 2.5 | 98.5 0.9 | 97.9 3.1 | 99.7 0.6 | 73.3 7.4 | 95.3 3.3 | ||||||||
| [0pt][2]Kolibri Precise | 55.7k | none | 116.4 | 417.4 | 1.44 | 79.2 5.9 | 95.1 2.5 | 93.1 2.7 | 98.6 0.9 | 98.6 2.0 | 99.7 0.6 | 75.9 7.3 | 95.4 3.4 | ||||||||
| BUFFER-X [ 39 ] | 0.91M | 7S, 12S | 1,416 | 1,922 | 6.21 | 66.4 6.7 | ▲ | 91.3 3.8 | ▲ | 72.7 7.8 | ▲ | 96.2 2.5 | ▲ | 82.1 7.8 | ▲ | 91.8 5.0 | ▲ | 66.9 8.6 | ▲ | 95.2 2.9 | ∘ |
| + refinement | 1,444 | 1,948 | 6.21 | 74.7 6.9 | ≈ | 92.9 3.6 | ∘ | 92.2 3.6 | ≈ | 97.5 2.3 | ∘ | 91.8 4.7 | ▲ | 93.4 4.4 | ▲ | 73.8 7.3 | ≈ | 97.1 2.0 | ∘ | ||
| + second stage | 1,480 | 1,997 | 6.21 | 77.4 6.3 | ▲ | 92.9 3.6 | ▲ | 92.1 4.0 | ∘ | 97.5 2.3 | ∘ | 92.5 4.5 | ▲ | 93.4 4.6 | ▲ | 76.6 6.8 | ≈ | 97.1 2.0 | ∘ | ||
| Sensor frames (679) | Rendered views (2,000) | |||||||||||
| Method | 5 cm | 0.25 m | 0.5 m | 5 cm | 0.25 m | 0.5 m | ||||||
| [0pt][2]Kolibri Fast | 68.7 | 84.9 | 92.9 | 94.6 | 94.7 | 94.9 | ||||||
| [0pt][2]Kolibri Precise | 73.1 | 87.1 | 93.3 | 94.8 | 94.9 | 95.0 | ||||||
| [0pt][2]Kolibri Fast + color pass | 69.1 | 86.2 | 93.9 | 94.5 | 94.7 | 94.8 | ||||||
| [0pt][2]Kolibri Precise + color pass | 73.4 | 88.0 | 94.4 | 94.8 | 94.9 | 94.9 | ||||||
| BUFFER-X [ 39 ] | 58.3 | ▲ | 72.9 | ▲ | 86.6 | ▲ | 86.2 | ▲ | 85.7 | ▲ | 89.2 | ▲ |
| Recall (%) | Median ms | ||||
|---|---|---|---|---|---|
| Own room | All rooms | Loss (points) | Own room | All rooms | |
| Fast | 81.2 | 79.9 | 1.3 [0.8, 1.8] | 65 | 168 |
| Precise | 85.1 | 83.7 | 1.4 [0.9, 1.9] | – | 230 |
| Search | Map keypoints | Inlier dist. | 5 cm | 0.5 m | Time |
|---|---|---|---|---|---|
| GeDi, whole scan | 100,000 | 2 cm ∗ | 10.0 | 10.0 | 0.7 min |
| GeDi, whole scan | 100,000 | 10 cm | 13.3 | 16.7 | 0.6 min |
| GeDi, whole scan | 100,000 | 20 cm | 16.7 | 20.0 | 0.6 min |
| GeDi, whole scan | 350,000 | 2 cm ∗ | 13.3 | 16.7 | 2.0 min |
| GeDi, whole scan | 350,000 | 5 cm | 33.3 | 36.7 | 2.0 min |
| GeDi, whole scan | 350,000 | 10 cm | 40.0 | 43.3 | 2.1 min |
| Region searched | Map points | Frame inside | 5 cm | 0.25 m | 0.5 m |
|---|---|---|---|---|---|
| 3 m cube around the seen points | 24,570 | 75% | 50.0 | 66.7 | 83.3 |
| 6 m cube around the seen points | 103,354 | 99% | 30.0 | 50.0 | 66.7 |
| 4 m square around the camera | 53,413 | 39% | 23.3 | 30.0 | 40.0 |
| 8 m square around the camera | 174,401 | 89% | 30.0 | 36.7 | 53.3 |
| 16 m square around the camera | 628,505 | 100% | 23.3 | 23.3 | 36.7 |
| 32 m square around the camera | 1,537,817 | 100% | 13.3 | 20.0 | 26.7 |
| Method | 5 cm | 0.25 m | 0.5 m | Time |
| [0pt][2]Kolibri Fast | 9.1 | 14.5 | 26.3 | |
| [0pt][2]Kolibri Precise | 10.4 | 14.5 | 27.2 | |
| [0pt][2]Kolibri Fast + color pass | 17.3 | 27.6 | 47.2 | |
| [0pt][2]Kolibri Precise + color pass | 18.3 | 27.5 | 48.6 | |
| GeDi [ 33 ] , whole scan | 36.7 | 50.9 | 77.1 | 2.8 min |
| GeDi, 3 m cubes | 42.2 | 61.3 | 91.5 | 11.0 min |
| Narrower lens | Focal length error | Radial distortion | Color offset | ||||||||||
| Method | Unchanged | 45° | 30° | +5% | +10% | 5% | 10% | +0.1 | +0.2 | 0.1 | 0.2 | 5 px | 10 px |
| Recall at 5 cm and 5° | |||||||||||||
| [0pt][2]Kolibri Fast | 75.6 | 71.8 | 60.9 | 56.4 | 21.7 | 53.9 | 16.5 | 73.1 | 63.9 | 73.0 | 62.5 | 74.9 | 75.4 |
| [0pt][2]Kolibri Precise | 79.1 | 75.3 | 63.9 | 51.2 | 16.3 | 47.5 | 11.7 | 76.9 | 67.4 | 75.0 | 61.8 | 78.6 | 78.5 |
| BUFFER-X [ 39 ] | 74.4 | 69.7 | 52.0 | 56.3 | 21.5 | 53.6 | 16.2 | 72.0 | 62.9 | 71.4 | 61.7 | 74.4 | 74.4 |
| GeDi [ 33 ] | 76.2 | 71.6 | 54.9 | 56.8 | 21.9 | 53.6 | 16.5 | 73.2 | 63.8 | 73.4 | 62.3 | 76.2 | 76.2 |
Robot Localization
Robot localization estimates a robot’s position and orientation from onboard sensors, maps, and observations of its surroundings.
See all 188 papers