ReFineR: Single image to dense point cloud registration at pixel level with point splatting
Organizations: Mines Paris, PSL University Centre for robotics (CAOR) 75006 Paris, France · ChapsVision 4 Rue du Port aux Vins, 92150 Suresnes, France
Abstract
Cameras and LiDARs (Light Detection and Ranging) provide complementary observations of the same scene: cameras capture rich appearance cues while LiDAR sensors provide accurate geometric measurements. In this work, we address the problem of fine image-to-point cloud registration from a single image and a roughly estimated camera pose. On robotic platforms, residual reprojection errors of several pixels are commonly observed even after calibration due to temporal desynchronization, mechanical vibrations, or inaccuracies in point cloud reconstruction. Such errors can lead to decimeter-level geometric discrepancies at moderate distances and significantly affect multiple downstream tasks, such as dense cross-modal supervision, texture mapping, and semantic label transfer. To address this challenge, we introduce ReFineR: an offline framework to finely register individual images to pre-reconstructed point clouds. ReFineR builds upon differentiable point splatting rasterization and a smooth Gradient Orientation Measure (GOM) to iteratively refine the camera pose through analysis-by-synthesis optimization. We evaluated the method on a curated dataset of manually calibrated images collected from heterogeneous data sources. Furthermore, we show that an automatic point cloud rescale factor can be estimated prior to optimization, enabling robust refinement across a wide range of scenarios using a single scale-invariant formulation. The project page can be found at https://refiner-project.github.io/
Figures & tables
| ETH3D | Tanks and Temples | CARLA | Livox | nuScenes | RGBDScenes | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | ||||||||||||
| 1 | 2 | 5 | 2 | 3 | 5 | 1 | 2 | 5 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10 | |||||||
| One stage | |||||||||||||||||||||||
| GS-SLAM [ Matsuki et al.(2024)Matsuki, Murai, Kelly, and Davison ] | 0 | 0 | 0 | 123.23 64.59 | 0 | 0 | 0 | 264.12 181.23 | 0 | 0 | 0 | 371.52 214.75 | 0 | 0 | 113.85 34.13 | 0 | 0 | 0 | 309.77 110.45 | 0 | 0 | 0 | 90.32 32.97 |
| Yuan et al [ Yuan et al.(2021)Yuan, Liu, Hong, and Zhang ] | 0 | 2 | 20 | 10.29 9.74 | 1 | 17 | 52 | 4.39 7.71 | 0 | 2 | 4 | 15.20 41.70 | 25 | 100 | 5.83 1.21 | 0 | 0 | 5 | 21.10 37.53 | 0 | 0 | 5 | 26.95 26.01 |
| Koide et al [ Koide et al.(2023)Koide, Oishi, Yokozuka, and Banno ] | 0 | 3 | 25 | 11.10 30.86 | 0 | 0 | 4 | 29.00 67.16 | 2 | 8 | 28 | 8.82 10.73 | 25 | 75 | 6.61 2.60 | 0 | 0 | 40 | 12.03 12.21 | 0 | 0 | 5 | 36.19 25.14 |
| ETH3D | Tanks and Temples | Livox | nuScenes | RGBDScenes | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | ||||||||||
| 1 | 2 | 5 | 2 | 3 | 5 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10 | ||||||
| One stage | |||||||||||||||||||
| GS-SLAM [ Matsuki et al.(2024)Matsuki, Murai, Kelly, and Davison ] | 19 | 40 | 47 | 7.54 39.00 | 45 | 62 | 67 | 2.06 59.53 | 20 | 25 | 56.41 32.79 | 0 | 8 | 8 | 131.07 94.68 | 1 | 5 | 11 | 37.54 29.24 |
| Koide et al [ Koide et al.(2023)Koide, Oishi, Yokozuka, and Banno ] | 57 | 98 | 100 | 0.93 0.29 | 51 | 95 | 100 | 1.99 0.50 | 75 | 100 | 4.66 0.28 | 0 | 10 | 40 | 11.14 7.43 | 25 | 45 | 85 | 5.32 22.24 |
| Taylor et al [ Taylor et al.(2015)Taylor, Nieto, and Johnson ] | 1 | 1 | 29 | 6.76 5.05 | 3 | 7 | 23 | 7.25 3.48 | 0 | 75 | 9.09 6.69 | 0 | 0 | 15 | 22.44 46.66 | 0 | 5 | 55 | 9.07 6.65 |
| ETH3D | Tanks and Temples | CARLA | Livox | nuScenes | RGBDScenes | |||||||||||||||||||
| pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | |||||||||||||
| 1 | 2 | 5 | 1 | 2 | 3 | 5 | 1 | 2 | 5 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10 | |||||||
| (I) | ||||||||||||||||||||||||
| W/o rescale | 57 | 97 | 100 | 0.90 0.38 | 0 | 70 | 95 | 100 | 1.75 0.48 | 17 | 61 | 88 | 1.66 2.97 | 50 | 100 | 5.10 0.96 | 9 | 50 | 95 | 5.00 3.59 | 10 | 44 | 93 | 5.43 3.35 |
| MatchAnything* + ReFineR (W/ rescale) | 58 | 96 | 100 | 0.90 0.40 | 0 | 70 | 95 | 100 | 1.76 0.51 | 16 | 68 | 91 | 1.54 3.04 | 55 | 100 | 4.94 0.94 | 6 | 54 | 97 | 4.90 2.75 | 13 | 43 | 92 | 5.36 4.25 |
| (II) | ||||||||||||||||||||||||
| CARLA | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | rot (° ) | transl (cm ) | ||||||||
| 1 | 2 | 3 | 5 | 10 | 20 | 50 | 100 | ||||
| 0px initialization | |||||||||||
| MatchAnything | 3 | 26 | 44 | 85 | 98 | 100 | 100 | 100 | 3.20 2.60 | 0.14 0.08 | 33.94 27.77 |
| ReFineR | 20 | 76 | 92 | 100 | 100 | 100 | 100 | 100 | 1.47 0.79 | 0.06 0.04 | 2.99 1.74 |
| MatchAnything* + ReFineR | 16 | 71 | 83 | 95 | 96 | 99 | 100 | 100 | 1.49 2.72 | 0.05 0.09 | 3.03 6.32 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Sensor / Parameter | value |
| Camera | |
| image_size_x | 1920 |
| image_size_y | 1080 |
| fov | 90 |
| LiDAR | |
| points_per_second | 500,000 |
| Set / Scene / Image | — Official registration (px) | — Manual registration (px) |
|---|---|---|
| Tanks and Temples | ||
| Church | ||
| 21.png | 5.13 | 1.37 |
| 40.png | 4.35 | 1.89 |
| 83.png | 7.27 | 1.99 |
| 110.png | 4.4 | 1.28 |
| ETH3D | Tanks and Temples | CARLA | Livox | nuScenes | RGBDScenes | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | |
| One stage | ||||||||||||
| GS-SLAM [ Matsuki et al.(2024)Matsuki, Murai, Kelly, and Davison ] | 8.19 2.30 | 57.08 37.98 | 8.05 1.32 | 101.69 69.42 | 9.00 1.23 | 333.86 101.20 | 7.58 1.15 | 71.95 38.52 | 6.92 2.08 | 180.34 37.02 | 7.93 2.57 | 22.35 5.49 |
| Yuan et al [ Yuan et al.(2021)Yuan, Liu, Hong, and Zhang ] | 0.75 0.92 | 7.99 22.47 | 0.30 0.47 | 6.12 11.04 | 0.79 1.51 | 199.68 497.18 | 0.46 0.12 | 6.79 6.46 | 1.17 2.76 | 90.93 179.32 | 4.92 4.11 | 12.78 20.35 |
| Koide et al [ Koide et al.(2023)Koide, Oishi, Yokozuka, and Banno ] | 0.79 2.53 | 11.52 17.04 | 1.33 3.68 | 32.38 67.31 | 0.48 0.79 | 94.46 351.19 | 0.35 0.47 | 9.78 8.92 | 0.49 0.60 | 46.30 67.05 | 6.60 3.71 | 23.38 14.46 |
| Taylor et al [ Taylor et al.(2015)Taylor, Nieto, and Johnson ] | 0.55 0.48 | 6.28 10.30 | 0.38 0.35 | 7.19 12.59 | 0.37 0.41 | 84.86 172.55 | 0.57 0.25 | 8.17 7.14 | 0.76 1.09 | 65.97 165.04 | 2.26 1.02 | 5.64 2.86 |
| ETH3D | Tanks and Temples | Livox | nuScenes | RGBDScenes | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | rot (° ) | transl (cm ) | |
| One stage | ||||||||||
| GS-SLAM [ Matsuki et al.(2024)Matsuki, Murai, Kelly, and Davison ] | 0.47 1.87 | 4.31 20.80 | 0.10 1.36 | 1.62 67.28 | 2.57 2.03 | 74.66 39.92 | 4.37 2.66 | 131.13 57.91 | 2.81 2.22 | 17.61 7.13 |
| Koide et al [ Koide et al.(2023)Koide, Oishi, Yokozuka, and Banno ] | 0.05 0.03 | 0.38 0.56 | 0.06 0.04 | 1.82 0.73 | 0.26 0.09 | 8.13 3.55 | 0.48 0.53 | 30.81 61.62 | 0.77 3.38 | 2.06 11.74 |
| Taylor et al [ Taylor et al.(2015)Taylor, Nieto, and Johnson ] | 0.55 0.53 | 6.03 10.20 | 0.33 0.43 | 6.30 14.72 | 0.93 0.29 | 9.58 7.64 | 0.92 1.74 | 50.40 183.95 | 2.13 0.82 | 5.92 2.40 |
| MatchAnything [ He et al.(2025)He, Yu, Peng, Tan, Shen, Bao, and Zhou ] | 0.07 0.10 | 0.66 0.79 | 0.07 0.05 | 1.64 1.52 | 0.26 0.15 | 3.14 5.77 | 0.11 0.06 | 11.24 10.67 | 0.40 0.24 | 0.85 0.64 |
| ETH3D | Tanks and Temples | CARLA | Livox | nuScenes | RGBDScenes | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | pix@ (% ) | (px ) | |||||||||||||
| 1 | 2 | 5 | 1 | 2 | 3 | 5 | 1 | 2 | 5 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10 | |||||||
| (VII) | ||||||||||||||||||||||||
| Rendered depth | 39 | 93 | 100 | 1.04 0.55 | 0 | 58 | 92 | 100 | 1.95 0.76 | 8 | 36 | 78 | 2.45 7.23 | 25 | 95 | 6.48 2.32 | 3 | 36 | 92 | 5.66 8.57 | 6 | 22 | 80 | 6.61 4.58 |
| Monocular normals | 0 | 2 | 9 | 8.34 2.26 | 0 | 28 | 90 | 100 | 2.43 0.58 | 2 | 23 | 71 | 3.29 6.49 | 65 | 100 | 4.94 0.27 | 0 | 12 | 71 | 7.15 13.75 | 18 | 57 | 94 | 4.47 2.83 |
| Monocular depth | 0 | 0 | 24 | 7.20 5.17 | 0 | 5 | 32 | 83 | 3.62 2.67 | 2 | 7 | 41 | 5.76 559.08 | 5 | 60 | 9.77 2.87 | 0 | 2 | 37 | 11.88 34.59 | 5 | 61 | 90 | 4.59 3.03 |
| ETH3D | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | rot (° ) | transl (cm ) | ||||||||
| 1 | 2 | 3 | 5 | 10 | 20 | 50 | 100 | ||||
| 0px initialization | |||||||||||
| MatchAnything | 14 | 67 | 98 | 100 | 100 | 100 | 100 | 100 | 1.66 0.66 | 0.10 0.05 | 0.94 1.55 |
| ReFineR | 64 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 0.89 0.35 | 0.06 0.04 | 0.70 0.62 |
| MatchAnything* + ReFineR | 65 | 97 | 100 | 100 | 100 | 100 | 100 | 100 | 0.87 0.38 | 0.06 0.05 | 0.68 0.74 |
| CARLA (normals) | ETH3D (intensity) | |
|---|---|---|
| Time (s) | Time (s) | |
| GS-SLAM [ Matsuki et al.(2024)Matsuki, Murai, Kelly, and Davison ] | 36.13 | 9.68 |
| Render | 11.31 | 3.48 |
| Loss | 0.12 | 0.62 |
| Backward | 22.24 | 4.62 |
| Yuan et al [ Yuan et al.(2021)Yuan, Liu, Hong, and Zhang ] | 47.03 | N/A |
| time / iter (s) | Num Points (M) | |
|---|---|---|
| ETH3D | 0.03 0.02 | 3.21 2.82 |
| Tanks and Temples | 0.14 0.07 | 19.92 11.40 |
| CARLA | 0.08 0.05 | 3.74 2.72 |
| Livox | 0.07 0.02 | 5.01 1.68 |
| nuScenes | 0.02 0.00 | 1.02 0.35 |
| RGBDScenes | 0.03 0.01 | 0.65 0.17 |
| ETH3D | CARLA | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pix@ (% ) | (px ) | time (s) | pix@ (% ) | (px ) | time (s) | |||||||||
| 1 | 2 | 3 | 5 | 10 | 1 | 2 | 3 | 5 | 10 | |||||
| ReFineR (s=100%) | 41 | 73 | 78 | 79 | 96 | 1.10 3.17 | 6.83 5.03 | 12 | 44 | 57 | 74 | 92 | 2.36 3.45 | 11.62 7.98 |
| ReFineR (s=75%) | 31 | 64 | 68 | 73 | 96 | 1.35 3.26 | 6.73 4.15 | 2 | 25 | 48 | 60 | 92 | 3.58 3.36 | 12.38 10.85 |
| ReFineR (s=50%) | 29 | 58 | 61 | 69 | 92 | 1.45 3.52 | 5.35 3.09 | 2 | 22 | 36 | 54 | 88 | 4.27 4.01 | 9.63 8.06 |
| ReFineR (s=25%) | 19 | 43 | 46 | 51 | 92 | 4.20 3.71 | 4.00 2.17 | 0 | 8 | 29 | 48 | 80 | 5.17 3.92 | 5.97 4.05 |
Point Cloud Registration
Point cloud registration aligns 3D point sets by estimating the transformations and correspondences that make their geometry consistent.
See all 119 papers