Organizations: School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Meituan Inc, Beijing 100000, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China
Image-to-point cloud registration methods typically follow a coarse-to-fine pipeline, extracting patch-level correspondences and refining them into dense pixel-to-point matches. However, in scenes with repetitive patterns, images often lack sufficient 3D structural cues and alignment with point clouds, leading to incorrect matches. Moreover, prior methods usually overlook structural consistency, limiting the full exploitation of correspondences. To address these issues, we propose two novel modules: the Local Geometry Enhancement (LGE) module and the Graph Distribution Consistency (GDC) module. LGE enhances both image and point cloud features with normal vectors, injecting geometric structure into image features to reduce mismatches. GDC constructs a graph from matched points to update features and explicitly constrain similarity distributions. Extensive experiments and ablations on two benchmarks, RGB-D Scenes v2 and 7-Scenes, demonstrate that our approach achieves state-of-the-art performance in image-to-point cloud registration.
Figures & tables
Fig. 1 : (a) Visualization of reduced mismatches aided by normal-based structural information. The normals provide shared attributes between the image and the point cloud, enhancing the matching accuracy between them. (b) Visualization of structural similarity distribution constraint. By enforcing the structural consistency constraint, the alignment of image and point cloud keypoints is improved, leading to a better understanding of the scene.
Fig. 2 : Overall pipeline of the GLASS. It includes the Local Geometry Enhancement (LGE) and Graph Distribution Consistency (GDC) modules. In LGE, the image branch predicts surface normals via an Image Normal Prediction Head, trained using pseudo normal labels generated by Depth Anything v2. The point cloud branch computes normals using a Point Normal Estimation Unit. By fusing normals with initial features, the model gains structural awareness and reduces mismatches. In GDC, matching points in the local neighborhoods of both image and point cloud are constructed as graphs to update features. Similarity distributions are then constrained to enforce structural consistency, improving the quality of dense correspondences.
Model
Scene.11
Scene.12
Scene.13
Scene.14
Mean
Mean depth (m)
1.74
1.66
1.18
1.39
1.49
Inlier Ratio ↑
FCGF-2D3D
6.8
8.5
11.8
5.4
8.1
P2-Net
9.7
12.8
17.0
9.3
12.2
Predator-2D3D
17.7
19.4
17.2
8.4
15.7
2D3D-MATR
32.8
34.4
39.2
23.3
32.4
TABLE I : Evaluation results on RGB-D Scenes v2. Red numbers highlight the best, the second best are Boldfaced and the baseline are underlined .
Mod
Chs
Fr
Hds
Off
Pmp
Ktn
Strs
Mean
Mdpt
1.78
1.55
0.80
2.03
2.25
2.13
1.84
1.77
Inlier Ratio ↑
FCG
34.2
32.8
14.8
26
23.3
22.5
6.0
22.8
P2N
55.2
46.7
13.0
36.2
32.0
32.8
5.8
31.7
PRE
34.7
33.8
16.6
25.9
23.1
22.2
7.5
23.4
2D3
72.1
66.0
31.3
60.7
50.2
52.5
18.1
50.1
TABLE II : Evaluation results on 7-Scenes. Models are abbreviated using the first three letters, and scenes are abbreviated. Red numbers highlight the best, the second best are Boldfaced and the baseline are underlined .
Method
Type
RTE(m) ↓
RRE ( ∘ ) ↓
vpc + GeoTransformer [ 21 ]
Point-to-Point
4.27 ± 7.14
8.67 ± 8.55
vpc + Hunter
Point-to-Point
4.59 ± 5.22
6.23 ± 5.17
DeepI2P(2D) [ 39 ]
Image-to-Point
5.15 ± 7.35
9.14 ± 8.02
CorrI2P [ 38 ]
Image-to-Point
4.24 ± 7.26
6.47 ± 5.20
VP2P [ 71 ]
Image-to-Point
2.05 ± 3.23
4.01 ± 6.37
2D3D-MATR [ 2 ]
Image-to-Point
1.86 ± 3.79
2.59 ± 4.46
TABLE III: Evaluation Results on KITTI Dataset.
Dataset
RGB-D Scenes v2
7-Scenes
Model
Scene-11
Scene-12
Scene-13
Scene-14
Mean
Chess
Fire
Heads
Office
Pumpkin
Kitchen
Stairs
Mean
Mdpt(m)
1.74
1.66
1.18
1.39
1.49
1.78
1.55
0.80
2.03
2.25
2.13
1.84
1.49
Mean_RRE (°) ↓
2D3D-MATR
2.294
2.628
3.823
3.358
3.026
2.298
3.144
7.549
2.285
2.439
2.620
2.705
3.291
B2-3Dnet
2.245
2.317
2.604
3.233
2.600
2.205
3.105
7.414
2.294
2.421
2.618
2.564
3.232
CA-I2P
2.008
2.031
3.306
2.890
2.559
2.354
3.093
7.105
2.272
2.297
2.624
2.655
3.200
TABLE IV : Evaluation results on RGB-D Scenes v2 and 7-Scenes using Relative Rotation Error (RRE) and Relative Translation Error (RTE). Bold-faced numbers highlight the best and the second best are underlined .
Method
LGE
GDC
GAT
DINO
PIR
IR
FMR
RR
M1
48.5
32.5
91.0
56.4
M2
✓
54.3
37.8
92.4
73.2
M3
✓
✓
65.3
42.9
93.2
79.8
M4
✓
58.9
37.2
92.4
62.1
M5
✓
✓
59.2
35.9
92.4
64.1
M6
✓
✓
✓
59.1
39.2
93.3
75.3
TABLE V : Ablation study results on RGB-D Scenes v2. GAT represents a lightweight graph attention network. DINO represents DINO v2. Blue numbers highlight the best, the second best are underlined .
Model
2D3D-MATR
Flow-I2P
Ours
Ours w/o DINOv2
FPS ↑
7.852
6.061
1.382
5.884
GFLOPs ↓
388.2
525.1
889.1
540.2
Memory (GB) ↓
5.637
2.844
12.1
5.734
Param (M) ↓
28.2
28.7
322.9
29.1
TABLE IX : Efficiency analysis and ablation study
Fig. 3 : Visualization of modality differences. With training, the modalities and distributions of the point cloud and image become more aligned.
Fig. 4 : Visualization of the image-to-point cloud matching results of GLASS. To rigorously analyze the performance, we set the error threshold to a strict 30px. As seen, our method significantly improves the matching performance in challenging scenarios.
Fig. 5 : The visualization of the point cloud projection onto the image shows that our method achieves an accurate rigid transformation, without causing significant misalignment.
Scenes
Pumpkin
Stairs
Previous
83.8
29.7
Modify
84.3
36.8
TABLE X : Comparison of RR Performance on Pumpkin and Stairs Scenes
Fig. 6 : Limitation visualization on Dataset. The figure illustrates the comparison from the baseline to our method in challenging regions. The comparison from left to right represents the baseline and our method.
School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China