Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

Pei An; Junfeng Ding; Jiaqi Yang; Yulong Wang; Jie Ma; Liangliang Nan

cs.CVMar 30, 2026

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

Authors: Pei An, Junfeng Ding, Jiaqi Yang, Yulong Wang, Jie Ma, Liangliang Nan

Organizations: Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Huazhong Agricultural University, China · Delft University of Technology, Netherlands

Abstract

Image-to-point-cloud (I2P) registration aims to align 2D images with 3D point clouds by establishing reliable 2D-3D correspondences. The drastic modality gap between images and point clouds makes it challenging to learn features that are both discriminative and generalizable, leading to severe performance drops in unseen scenarios. We address this challenge by introducing a heterogeneous graph that enables refining both cross-modal features and correspondences within a unified architecture. The proposed graph represents a mapping between segmented 2D and 3D regions, which enhances cross-modal feature interaction and thus improves feature discriminability. In addition, modeling the consistency among vertices and edges within the graph enables pruning of unreliable correspondences. Building on these insights, we propose a heterogeneous graph embedded I2P registration method, termed Hg-I2P. It learns a heterogeneous graph by mining multi-path feature relationships, adapts features under the guidance of heterogeneous edges, and prunes correspondences using graph-based projection consistency. Experiments on six indoor and outdoor benchmarks under cross-domain setups demonstrate that Hg-I2P significantly outperforms existing methods in both generalization and accuracy. Code is released on https://github.com/anpei96/hg-i2p-demo.

Figures & tables

Figure 1: Motivation of Hg-I2P . To achieve generalizable I2P registration, we reformulate the baseline architecture using a heterogeneous graph. Its heterogeneous edges enhance cross-modal feature interaction, improving feature discriminability. Projection constraints within the graph enable consistency-based pruning of mismatched correspondences, enhancing robustness. The resulting framework, Hg-I2P, jointly refines both features and matches, achieving strong generalization across unseen scenes.

Figure 2: Constructing a heterogeneous graph ( $\mathcal{G}_{H}$ ) for cross-modal reasoning. This graph models the relationship between 2D and 3D regions. By explicitly linking visual and geometric entities through heterogeneous edges, it captures structured 2D-3D dependencies essential for joint reasoning.

Figure 3: Pipeline of Hg-I2P . MP-mining learns heterogeneous edges $\mathcal{E}_{\text{I2P}}$ and constructs the graph $\mathcal{G}_{H}$ . HE-adapting refines cross-modal features via message passing along heterogeneous edges, improving cross-modal alignment. HC-pruning enforces projection consistency within $\mathcal{G}_{H}$ to remove outliers. Together, these components refine both features and correspondences for robust I2P registration.

Figure 4: Overview of HC-pruning. It filters incorrect correspondences using two projection consistency criteria derived from $\mathcal{G}_{H}$ .

IR	C $\rightarrow$ F	C $\rightarrow$ H	C $\rightarrow$ O	C $\rightarrow$ P	C $\rightarrow$ K	C $\rightarrow$ S	AVG
MATR	0.455	0.359	0.420	0.411	0.390	0.288	0.387
Top-I2P	0.491	0.427	0.455	0.461	0.437	0.327	0.433
MinCD	0.542	0.424	0.502	0.408	0.416	0.379	0.445
Hg-I2P †	0.547	0.467	0.492	0.520	0.475	0.332	0.472
Hg-I2P (Ours)	0.592	0.588	0.744	0.533	0.557	0.469	0.581
RR	C $\rightarrow$ F	C $\rightarrow$ H	C $\rightarrow$ O	C $\rightarrow$ P	C $\rightarrow$ K	C $\rightarrow$ S	AVG

Table 1: Cross-scene I2P registration results on the 7-Scenes dataset [ 14 ] .

{\dagger}

denotes the Hg-I2P variant without HC-pruning.

IR	RG-1	RG-2	RG-3	RG-4	RG-5	RG-6	RG-7	AVG
MATR	0.351	0.353	0.336	0.316	0.250	0.209	0.222	0.291
Top-I2P	0.397	0.413	0.389	0.398	0.252	0.247	0.253	0.335
MinCD	0.427	0.415	0.405	0.412	0.310	0.296	0.329	0.371
Hg-I2P (Ours)	0.503	0.499	0.444	0.456	0.325	0.285	0.314	0.404
RR	RG-1	RG-2	RG-3	RG-4	RG-5	RG-6	RG-7	AVG
MATR	0.970	0.880	0.871	0.741	0.480	0.449	0.458	0.692

Table 2: Cross-dataset I2P registration performance across multiple datasets, including RGBD-V2 [ 19 ] , ScanNet [ 10 ] , and a self-collected dataset.

Methods	Venue	IR	RR
LCD †	AAAI 2020	0.307	N/A
Glue †	CVPR 2020	0.183	0.065
P2-Net	ICCV 2021	0.303	0.711
MATR	ICCV 2023	0.451	0.842
FreeReg †	ICLR 2024	0.568	0.780
Top-I2P	IJCAI 2025	0.493	0.901

Table 3: Comparisons of I2P registration methods on the ScanNet dataset [ 10 ] .

{\dagger}

denotes evaluation with an RR threshold of 30 cm, while other methods use 5 cm.

Freiburg_01	IR	RR	RTE/m	RRE/deg
MATR	0.319	0.595	0.037	1.238
Top-I2P	0.306	0.655	0.036	1.179
Hg-I2P (Ours)	0.380	0.681	0.035	1.147
Freiburg_02	IR	RR	RTE/m	RRE/deg
MATR	0.223	0.223	0.052	1.469
Top-I2P	0.234	0.310	0.054	1.502

Table 4: Performance comparison on the TUM dataset [ 31 ] (RR threshold is 10 cm). Hg-I2P consistently achieves the highest IR and RR, and lowest RTE and RRE across all sequences.

Figure 5: Effect of the proposed HE-adapting module. (a) Baseline without HE-adapting. (b) With HE-adapting, cross-modal features are more accurately aligned, producing a higher number of inliers.

Methods	Venue	RTE/m	RRE/deg
CorrI2P	TCSVT 2022	0.740	2.070
VP2P-match	NeurIPS 2023	0.750	3.290
CoFiI2P	RAL 2024	0.290	1.140
CMR-Agent	IROS 2024	0.195	0.589
GraphI2P	CVPR 2025	0.320	1.650
ImCorr	CVPR 2025	0.200	1.240

Table 5: Comparisons on the KITTI dataset [ 13 ] .

{\dagger}

indicates that LiDAR point clouds are constructed from multiple scans, a common setup in visual localization using pre-built maps.

Methods	Venue	IR	RR
P2-Net	ICCV 2021	0.122	0.384
MATR	ICCV 2023	0.324	0.564
FreeReg	ICLR 2024	0.309	0.573
DiffReg †	ECCV 2024	0.378	0.874
Bridge	AAAI 2025	0.351	0.634
DiffI2P	ICCV 2025	0.369	0.605

Table 6: Comparisons on the RGBD-V2 dataset [ 19 ] .

{\dagger}

denotes methods requiring depth input from RGB-D camera.

Figure 6: Qualitative comparison of I2P registration results. (a) MATR [ 23 ] . (b) Top-I2P [ 2 ] . (c) Hg-I2P without HC-pruning. (d) Full Hg-I2P which produces fewer outliers than prior methods.

$\mathbf{E}_{\text{I2P}}^{1}$	$\mathbf{E}_{\text{I2P}}^{2}$	$\mathbf{E}_{\text{I2P}}^{3}$	Precision	IR	RR
$\checkmark$			0.826	0.478	0.610
$\checkmark$	$\checkmark$		0.874	0.490	0.634
$\checkmark$	$\checkmark$	$\checkmark$	0.893 ( +6.7% )	0.494 ( +1.6% )	0.661 ( +5.1% )

Table 7: Ablation study of the MP-mining module.

\mathbf{E}_{\text{I2P}}^{i}

denotes the inclusion of the

i

-th feature heterogeneous edge prediction.

Parameter $\beta$ in Eq. (8)	0.0	0.05	0.10	0.15	0.20	0.25
IR	0.462	0.478	0.494	0.482	0.454	0.437
RR	0.620	0.648	0.661	0.612	0.603	0.582

Table 8: Ablation study of the HE-adapting module. Parameter

\beta

controls the strength of cross-modal feature interaction through heterogeneous edges.

Criterion I	Criterion II	IR	RR
		0.494	0.661
$\checkmark$		0.523	0.682
$\checkmark$	$\checkmark$	0.558 ( +6.4% )	0.690 ( +2.9% )

Table 9: Ablation study of the HC-pruning module. “Criterion X” indicates the use of the

X

-th HC-pruning criterion.

Figure 7: Examples from the self-collected indoor I2P dataset. Representative RGB and depth scenes captured using an Intel RealSense D415 in indoor environments such as tables, living rooms, and shelf-like spaces. The dataset includes realistic depth noise and fine-grained geometric variation and is used exclusively for zero-shot evaluation of I2P registration.

Figure 8: 2D and 3D SAM-based segmentation results. The resulting 2D and 3D region sets exhibit strong shape and boundary consistency, enabling reliable heterogeneous graph construction for cross-modal representation learning.

Methods	Venue	RTE/m	RRE/deg
MATR	ICCV 2023	0.029	0.955
Top-I2P	IJCAI 2025	0.030	0.963
MinCD	ICCV 2025	0.027	0.937
Hg-I2P		0.028	0.954
Hg-I2P †		0.026	0.903

Table 10: Mean RTE and RRE of MATR, Top-I2P, MinCD, and Hg-I2P in the cross-scene setting on 7-Scenes. The variant Hg-I2P † excludes HC-pruning. The results highlight the impact of heterogeneous graph modeling and pruning on registration accuracy.

Figure 9: Qualitative I2P registration results of MATR [ 23 ] (left), Top-I2P [ 2 ] (middle), and the proposed Hg-I2P (right) on the KITTI dataset [ 13 ] . Hg-I2P produces more accurate and stable 2D-3D correspondences across diverse traffic scenes, benefiting from heterogeneous graph modeling and HC-pruning. Green and red lines denote correct and incorrect correspondences, respectively.

Figure 10: Examples of SAM segmentation results on outdoor images from KITTI [ 13 ] . Despite the presence of objects with irregular structures and large depth variability, SAM reliably identifies fine-grained regions, contributing to the robustness of Hg-I2P in outdoor environments after overlap extraction.

Figure 11: Qualitative comparison of Top-I2P [ 2 ] (left), MinCD [ 3 ] , and Hg-I2P (right, ours) on ScanNet [ 10 ] . Hg-I2P consistently yields denser correct matches and significantly fewer outliers because of its heterogeneous graph design and multi-path relation mining. Green and red lines denote the correct and incorrect correspondences, respectively.

Figure 12: Qualitative comparison among MATR [ 23 ] (left), Hg-I2P without HC-pruning (middle), and Hg-I2P (right) on the 7-Scenes dataset [ 13 ] . HC-pruning noticeably reduces incorrect correspondences and enhances the inlier ratio by enforcing graph-aware geometric consistency. Green and red lines denote correct and incorrect correspondences, respectively.

Figure 13: Examples where Hg-I2P fails to generate reliable correspondences. Errors arise from incorrect heterogeneous edge prediction and overly relaxed pruning thresholds, often triggered by SAM over-segmentation or highly cluttered scenes. Green and red lines denote correct and incorrect correspondences, respectively.

$\delta_{\text{rej}}$	5	10	15	20	25
Registration recall (RR)	0.587	0.652	0.682	0.670	0.662

Table 11: Ablation results showing how the geometric reprojection threshold

\delta_{\text{rej}}

influences registration recall. Moderate thresholds (e.g.,

\delta_{\text{rej}}=15

) achieve the best balance between retaining inliers and rejecting misaligned correspondences.

Scheme of $\tau_{\text{rej}}$	Fixed	Adaptive (Top-K selection)
Inlier ratio (IR)	0.537	0.558
Registration recall (RR)	0.685	0.690

Table 12: Comparison of fixed and top-K adaptive cosine-similarity threshold

\tau_{\text{rej}}

in HC-pruning. Adaptive selection improves both inlier ratio and registration recall.

$\lambda_{1}$	0.032	0.048	0.064	0.080	0.096
Registration recall (RR)	0.602	0.638	0.661	0.652	0.626

Table 13: Ablation study on

\lambda_{1}

(without HC-pruning), demonstrating that moderate weighting yields optimal registration recall by balancing heterogeneous edge prediction and correspondence learning.

Proc.	SAM	Ini.	MP-mining	HE-adapting	HC-pruning	All
Time	47.2	72.4	3.9	34.6	19.7	177.8

Table 14: Runtime of each pipeline component, including SAM segmentation, graph initialization (including feature extraction), multi-path relation mining, heterogeneous edge adaptation, and HC-pruning. Graph initialization and SAM dominate the total runtime (unit: ms).

Figure 14: Visualization of how different grid-based prompting schemes for SAM (e.g., $8\times 8$ vs. $16\times 16$ ) affect segmentation granularity and the resulting 2D–3D correspondences. Denser prompting generally improves alignment but increases inference time. Green and red lines denote correct and incorrect correspondences, respectively.

Prompt of 2D SAM	RR	Time
$8\times 8$ grid points	62.1	144.2 ms
$16\times 16$ grid points (used in experiments)	68.2	177.8 ms

Table 15: Evaluation of segmentation prompt density (

8\times 8

vs.

16\times 16

grid points) on registration accuracy and inference time. Denser prompting provides higher registration recall at the cost of additional computation.

Explore similar work

GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration
Mar 27, 2026Zhixin Cheng, Jiacheng Deng, Xinjun Li +5Point Cloud Registration Cross-Modal Image Registration
School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Meituan Inc, Beijing 100000, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China
Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration
Jul 19, 2026Xin Liu, Rong Qin, Huipeng Lin +5Point Cloud Registration Cross-Modal Feature Matching
VCIP & TMCC & DISSec, College of Computer Science, Nankai University, Tianjin, China. · AutoCity (Shenzhen) Autonomous Driving Co., Ltd, Shenzhen, China. · School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing, China. +4
OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration
Sep 29, 2026Pei An, Jiaqi Yang, Yulong Wang +2Cross-Attention Point Cloud Registration
Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Huazhong Agricultural University, China +2

Mar 27, 2026J/K move · Enter open · S save

GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration

Zhixin Cheng, Jiacheng Deng, Xinjun Li, Bohao Liao+4

School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Meituan Inc, Beijing 100000, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Point Cloud Registration Cross-Modal Image Registration Cross-Modal Feature Matching 3D Scene Understanding

Open paper AskPDF arXiv

Image-to-point cloud registration methods typically follow a coarse-to-fine pipeline, extracting patch-level correspondences and refining them into dense pixel-to-point matches. However, in scenes with repetitive patterns, images often lack sufficient 3D structural cues and alignment with point clouds, leading to incorrect matches. Moreover, prior methods usually overlook structural consistency, limiting the full exploitation of correspondences. To address these issues, we propose two novel modules: the Local Geometry Enhancement (LGE) module and the Graph Distribution Consistency (GDC) module. LGE enhances both image and point cloud features with normal vectors, injecting geometric structure into image features to reduce mismatches. GDC constructs a graph from matched points to update features and explicitly constrain similarity distributions. Extensive experiments and ablations on two benchmarks, RGB-D Scenes v2 and 7-Scenes, demonstrate that our approach achieves state-of-the-art performance in image-to-point cloud registration.

Abstract

Explore similar work

GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration

Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration

OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration