Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration

Pei An; Muyao Peng; Junfeng Ding; Jiaqi Yang; Liangliang Nan

cs.CVAug 28, 2026

Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration

Authors: Pei An, Muyao Peng, Junfeng Ding, Jiaqi Yang, Liangliang Nan

Organizations: Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Delft University of Technology, Netherlands

Abstract

Learning-based image-to-point-cloud (I2P) registration has garnered increasing attention in recent years. Nevertheless, existing methods still struggle with severe outliers under challenging scenarios with unseen, low-inlier, or distorted cases. A fast and robust 2D-3D correspondence pruning method is therefore highly desirable. Recently, a promising scheme lifts 2D-3D correspondences to 3D-3D correspondences using depth priors, casting correspondence pruning as a Sim(3) registration problem. However, depth priors estimated from monocular images are inherently noisy, which undermines the reliability of this scheme. In this paper, to explicitly model non-negligible depth noise, we reformulate correspondence pruning as an extended Sim(3) registration problem and propose a simple yet effective pruning algorithm termed Ex-Sim(3)-Reg. We further provide a theoretical analysis to justify the effectiveness of our method. Extensive experiments on the 7-Scenes, RGBD-V2, ScanNet, and TUM datasets demonstrate that Ex-Sim(3)-Reg achieves up to 24.7% improvement in registration recall over state-of-the-art baseline methods. Code is released at github.com/anpei96/ex-sim3-demo

Figures & tables

Figure 1: Motivation. We extend the Sim(3) registration to model the noise $\delta_{i}$ in depth priors as correspondence-specific scale perturbations $s_{i}=s\cdot\delta_{i}$ . This simple reformulation significantly improves pruning robustness against inaccurate depth priors.

Registration problem	Inlier constraint	Objectives	Num. of variables
SE(3)	$\mathbf{p}_{i}=\mathbf{R}\mathbf{q}_{i}+\mathbf{t}$	Estimate inliers $\{w_{i}\in\{0,1\}\}_{i=1}^{\mathrm{N}}$ , $\mathbf{R}$ , $\mathbf{t}$	$\mathrm{N}+6$
Regular Sim(3)	$\,\,\,\mathbf{p}_{i}=s\mathbf{R}\mathbf{q}_{i}+\mathbf{t}$	Estimate inliers $\{w_{i}\in\{0,1\}\}_{i=1}^{\mathrm{N}}$ , $\mathbf{R}$ , $\mathbf{t}$ , $s$	$\mathrm{N}+7$
Extended Sim(3)	$\,\,\mathbf{p}_{i}=s_{i}\mathbf{R}\mathbf{q}_{i}+\mathbf{t}$	Estimate inliers $\{w_{i}\in\{0,1\}\}_{i=1}^{\mathrm{N}}$ , $\mathbf{R}$ , $\mathbf{t}$ , $\{s_{i}\}_{i=1}^{\mathrm{N}}$	$2\mathrm{N}+6$

Table 1: Comparison of SE(3), Sim(3), and extended Sim(3) registration formulations. In extended Sim(3), each correspondence is associated with an individual scale factor

s_{i}

, increasing the number of unknowns from N+7 to 2N+6, which renders the problem underdetermined without additional structural constraints.

Figure 2: Pipeline of Ex-Sim(3)-Reg. Through scale hypothesis sampling and inlier candidate search, 3D correspondences $\{\mathbf{d}_{i}\}_{i=1}^{\mathrm{N}}$ are transformed to $\{\mathbf{e}_{j}\}_{j=1}^{\mathrm{M}}$ . This reduces the extended Sim(3) registration into an SE(3) registration problem, enabling robust inlier estimation using off-the-shelf SE (3) solvers.

Figure 3: Evaluation protocol. Inlier Ratio (IR) measures the quality of inlier candidates, while Registration Recall (RR) evaluates the accuracy of the estimated rotation $\mathbf{\tilde{R}}_{opt}$ and translation $\mathbf{\tilde{t}}_{opt}$ . Baseline denotes no pruning. Sim(3)-X estimates a global scale $\tilde{s}$ [ 32 ] , transforms correspondences from $\{\mathbf{d}_{i}\}_{i=1}^{\mathrm{N}}$ to $\{(\tilde{s}\mathbf{q}_{i},\mathbf{p}_{i})\}_{i=1}^{\mathrm{N}}$ , and then applies an SE(3) solver X to identify inliers. A complete protocol is in the supplemental materials.

Inlier Ratio $\uparrow$
Method	SC-1	SC-2	SC-3	SC-4	SC-5	SC-6	Mean
Baseline	0.295	0.257	0.325	0.265	0.227	0.228	0.266
Sim(3)-FastMAC	0.374	0.316	0.379	0.328	0.296	0.306	0.333
Sim(3)-SC2-PCR	0.481	0.435	0.431	0.446	0.436	0.475	0.451
Sim(3)-SC2-PCR++	0.295	0.254	0.325	0.259	0.229	0.251	0.268
Sim(3)-MAC	0.372	0.313	0.365	0.320	0.307	0.322	0.333

Table 2: Correspondence pruning performance on the 7-Scenes dataset [ 17 ] under the disturbed condition. Bold indicates the best mean performance across scenes.

Inlier Ratio $\uparrow$
Method	Freiburg_01	Freiburg_02	Freiburg_03	Mean
Baseline	0.242	0.265	0.183	0.230
Sim(3)-FastMAC	0.317	0.363	0.241	0.307
Sim(3)-SC2-PCR	0.387	0.415	0.331	0.378
Sim(3)-SC2-PCR++	0.267	0.339	0.235	0.280
Sim(3)-MAC	0.323	0.355	0.231	0.303

Table 3: Correspondence pruning performance on the TUM dataset [ 26 ] under the disturbed condition. Bold indicates the best mean performance.

Inlier Ratio $\uparrow$
Method	RG-1	RG-2	RG-3	RG-4	RG-5	RG-6	RG-7	Mean
Baseline	0.242	0.248	0.223	0.224	0.273	0.282	0.259	0.250
Sim(3)-FastMAC	0.309	0.326	0.275	0.283	0.354	0.341	0.350	0.320
Sim(3)-SC2-PCR	0.413	0.415	0.352	0.365	0.367	0.369	0.373	0.379
Sim(3)-SC2-PCR++	0.205	0.252	0.239	0.231	0.309	0.322	0.248	0.258
Sim(3)-MAC	0.322	0.317	0.281	0.281	0.307	0.319	0.308	0.305

Table 4: Cross-dataset generalization from 7-Scenes [ 17 ] to RGBD-v2 [ 19 ] . Bold indicates the best mean performance. Results demonstrate robustness to domain shift in depth priors and correspondence distributions.

Inlier Ratio $\uparrow$
Method	SN-1	SN-2	SN-3	SN-4	SN-5	SN-6	SN-7	Mean
Baseline	0.463	0.469	0.388	0.269	0.474	0.408	0.390	0.409
Sim(3)-FastMAC	0.553	0.570	0.478	0.344	0.566	0.501	0.566	0.511
Sim(3)-SC2-PCR	0.625	0.616	0.552	0.398	0.629	0.594	0.644	0.579
Sim(3)-SC2-PCR++	0.473	0.487	0.377	0.297	0.463	0.442	0.472	0.430
Sim(3)-MAC	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Table 5: Cross-dataset evaluation from 7-Scenes [ 17 ] to ScanNet [ 13 ] . ‘N/A’ indicates failure to return a result within 1 second, reflecting computational instability under severe outlier contamination.

Inlier Ratio $\uparrow$
Method	RG-11	RG-12	RG-13	RG-14	Mean
Baseline	0.195	0.271	0.075	0.123	0.166
Sim(3)-FastMAC	0.241	0.318	0.085	0.138	0.195
Sim(3)-SC2-PCR	0.334	0.392	0.112	0.159	0.249
Sim(3)-SC2-PCR++	0.231	0.294	0.085	0.130	0.185
Sim(3)-MAC	0.244	0.311	0.093	0.135	0.196

Table 6: Correspondence pruning performance on the RGBD-v2 dataset [ 19 ] under low-inlier conditions. We restrict the inlier number to simulate extreme outlier scenarios. These highlight robustness when geometric constraints are limited.

Method	Matr	+Ours	FreeReg	+Ours	Top-I2P	+Ours	MinCD	+Ours	Diff-I2P	+Ours
IR	0.451	0.662	0.568	0.690	0.493	0.674	0.464	0.678	0.521	0.682
RR	0.842	0.943	0.780	0.932	0.901	0.950	0.898	0.957	0.914	0.963

Table 7: Cross-domain I2P registration performance on the ScanNet dataset [ 13 ] . We integrate the proposed Ex-Sim(3)-Reg (Ours) into different baselines, which demonstrates the superior improvements.

Figure 4: Qualitative comparison on challenging scenes. Competing methods either remove excessive inliers or retain geometrically inconsistent matches. Our method preserves denser and spatially coherent inlier sets.

$K$	4	6	8	10	12
IR/RR	0.505/0.362	0.542/0.394	0.548/ 0.472	0.573/0.448	0.682 /0.427
Time	57.6ms	65.4ms	73.6ms	81.7ms	89.5ms

Table 8: Effect of the number of sub-intervals

K

on IR, RR, and runtime. Increasing

K

improves accuracy but with increased computational overhead.

$\delta_{thr}$	8	12	16	20	24
IR/RR	0.513/0.380	0.535/0.466	0.548/0.472	0.521/0.384	0.505/0.372
RTE	0.072m	0.068m	0.067m	0.070m	0.071m

Table 9: Impact of pixel reprojection threshold

\delta_{thr}

on IR, RR, and relative translation error (RTE). Larger thresholds improve robustness but may admit additional outliers, revealing the robustness–precision trade-off.

MAC [ 39 ]	FastMAC [ 40 ]	Scale-correction function $\varphi(\cdot)$	RR	Average Runtime/ms
	$\checkmark$		0.311	72.4
$\checkmark$		$\checkmark$	0.485	204.8
	$\checkmark$	$\checkmark$	0.472	73.6

Table 10: Ablation studies on MAC search and scale-correction function. Removing the MAC search significantly degrades inlier consistency, while omitting the correction function reduces robustness to depth noise. Runtime comparison illustrates the computational trade-offs.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Histogram of scales under different scale values in the 7-Scenes dataset [ 17 ] , illustrating the prevalence of $s_{i}$ inconsistencies in real applications, which supports the use of the extended Sim(3).

Figure 6: Evaluation protocol. Inlier Ratio (IR) measures the quality of inlier candidates, while Registration Recall (RR) evaluates the accuracy of the estimated rotation $\mathbf{\tilde{R}}_{opt}$ and translation $\mathbf{\tilde{t}}_{opt}$ . (a) Baseline denotes the case without pruning. (b) Sim(3)-X first estimates a global scale $\tilde{s}$ [ 32 ] , converts correspondences from $\{\mathbf{d}_{i}\}_{i=1}^{\mathrm{N}}$ to $\{(\tilde{s}\mathbf{q}_{i},\mathbf{p}_{i})\}_{i=1}^{\mathrm{N}}$ , and then applies an SE(3) solver X to identify inliers. (c) Ex-Sim(3)-reg is the proposed pruning method. Iteration time in RANSAC P3P is the same (1K) in all compared methods.

Figure 7: Qualitative comparisons of different methods on the 7-Scenes dataset [ 17 ] . Green and red lines denote correct and incorrect correspondences, respectively.

Hyperparameters	Value
Number of intervals ( $K$ )	8
Threshold for 3D distance error ( $d_{thr}$ )	0.2
Threshold for 2D reprojection error ( $\delta_{thr}$ )	16
Scale interval ( $\Delta s$ )	$0.15\tilde{s}$

Appendix

Table 11: Key hyperparameters of the proposed Ex-Sim(3)-Reg method.

Pruning settings	Implementation scheme	$N_{\text{corr}}$	IR
Disturbed condition	Adding noise on 3D point cloud	Nearly 0.1K	Nearly 25%
Unseen condition	Open-domain testing	Nearly 1.0K	Nearly 30%
Few-inlier condition	Training with insufficient samples	Nearly 0.1K	Nearly 15%

Appendix

Table 12: Experimental settings for correspondence pruning.

N_{\text{corr}}

denotes the average number of 2D-3D correspondences, and IR denotes the average inlier ratio before pruning. These statistics reflect the difficulty level of each dataset.

Figure 8: Qualitative comparisons of different methods on the RGBD-v2 dataset [ 19 ] . Green and red lines indicate correct and incorrect correspondences, respectively.

Figure 9: Qualitative comparisons of different methods on the ScanNet dataset [ 13 ] . Green and red lines indicate correct and incorrect correspondences, respectively.

Figure 10: Qualitative comparisons of different methods on the TUM dataset [ 26 ] . Green and red lines indicate correct and incorrect correspondences, respectively.

Figure 11: Qualitative comparisons of the proposed method at different timestamps on the TUM dataset [ 26 ] .

Method	Sim(3)-FastMAC	Sim(3)-SC2-PCR	Sim(3)-SC2-PCR++	Sim(3)-MAC	Sim(3)-TurboReg	Ours
RTE/m	0.066	0.067	0.085	0.075	0.068	0.066
RRE/deg	2.426	2.375	3.005	2.539	2.381	2.367

Appendix

Table 13: Evaluation on the noisy 7-Scenes dataset. Bold numbers denote the best mean performance in terms of RRE.

Method	Sim(3)-FastMAC	Sim(3)-SC2-PCR	Sim(3)-SC2-PCR++	Sim(3)-MAC	Sim(3)-TurboReg	Ours
RTE/m	0.059	0.051	0.057	0.064	0.058	0.036
RRE/deg	2.824	2.003	2.949	3.074	2.994	1.824

Appendix

Table 14: Evaluation on the noisy TUM dataset. Bold numbers denote the best mean performance in terms of RRE.

Method	Sim(3)-FastMAC	Sim(3)-SC2-PCR	Sim(3)-SC2-PCR++	Sim(3)-MAC	Sim(3)-TurboReg	Ours
RTE/m	0.043	0.034	0.050	N/A	0.034	0.025
RRE/deg	1.470	1.008	2.127	N/A	1.095	0.751

Appendix

Table 15: Evaluation under an unseen testing setup (training on 7-Scenes, testing on ScanNet). Bold numbers denote the best mean performance in terms of RRE.

Method	Sim(3)-FastMAC	Sim(3)-SC2-PCR	Sim(3)-SC2-PCR++	Sim(3)-MAC	Sim(3)-TurboReg	Ours
RTE/m	0.097	0.106	0.107	0.104	0.104	0.093
RRE/deg	2.854	3.111	3.145	2.947	2.913	2.839

Appendix

Table 16: Evaluation on the challenging RGBD-v2 dataset with a low inlier ratio. Bold numbers denote the best mean performance in terms of RRE.

Method	Sim(3)-FastMAC	Sim(3)-SC2-PCR	Sim(3)-SC2-PCR++	Sim(3)-MAC	Sim(3)-TurboReg	Ours
Time/ms	23.1	24.2	105.3	150.4	15.4	73.6

Appendix

Table 17: Runtime comparison (in milliseconds) of I2P pruning methods on the 7-Scenes dataset.

Figure 12: Failure case of the proposed pruning method due to unsuitable hyperparameter setting . See Sec. A3 for analysis. Green and red lines indicate the correct and incorrect correspondences, respectively.

Model Names	Descriptions	Depth Accuracy	Runtime	Model Size	Used in Main Paper
depth_anything_v2_metric_hypersim_vits	vits	Low	nearly 32 ms	99.2 MB
depth_anything_v2_metric_hypersim_vitb	vitb	Moderate	nearly 65 ms	390.0 MB
depth_anything_v2_metric_hypersim_vitl	vitl	High	nearly 138 ms	1.3 GB	$\checkmark$

Appendix

Table 18: Comparison of monocular depth estimation models. The baseline monocular depth estimation model used in our experiments is Depth Anything V2 [ 34 ] .

Models	Using vits.pth	Using vitb.pth	Using vitl.pth
Sim(3)-TurboReg [ 31 ]	0.083	0.102	0.118
Sim(3)-MAC [ 39 ]	0.092	0.095	0.109
Our method	0.333	0.317	0.277

Appendix

Table 19: Registration recall of different pruning methods using Depth Anything v2 with different model sizes on the TUM Freiburg_01 scene [ 26 ] .

Models	MASt3R [ 20 ]	Depth Anything v2 [ 34 ]	MoGe v2 [ 30 ]
Sim(3)-TurboReg [ 31 ]	0.092	0.118	0.145
Our method	0.264	0.277	0.323

Appendix

Table 20: Registration recall of the pruning methods with different monocular depth estimation models on the TUM Freiburg_01 scene [ 26 ] .

Figure 13: Effect of the proposed scale-aware strategy . (a) When all depth predictions share a single global scale, the transformed 3D points (shown in red ) do not align well with their corresponding reference points (shown in green ), revealing noticeable scale inconsistency. (b) By generating 3D points with multiple scales and selecting the optimal scale for each correspondence, the red points align better with the green ones. This correspondence-wise adjustment effectively reduces scale distortion and explains why the proposed extended Sim(3) based pruning method is more robust than existing approaches based on the regular Sim(3) registration.

2D-3D Image Registration

See all 21 papers

Jul 19, 2026

Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration

Recent detection-free approaches have shown significant efficacy in image-to-point cloud (I2P) registration by employing a coarse-to-fine matching pipeline. In the coarse stage, down-sampled image features and voxelized point cloud features are typically fused to establish initial coarse correspondences for subsequent refinement. However, existing methods largely overlook the critical role of point cloud density, which fundamentally dictates the quality of coarse correspondences and the final registration results. Specifically, excessively sparse point clouds lead to an insufficient number of inliers, while overly dense ones often introduce a high outlier ratio. Consequently, this creates an inherent density trade-off, thereby significantly limiting the registration accuracy of current approaches. For mitigating this trade-off, we propose a novel Cross-Coordinate Correspondences Pruning (CCP) strategy to acquire sufficient inliers while ensuring a low outlier ratio. To minimize interference from inter-modal coordinate discrepancies, we first project cross-coordinate coarse correspondences to the 2D image coordinate system for spatial unification. Subsequently, a lightweight pruning network is responsible for predicting the inlier confidences, which are used to filter coarse outliers, from coordinate geometric and modal feature dimensions. To maximize inlier recall, we further design a Multi-Density Point Ensemble (MDPE) strategy that consolidates and deduplicates pruned coarse correspondences across varying point cloud densities. Our method achieves a significant performance improvement, surpassing existing state-of-the-art methods by at least 8.6% in Registration Recall across various benchmarks.

Xin Liu, Rong Qin, Huipeng Lin +5

VCIP & TMCC & DISSec, College of Computer Science, Nankai University, Tianjin, China. · AutoCity (Shenzhen) Autonomous Driving Co., Ltd, Shenzhen, China. · School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing, China. +4

Point Cloud Registration Cross-Modal Feature Matching Cross-Modal Image Registration

Ask PDF

Mar 30, 2026

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

Image-to-point-cloud (I2P) registration aims to align 2D images with 3D point clouds by establishing reliable 2D-3D correspondences. The drastic modality gap between images and point clouds makes it challenging to learn features that are both discriminative and generalizable, leading to severe performance drops in unseen scenarios. We address this challenge by introducing a heterogeneous graph that enables refining both cross-modal features and correspondences within a unified architecture. The proposed graph represents a mapping between segmented 2D and 3D regions, which enhances cross-modal feature interaction and thus improves feature discriminability. In addition, modeling the consistency among vertices and edges within the graph enables pruning of unreliable correspondences. Building on these insights, we propose a heterogeneous graph embedded I2P registration method, termed Hg-I2P. It learns a heterogeneous graph by mining multi-path feature relationships, adapts features under the guidance of heterogeneous edges, and prunes correspondences using graph-based projection consistency. Experiments on six indoor and outdoor benchmarks under cross-domain setups demonstrate that Hg-I2P significantly outperforms existing methods in both generalization and accuracy. Code is released on https://github.com/anpei96/hg-i2p-demo.

Pei An, Junfeng Ding, Jiaqi Yang +3

Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Huazhong Agricultural University, China +1

2D-3D Image Registration Point Cloud Registration Cross-Modal Image Registration Cross-Modal Feature Matching Domain Generalization

Ask PDF

Sep 16, 2026

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.

Zhixin Cheng, Jiacheng Deng, Xiaotian Yin +3

School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Masked Autoencoders Point Cloud Registration Cross-Modal Feature Matching Cross-Modal Image Registration 2D-3D Image Registration

Ask PDF

Abstract

2D-3D Image Registration

Explore similar work

Cross-Coordinate Correspondence Pruning for Image-to-Point Cloud Registration

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration