DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

Wenxin Zhang; Hang Li; Zhiwei Xu; Qiankun Dong; Gang Wang; Tao Li

cs.CVAug 27, 2026

DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

Authors: Wenxin Zhang, Hang Li, Zhiwei Xu, Qiankun Dong, Gang Wang, Tao Li

Organizations: Nankai University, Tianjin, China · Haihe Lab of ITAI, Tianjin 300459, China

Abstract

Image-to-Point Cloud Registration aims to estimate the camera pose of a given image within a 3D scene point cloud, which is a fundamental task in autonomous driving and large-scale outdoor localization. Recent implicit correspondence learning methods have improved registration performance by learning cross-modal alignment in an end-to-end framework, leading to more accurate camera pose estimation. However, due to the inherent modality discrepancy between images and sparse LiDAR point clouds, reliable cross-modal correspondence learning remains challenging. To address this issue, we propose Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration (DPA-I2P). Unlike naive depth or feature concatenation, Ray-Conditioned Metric Depth Encoding (RMDE) and Projection-Consistent Vision Lifting (PVL) exploit depth and visual cues in a structured, geometry-aware manner. In addition, Cross-Modal Query Pruning (CQP) suppresses unreliable queries during early refinement to improve matching stability. Experiments on KITTI and nuScenes demonstrate the effectiveness of the proposed method. On KITTI, DPA-I2P reduces RTE and RRE by 45.0% and 55.6% over the strongest implicit baseline, respectively. On nuScenes, DPA-I2P also improves registration accuracy over the evaluated baselines, suggesting better transferability to different driving scenes.

Figures & tables

Fig. 1: Motivation of DPA-I2P. Existing implicit correspondence learning methods suffer from weak cross-modal feature consistency. DPA-I2P integrates depth and projection cues to construct a discriminative cross-modal feature space for more reliable correspondence exploration.

Fig. 2: Overview of DPA-I2P. Given an image and point cloud, the framework first integrates metric depth cues with a coarse projection-consistent prior via RMDE and PVL for feature enhancement, together with an additional CQP module for query refinement. It then performs implicit correspondence learning for final pose regression.

Methods	KITTI			nuScenes
Methods	RTE $\downarrow$	RRE $\downarrow$	Acc. $\uparrow$	RTE $\downarrow$	RRE $\downarrow$	Acc. $\uparrow$
Grid Cls.+PnP [ 10 ]	3.64 $\pm$ 3.46	19.19 $\pm$ 28.96	11.22	3.02 $\pm$ 2.40	12.66 $\pm$ 21.01	2.45
DeepI2P(3D) [ 10 ]	4.06 $\pm$ 3.54	24.73 $\pm$ 31.69	3.77	2.88 $\pm$ 2.12	20.65 $\pm$ 12.24	2.26
DeepI2P(2D) [ 10 ]	3.59 $\pm$ 3.21	11.66 $\pm$ 18.16	25.95	2.78 $\pm$ 1.99	4.80 $\pm$ 6.21	38.10
CorrI2P [ 8 ]	3.78 $\pm$ 65.16	5.89 $\pm$ 20.34	72.42	3.04 $\pm$ 60.76	3.73 $\pm$ 9.03	49.00
VP2P-Match [ 9 ]	0.75 $\pm$ 1.13	3.29 $\pm$ 7.99	83.04	0.89 $\pm$ 1.44	2.15 $\pm$ 7.03	88.33

TABLE I: Registration accuracy on the KITTI and nuScenes datasets. Lower is better for RTE and RRE, higher is better for Acc.

Panel of Figure 4 — Fig. 4: Qualitative comparison of image-to-point cloud registration results on the nuScenes dataset.

Panel of Figure 5 — Fig. 5: Visual illustration of 2D-3D correspondences and registration accuracy. Green lines represent correct correspondences.

Methods	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
w/o RMDE	0.12 $\pm$ 0.11	0.58 $\pm$ 0.72	99.61
w/o PVL	0.12 $\pm$ 0.14	0.74 $\pm$ 0.87	99.46
w/o CQP	0.18 $\pm$ 0.22	0.65 $\pm$ 0.51	98.82
DPA-I2P (Full)	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70

TABLE II: The effect of each design in our framework.

RMDE variant	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
No depth prior	0.12 $\pm$ 0.11	0.58 $\pm$ 0.72	99.61
Raw depth concatenation	0.12 $\pm$ 0.12	0.57 $\pm$ 0.70	99.63
Depth + ray direction	0.12 $\pm$ 0.12	0.57 $\pm$ 0.69	99.64
Surface-point encoding	0.12 $\pm$ 0.12	0.56 $\pm$ 0.69	99.66
Uniform ray sampling w/o confidence	0.11 $\pm$ 0.12	0.56 $\pm$ 0.68	99.67
Full RMDE	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70

TABLE III: Ablation study of RMDE components on KITTI. All variants use the same frozen UniDepthV2 prior when depth is involved, and the remaining framework is kept unchanged.

Query strategy	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
No query pruning	0.18 $\pm$ 0.22	0.65 $\pm$ 0.51	98.82
Early-only	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70
All-stage	0.14 $\pm$ 0.43	0.66 $\pm$ 0.87	98.90

TABLE IV: Ablation studies on query pruning strategies on the KITTI dataset.

Methods	Network size (MB)	Mem. (GB)	Inference (ms)
Grid Cls. [ 10 ]	100.75	2.39	11.20
DeepI2P [ 10 ]	100.12	2.01	7.55
CorrI2P [ 8 ]	141.07	2.88	13.75
VP2P-Match [ 9 ]	146.42	17.17	30.63
ICLI2P [ 12 ]	175.92	10.74	35.12
DPA-I2P (Ours)	179.93	11.15	36.81

TABLE V: Efficiency comparison on the KITTI dataset. Network size, GPU memory, and inference time are measured on an NVIDIA GeForce RTX 4090 GPU. The reported inference time includes feature extraction, correspondence refinement, and differentiable pose solving. Offline metric depth pre-computation is not included.

Point Cloud Registration

Point cloud registration aligns 3D point sets by estimating the transformations and correspondences that make their geometry consistent.

See all 110 papers

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration
Sep 16, 2026Zhixin Cheng, Jiacheng Deng, Xiaotian Yin +3Masked Autoencoders Point Cloud Registration
School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China
DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers
Sep 11, 2026Lanke Frank Tarimo Fu, Maurice Fallon3D ViTs Camera Pose Estimation
University of Oxford, Oxford OX2 6NN, United Kingdom
Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs
Mar 30, 2026Pei An, Junfeng Ding, Jiaqi Yang +32D-3D Image Registration Point Cloud Registration
Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Huazhong Agricultural University, China +1

Sep 16, 2026J/K move · Enter open · S save

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

Zhixin Cheng, Jiacheng Deng, Xiaotian Yin, Baoqun Yin+2

School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Masked Autoencoders Point Cloud Registration Cross-Modal Feature Matching Cross-Modal Image Registration 2D-3D Image Registration

Open paper AskPDF arXiv

Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.

Abstract

Point Cloud Registration

Explore similar work

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs