DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

Wenxin Zhang; Hang Li; Zhiwei Xu; Qiankun Dong; Gang Wang; Tao Li

cs.CVAug 27, 2026

DPA-I2P: Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration in Autonomous Driving

Authors: Wenxin Zhang, Hang Li, Zhiwei Xu, Qiankun Dong, Gang Wang, Tao Li

Organizations: Nankai University, Tianjin, China · Haihe Lab of ITAI, Tianjin 300459, China

Abstract

Image-to-Point Cloud Registration aims to estimate the camera pose of a given image within a 3D scene point cloud, which is a fundamental task in autonomous driving and large-scale outdoor localization. Recent implicit correspondence learning methods have improved registration performance by learning cross-modal alignment in an end-to-end framework, leading to more accurate camera pose estimation. However, due to the inherent modality discrepancy between images and sparse LiDAR point clouds, reliable cross-modal correspondence learning remains challenging. To address this issue, we propose Depth-Guided Projective Alignment for Image-to-Point-Cloud Registration (DPA-I2P). Unlike naive depth or feature concatenation, Ray-Conditioned Metric Depth Encoding (RMDE) and Projection-Consistent Vision Lifting (PVL) exploit depth and visual cues in a structured, geometry-aware manner. In addition, Cross-Modal Query Pruning (CQP) suppresses unreliable queries during early refinement to improve matching stability. Experiments on KITTI and nuScenes demonstrate the effectiveness of the proposed method. On KITTI, DPA-I2P reduces RTE and RRE by 45.0% and 55.6% over the strongest implicit baseline, respectively. On nuScenes, DPA-I2P also improves registration accuracy over the evaluated baselines, suggesting better transferability to different driving scenes.

Figures & tables

Fig. 1: Motivation of DPA-I2P. Existing implicit correspondence learning methods suffer from weak cross-modal feature consistency. DPA-I2P integrates depth and projection cues to construct a discriminative cross-modal feature space for more reliable correspondence exploration.

Fig. 2: Overview of DPA-I2P. Given an image and point cloud, the framework first integrates metric depth cues with a coarse projection-consistent prior via RMDE and PVL for feature enhancement, together with an additional CQP module for query refinement. It then performs implicit correspondence learning for final pose regression.

Methods	KITTI			nuScenes
Methods	RTE $\downarrow$	RRE $\downarrow$	Acc. $\uparrow$	RTE $\downarrow$	RRE $\downarrow$	Acc. $\uparrow$
Grid Cls.+PnP [ 10 ]	3.64 $\pm$ 3.46	19.19 $\pm$ 28.96	11.22	3.02 $\pm$ 2.40	12.66 $\pm$ 21.01	2.45
DeepI2P(3D) [ 10 ]	4.06 $\pm$ 3.54	24.73 $\pm$ 31.69	3.77	2.88 $\pm$ 2.12	20.65 $\pm$ 12.24	2.26
DeepI2P(2D) [ 10 ]	3.59 $\pm$ 3.21	11.66 $\pm$ 18.16	25.95	2.78 $\pm$ 1.99	4.80 $\pm$ 6.21	38.10
CorrI2P [ 8 ]	3.78 $\pm$ 65.16	5.89 $\pm$ 20.34	72.42	3.04 $\pm$ 60.76	3.73 $\pm$ 9.03	49.00
VP2P-Match [ 9 ]	0.75 $\pm$ 1.13	3.29 $\pm$ 7.99	83.04	0.89 $\pm$ 1.44	2.15 $\pm$ 7.03	88.33

TABLE I: Registration accuracy on the KITTI and nuScenes datasets. Lower is better for RTE and RRE, higher is better for Acc.

Panel of Figure 4 — Fig. 4: Qualitative comparison of image-to-point cloud registration results on the nuScenes dataset.

Panel of Figure 5 — Fig. 5: Visual illustration of 2D-3D correspondences and registration accuracy. Green lines represent correct correspondences.

Methods	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
w/o RMDE	0.12 $\pm$ 0.11	0.58 $\pm$ 0.72	99.61
w/o PVL	0.12 $\pm$ 0.14	0.74 $\pm$ 0.87	99.46
w/o CQP	0.18 $\pm$ 0.22	0.65 $\pm$ 0.51	98.82
DPA-I2P (Full)	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70

TABLE II: The effect of each design in our framework.

RMDE variant	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
No depth prior	0.12 $\pm$ 0.11	0.58 $\pm$ 0.72	99.61
Raw depth concatenation	0.12 $\pm$ 0.12	0.57 $\pm$ 0.70	99.63
Depth + ray direction	0.12 $\pm$ 0.12	0.57 $\pm$ 0.69	99.64
Surface-point encoding	0.12 $\pm$ 0.12	0.56 $\pm$ 0.69	99.66
Uniform ray sampling w/o confidence	0.11 $\pm$ 0.12	0.56 $\pm$ 0.68	99.67
Full RMDE	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70

TABLE III: Ablation study of RMDE components on KITTI. All variants use the same frozen UniDepthV2 prior when depth is involved, and the remaining framework is kept unchanged.

Query strategy	RTE (m) $\downarrow$	RRE ( ∘ ) $\downarrow$	Acc. (%) $\uparrow$
No query pruning	0.18 $\pm$ 0.22	0.65 $\pm$ 0.51	98.82
Early-only	0.11 $\pm$ 0.12	0.55 $\pm$ 0.67	99.70
All-stage	0.14 $\pm$ 0.43	0.66 $\pm$ 0.87	98.90

TABLE IV: Ablation studies on query pruning strategies on the KITTI dataset.

Methods	Network size (MB)	Mem. (GB)	Inference (ms)
Grid Cls. [ 10 ]	100.75	2.39	11.20
DeepI2P [ 10 ]	100.12	2.01	7.55
CorrI2P [ 8 ]	141.07	2.88	13.75
VP2P-Match [ 9 ]	146.42	17.17	30.63
ICLI2P [ 12 ]	175.92	10.74	35.12
DPA-I2P (Ours)	179.93	11.15	36.81

TABLE V: Efficiency comparison on the KITTI dataset. Network size, GPU memory, and inference time are measured on an NVIDIA GeForce RTX 4090 GPU. The reported inference time includes feature extraction, correspondence refinement, and differentiable pose solving. Offline metric depth pre-computation is not included.

Point Cloud Registration

Point cloud registration aligns 3D point sets by estimating the transformations and correspondences that make their geometry consistent.

See all 110 papers

Sep 16, 2026

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.

Zhixin Cheng, Jiacheng Deng, Xiaotian Yin +3

School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Masked Autoencoders Point Cloud Registration Cross-Modal Feature Matching Cross-Modal Image Registration 2D-3D Image Registration

Ask PDF

Sep 11, 2026

DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

We present DRS-VPT, a feed-forward transformer architecture for foundational image-to-scan registration. Given query images and a reference 3D point cloud, the model predicts the scan pose and point map alongside the poses and point maps of each camera, all expressed in the first camera's frame. It additionally predicts a coarse-to- fine pyramid of per-point and per-pixel features for direct reprojective alignment of the scan to the first image. This formulation unifies downstream tasks such as camera-LiDAR calibration in autonomous driving and indoor camera-to-map relocalization. A single DRS-VPT model achieves state-of-the-art performance for image-to-LiDAR registration in autonomous driving, competitive indoor relocalization without training map-specific weights, and strong zero-shot transfer to unseen environments. We also show qualitatively that the model learns complex scan-to-image projection properties such as occlusion of back-facing points.

Lanke Frank Tarimo Fu, Maurice Fallon

University of Oxford, Oxford OX2 6NN, United Kingdom

3D ViTs Camera Pose Estimation Camera-LiDAR Fusion Point Cloud Registration Cross-Modal Image Registration 2D-3D Image Registration

Ask PDF

Mar 30, 2026

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

Image-to-point-cloud (I2P) registration aims to align 2D images with 3D point clouds by establishing reliable 2D-3D correspondences. The drastic modality gap between images and point clouds makes it challenging to learn features that are both discriminative and generalizable, leading to severe performance drops in unseen scenarios. We address this challenge by introducing a heterogeneous graph that enables refining both cross-modal features and correspondences within a unified architecture. The proposed graph represents a mapping between segmented 2D and 3D regions, which enhances cross-modal feature interaction and thus improves feature discriminability. In addition, modeling the consistency among vertices and edges within the graph enables pruning of unreliable correspondences. Building on these insights, we propose a heterogeneous graph embedded I2P registration method, termed Hg-I2P. It learns a heterogeneous graph by mining multi-path feature relationships, adapts features under the guidance of heterogeneous edges, and prunes correspondences using graph-based projection consistency. Experiments on six indoor and outdoor benchmarks under cross-domain setups demonstrate that Hg-I2P significantly outperforms existing methods in both generalization and accuracy. Code is released on https://github.com/anpei96/hg-i2p-demo.

Pei An, Junfeng Ding, Jiaqi Yang +3

Huazhong University of Science and Technology, China · Northwestern Polytechnical University, China · Huazhong Agricultural University, China +1

2D-3D Image Registration Point Cloud Registration Cross-Modal Image Registration Cross-Modal Feature Matching Domain Generalization

Ask PDF

Abstract

Point Cloud Registration

Explore similar work

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs