RegFormer++: An Efficient Large-Scale 3D LiDAR Point Registration Network with Projection-Aware 2D Transformer
Authors: Jiuming Liu, Guangming Wang, Zhe Liu, Chaokang Jiang, Haoang Li, Mengmeng Liu, Tianchen Deng, Marc Pollefeys, Michael Ying Yang, Hesheng Wang
Organizations: School of Automation and Intelligent Sensing, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Jiao Tong University, Shanghai 200240, China. · Department of Engineering, University of Cambridge, UK. · China University of Mining and Technology. · Thrust of Robotics and Autonomous Systems and the Thrust of Intelligent Transportation, Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China. · University of Twente. · both ETH Zurich and Microsoft. · University of Bath.
Although point cloud registration has achieved remarkable advances in object-level and indoor scenes, large-scale LiDAR registration methods has been rarely explored before. Challenges mainly arise from the huge point scale, complex point distribution, and numerous outliers within outdoor LiDAR scans. In addition, most existing registration works generally adopt a two-stage paradigm: They first find correspondences by extracting discriminative local descriptors and then leverage robust estimators (e.g. RANSAC) to filter outliers, which are highly dependent on well-designed descriptors and post-processing choices. To address these problems, we propose a novel end-to-end differential transformer network, termed RegFormer++, for large-scale point cloud alignment without requiring any further post-processing. Specifically, a hierarchical projection-aware 2D transformer with linear complexity is proposed to project raw LiDAR points onto a cylindrical surface and extract global point features, which can improve resilience to outliers due to long-range dependencies. Because we fill original 3D coordinates into 2D projected positions, our designed transformer can benefit from both high efficiency in 2D processing and accuracy from 3D geometric information. Furthermore, to effectively reduce wrong point matching, a Bijective Association Transformer (BAT) is designed, combining both cross attention and all-to-all point gathering. To improve training stability and robustness, a feature-transformed optimal transport module is also designed for regressing the final pose transformation. Extensive experiments on KITTI, NuScenes, and Argoverse datasets demonstrate that our model achieves state-of-the-art performance in terms of both accuracy and efficiency.
Figures & tables
Figure 1: Comparison with previous point cloud registration works. Previous methods (A) extract local descriptors and establish explicit correspondences, while our RegFormer++ rely on globally-aware features and learn cross-frame association by cross attention requiring no correspondences. Furthermore, prior literature resorts to the RANSAC [ 24 ] algorithm for outlier filtering. In contrast, our method leverage attention mechanism to softly remove outliers.
Figure 2: The overall architecture of RegFormer++. We first project point clouds onto a 2D surface and feed its patches into three cascaded feature extraction transformers. For the cross-frame association, we design a Bijective Association Transformer module which includes a cross attention module and an all-to-all point gathering. Finally, transformations are refined iteratively.
Figure 3: Cylindrical projection. We project 3D point clouds onto a 2D surface and fill each pixel with its raw x,y,z coordinates. A projection mask is also proposed to remove invalid positions.
Figure 4: Visualization of kernel-based efficient patch embedding. For each center point, its surrounding neighbor points within a predefined 2D kernel are indexed and grouped. Then, outliers that are harmful to registration are filtered. Above grouping and filtering in 2D space actually happen in 3D space based on the cylindrical projection process.
Figure 5: All-to-all point gathering. After the cross-attention mechanism leveraged for preliminary information exchange between two frames, geometric characteristics of conditioned features F~3S,F~3T are fully considered to generate the initial motion embeddings. The initial embeddings include coordinate information, point and neighbor similarity features, and content features.
Module
Layer type
N
Sample rate
Channels
MLP width
Heads
Dropout
Point Swin Transformer (stage 1)
2
[H/4×W/8]
16
[16,64,16]
2
0.1
Point Swin Transformer (stage 2)
2
[H/8×W/16]
32
[32,128,32]
4
0.1
Feature Extraction Transformer
Point Swin Transformer (stage 3)
6
[H/16×W/32]
64
[64,256,64]
8
0.1
self-attention layer
6
—
64
[64,256,64]
8
0.1
Bijective Association Transformer
cross-attention layer
6
—
64
[64,256,64]
8
0.1
Table 1: Detailed network settings in transformer layers. N denotes the number of attention blocks. MLP width indicates the expansion layer of MLP in feed-forward networks.
Module
Layer type
K
Sample rate
MLP width
All-to-all point gathering
112
1
[128,64,64], [128,64]
Shared MLP for ME3
—
1
[128,64]
FC1 for q3 , FC2 for t3
—
1
[4], [3]
Estimation of Initial Transformation
Attentive cost volume
4, 6
1
[128,64,64], [128,64]
Set upconv
8
2×2
[128,64], [64]
Shared MLP for ME2
—
1
[128,64]
Table 2: Detailed network parameters in the initial pose estimation and iterative refinement layers. K points are selected in the K Nearest Neighbors (KNN) of the all-to-all point gathering layer, set upconv layer, and attentive cost volume layer. MLP width means the number of output channels for each layer of MLP.
RTE(m)
RRE(°)
Method
AVG
STD
AVG
STD
RR(%)
Time(ms)/Points
NT(ms)
FGR [ 112 ]
0.93
0.59
0.96
0.81
39.4%
506.1/11445
44.22
RANSAC [ 24 ]
0.13
0.07
0.54
0.40
91.9%
549.6/16384
33.54
DCP [ 88 ]
1.03
0.51
2.07
1.19
47.3%
46.4/1024
45.31
IDAM [ 37 ]
0.66
0.48
1.06
0.94
70.9%
33.4/4096
8.15
FMR [ 32 ]
0.66
0.42
1.49
0.85
90.6%
85.5/12000
7.13
Table 3: Comparison results with state-of-the-arts on the KITTI dataset. Input point pairs are 10-frame away prepared by [ 57 ] . The best performance is highlighted in bold. Registration Recall (RR) is defined as the success ratio where RRE < 5°and RTE < 2m. ‘NT’ means normalized time per thousand points.
Method
Estimator
RTE(cm)
RRE(°)
RR(%)
Time(s)
3DFeat-Net [ 95 ]
RANSAC
25.9
0.25
96.0%
3.4
FCGF [ 18 ]
RANSAC
9.5
0.30
96.6%
3.4
D3Feat [ 7 ]
RANSAC
7.2
0.30
99.8%
3.1
Predator [ 30 ]
RANSAC
6.8
0.27
99.8%
5.2
CoFiNet [ 99 ]
RANSAC
8.5
0.41
99.8%
1.9
SpinNet [ 4 ]
RANSAC
9.9
0.47
99.1%
60.6
Table 4: Comparison results with methods using traditional estimators. Methods are classified into two main categories: RANSAC-based and Graph-based. † indicates methods using optimization algorithm. Other methods including ours are learning-based. Input point pairs are at least 10m away as in [ 71 ] The best performances are respectively highlighted in bold. RR is defined as the success ratio where RRE < 5°and RTE < 2m.
Figure 6: Registration recall with different RRE and RTE thresholds on the KITTI dataset.
Method
RTE(m)
RRE(°)
Recall(%)
Time/Points
NT(ms)
FGR [ 112 ]
0.71
1.01
32.2%
284.6/11445
24.87
RANSAC [ 24 ]
0.21
0.74
60.9%
268.2/8192
32.74
DCP [ 88 ]
1.09
2.07
58.6%
45.5/1024
44.43
IDAM [ 37 ]
0.47
0.79
88.0%
32.6/4096
7.96
FMR [ 32 ]
0.60
1.61
92.1%
61.1/12000
5.09
DGR [ 17 ]
0.21
0.48
98.4%
523.0/8192
63.84
Table 5: Comparison results with state-of-the-arts on the NuScenes dataset. Input point pairs are 10-frame away prepared by [ 57 ] . The best performance is highlighted in bold. Registration Recall (RR) is defined as the success ratio where RRE < 5°and RTE < 2m. ‘NT’ means normalized time per thousand points.
Figure 7: Registration results on KITTI and Nuscenes datasets. Point clouds colored red and green indicate the input source and target point clouds. Transformed source points by the estimated pose of HRegNet and ours are colored pink and blue respectively. The ground truth is colored purple. Our RegFormer++ can align low-overlap input points even with large translations (b) and (d) or large rotations (a) and (c).
Figure 8: Visualization of registration errors. Point clouds colored yellow and blue indicate transformed source points by the ground truth (GT) and our estimated pose. Registration errors are visualized by a red vector pointing from estimated points to the GT ones.
RTE(m)
RRE(°)
Method
AVG
STD
AVG
STD
RR(%)
HRegNet [ 57 ]
1.12
0.53
1.04
1.12
25.7%
RegFormer++
0.01
0.01
0.11
0.27
85.5%
Table 6: Comparison results on the Argoverse dataset [ 12 ] . We follow the standard dataset settings to evaluate the performance and train both HRegNet and our method from scratch.
Gazebo
Wood
Method
Summer
Winter
Autumn
Summer
Avg.
FPFH [ 74 ]
38.6%
14.2%
14.8%
20.8%
22.1%
SHOT [ 80 ]
73.9%
45.7%
60.9%
64.0%
61.1%
3DMatch [ 102 ]
22.8%
8.3%
13.9%
22.4%
16.9%
CGF [ 35 ]
37.5%
13.8%
10.4%
19.2%
20.2%
PerfectMatch [ 29 ]
91.3%
84.1%
67.8%
72.8%
79.0%
Table 7: Generalization results on the ETH dataset. All models are pre-trained on other datasets and then directly evaluated on the unseen ETH dataset [ 68 ] .
00
01
02
03
04
05
06
07 †
08 †
09 †
10 †
Mean on 07-10
Method
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
trel
rrel
ICP-po2po
6.88
2.99
11.21
2.58
8.21
3.39
11.07
5.05
6.64
4.02
3.97
1.93
1.95
1.59
5.17
3.35
10.04
4.93
6.93
2.89
8.91
4.74
7.36
3.41
ICP-po2pl
3.80
1.73
13.53
2.58
9.00
2.74
2.72
1.63
2.96
2.58
2.29
1.08
1.77
1.00
1.55
1.42
4.42
2.14
3.95
1.71
6.13
2.60
4.74
1.93
GICP [ 75 ]
1.29
0.64
4.39
0.91
2.53
0.77
1.68
1.08
3.76
1.07
1.02
0.54
0.92
0.46
0.64
0.45
1.58
0.75
1.97
0.77
1.31
0.62
1.92
0.73
CLS [ 81 ]
2.11
0.95
4.22
1.05
2.29
0.86
1.63
1.09
1.59
0.71
1.98
0.92
0.92
0.46
1.04
0.73
2.14
1.05
1.95
0.92
3.46
1.28
2.15
1.00
LOAM ∗ [ 105 ]
15.99
6.25
3.43
0.93
9.40
3.68
18.18
9.91
9.59
4.57
9.16
4.10
8.91
4.63
10.87
6.76
12.72
5.77
8.10
4.30
12.67
8.79
10.82
5.43
Table 8: Comparison with the state-of-the-art on the LiDAR odometry task. trel , rrel indicate the average translation RMSE (%) and rotation RMSE ( ∘ /100m) respectively on all subsequences in the length of 100,200,...,800m . ‘ ∗ ’ means that only odometry results without mapping are listed for a fair comparison. ‘ † ’ means the evaluation sequences. ‘ NG ’ means results are not given. We re-evaluate the results of PWCLO [ 86 ] and obtain the same results as in [ 1 ] . The best results for each sequence are bold , and the second best is underlined .
Figure 9: Visualization of 2D and 3D trajectories of LiDAR odometry. The blue and red lines are respectively our estimated and the ground truth trajectory.
Figure 10: Comparison results with traditional LiDAR odometry LOAM.
Model
RTE(m)
RRE(°)
Recall(%)
(a) Transformation q3 , t3
0.79 ± 0.47
1.46 ± 0.98
74.9%
(b) Transformation q2 , t2
0.53 ± 0.37
1.37 ± 0.91
88.9%
(c) Transformation q1 , t1
0.20 ± 0.19
0.57 ± 0.50
99.7%
(d) Transformation q0 , t0
0.04 ± 0.06
0.14 ± 0.13
100%
Table 9: Ablation studies of the hierarchical architecture.
Model
RTE(m)
RRE(°)
Recall(%)
(a) w/o projection mask
0.14 ± 0.13
0.41 ± 0.43
99.5%
(b) replaced by 2D CNN
0.51 ± 0.39
0.76 ± 0.73
83.6%
(c) replaced by PointNet++ [ 70 ]
0.21 ± 0.20
0.47 ± 0.51
93.6%
(d) replaced by Point transformer [ 110 ]
0.11 ± 0.14
0.24 ± 0.26
99.6%
Ours (Full)
0.04 ± 0.06
0.14 ± 0.13
100%
Table 10: Ablation studies of components in Point Swin Transformer (PST).
Model
RTE(m)
RRE(°)
Recall(%)
(a) w/o cross attention
0.09 ± 0.11
0.20 ± 0.29
98.8%
(b) w/o all-to-all point gathering
0.86 ± 0.39
1.69 ± 1.37
67.2%
(c) replaced with cost volume in [ 86 ]
0.69 ± 0.33
0.98 ± 0.82
71.2%
(d) replaced with cost volume in [ 83 ]
0.15 ± 0.17
0.21 ± 0.25
94.8%
Ours (Full)
0.04 ± 0.06
0.14 ± 0.13
100%
Table 11: Ablation studies of components in Bijective Association Transformer (BAT) module.
SF
FE trans
RTE(m)
RRE(°)
Recall(%)
0.07 ± 0.12
0.20 ± 0.21
99.7%
✓
0.06 ± 0.10
0.19 ± 0.17
99.8%
✓
0.05 ± 0.06
0.18 ± 0.15
99.9%
✓
✓
0.04 ± 0.06
0.14 ± 0.13
100%
Table 12: Ablation studies of the transformed features in the optimal transport (OT) layer.
Figure 11: Visualization of attention weights. We give two samples here, where the first two rows respectively represent corresponding pictures and projected point clouds. Attention weights are visualized in the last row. Dynamic objects (red box) have lower attention weights, and static objects (green box) have higher weights.
Figure 12: Visualization of the cross attention mechanism in BAT. Points A and B are corresponding points respectively in the source and target frame. Point C is the transformed position of A by conditioned features after cross-attention block.
Learning-based global point cloud registration has achieved remarkable progress, yet its reliance on geometric representations makes existing methods sensitive to variations in point density, scan pattern, viewpoint, and sensor characteristics. We propose CVSD-Reg, a robust global LiDAR registration framework that distills visual semantic priors from a vision foundation model into LiDAR representations. In Stage 1, a Point Transformer V3 student learns from a frozen DINOv2 teacher through contrastive distillation and spherical-manifold alignment, which preserves the hyperspherical geometry of the teacher embedding space. Self-supervised InfoNCE consistency and soft SE(3) invariance further encourage viewpoint-robust descriptors. In Stage 2, the distilled representation is adapted to registration through correspondence learning, density-aware point-dropout augmentation, and end-to-end pose optimization. With a single checkpoint, CVSD-Reg generalizes to both single-sensor and zero-shot cross-sensor scenarios without sensor-specific adaptation and remains entirely camera-free at inference. On KITTI, nuScenes, and HeLiPR, CVSD-Reg achieves strict success rate ([email protected],m/1∘) of 97.7%, 99.0%, and 99.3%, respectively, including 97.3% on sparse 16-beam Velodyne scans. It outperforms state-of-the-art geometric registration methods by up to 44.0 percentage points without requiring camera inputs or post-hoc ICP refinement.
We present DRS-VPT, a feed-forward transformer architecture for foundational image-to-scan registration. Given query images and a reference 3D point cloud, the model predicts the scan pose and point map alongside the poses and point maps of each camera, all expressed in the first camera's frame. It additionally predicts a coarse-to- fine pyramid of per-point and per-pixel features for direct reprojective alignment of the scan to the first image. This formulation unifies downstream tasks such as camera-LiDAR calibration in autonomous driving and indoor camera-to-map relocalization. A single DRS-VPT model achieves state-of-the-art performance for image-to-LiDAR registration in autonomous driving, competitive indoor relocalization without training map-specific weights, and strong zero-shot transfer to unseen environments. We also show qualitatively that the model learns complex scan-to-image projection properties such as occlusion of back-facing points.
Lanke Frank Tarimo Fu, Maurice Fallon
University of Oxford, Oxford OX2 6NN, United Kingdom
Unsupervised registration of large-scale LiDAR point clouds remains challenging due to the geometric ambiguity inherent in outdoor scenes, which degrades pseudo-label quality and leads to suboptimal convergence, particularly for sparse, low-resolution scans such as those from nuScenes. We reveal that registration models intrinsically encode semantic awareness that strongly correlates with registration accuracy, albeit without explicit semantic supervision. However, this native awareness is fragile: noisy supervision arising from geometric ambiguity in unsupervised settings rapidly erodes the learned semantic structure, causing performance collapse. To this end, we propose CAESAR, a teacher-student framework guided by an off-the-shelf 3D segmentation model exclusively during training. We observe that potential inlier matches are often buried just beneath a few spurious neighbors in the noisy feature space, motivating Dual-Cue Guided Re-Matching to recover them through reselection rather than simply rejecting. Building on this, a train-only Semantic-Geometric Label Mining performs lightweight, batch-specific teacher refinement and mines reliable pseudo-labels under semantic guidance. We further introduce Semantic Predictive Distillation to consolidate the student's semantic awareness in the feature space. Extensive experiments on KITTI and nuScenes demonstrate state-of-the-art performance, with pronounced gains on the challenging nuScenes benchmark. Crucially, CAESAR incurs zero inference overhead and requires no semantic annotations on the registration data. Code will be released.
Kezheng Xiong, Shiyun Xu, Sheng Ao +3
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China · Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China