Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
Authors: Ni Ou, Zhuo Chen, Xinru Zhang, Junzheng Wang
Organizations: Ni Ou, Junzheng Wang are with the School of Automation, Beijing Institute of Technology, Beijing, 100081, China. · Zhuo Chen is with the Robot Perception Lab, Centre for Robotics Research, Department of Engineering, King’s College London, London WC2R 2LS, United Kingdom. · Xinru Zhang is with the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, 100081, China.
Accurate camera-LiDAR fusion relies on precise extrinsic calibration, which fundamentally depends on establishing reliable cross-modal correspondences under potentially large misalignments. Existing learning-based methods typically project LiDAR points into depth maps for feature fusion, which distorts 3D geometry and degrades performance when the extrinsic initialization is far from the ground truth. To address this issue, we propose an extrinsic-aware cross-attention framework that directly aligns image patches and LiDAR point groups in their native domains. The proposed attention mechanism explicitly injects extrinsic parameter hypotheses into the correspondence modeling process, enabling geometry-consistent cross-modal interaction without relying on projected 2D depth maps. Extensive experiments on the KITTI and nuScenes benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in both accuracy and robustness. Under large extrinsic perturbations, our approach achieves accurate calibration in 88% of KITTI cases and 99% of nuScenes cases, substantially surpassing the second-best baseline. We have open sourced our code on https://github.com/gitouni/ProjFusion to benefit the community.
Figures & tables
Fig. 1: Illustration of the problem caused by miscalibrated depth maps and our proposed method. (a) Projecting LiDAR points onto the image plane produces incomplete and distorted object structures. (b) Image patches and point groups are encoded separately and fused through extrinsic-aware cross-attention, enabling structure-preserving cross-modal feature interaction.
Fig. 2: Overall framework of our method. FI and FP denote the sequences of image and point features, respectively, while CI and CP represent their corresponding positional embeddings. ξrot and ξtsl are the rotational and translational components of ξ defined in Eq. 1 .
Dataset
Range
Method
Rotation ( ∘ ) ↓
Translation (cm) ↓
Success Rate (%) ↑
RMSE
MAE
RMSE
MAE
L1
L2
KITTI [ 22 ]
15∘15cm
CoFiI2P [ 15 ]
4.613 ± 3.071
2.066 ± 1.228
134.8 ± 75.09
62.64 ± 32.60
0.00%
0.04%
DirectCalib [ 40 ]
13.09 ± 23.55
6.315 ± 11.38
194.9 ± 1967
98.42 ± 1099
0.26%
1.54%
CalibAnything [ 37 ]
18.27 ± 14.78
9.439 ± 7.889
27.25 ± 15.11
13.79 ± 7.742
0.00%
1.90%
CalibNet [ 16 ]
2.019 ± 2.104
0.764 ± 0.726
5.798 ± 3.598
2.836 ± 1.783
8.00%
32.28%
RGGNet [ 17 ]
3.878 ± 3.380
1.421 ± 1.199
6.069 ± 4.037
2.971 ± 2.016
5.40%
18.56%
TABLE I: Calibration Results on KITTI and nuScenes Datasets (Mean ± Standard Deviation)
Fig. 3: LiDAR projection maps generated using the predicted extrinsic matrix TCLpred from different methods across urban, rural, highway, and nighttime scenes. Selected regions of interest (ROIs) are zoomed in for clearer visualization.
Index
Dual Branches
Projection Margin
Encoding Space
Positional Embedding
Image Encoder
Rotation RMSE ( ∘ ) ↓
Translation RMSE (cm) ↓
Success Rate (%) ↑
L1
L2
1
✓
✗
3D
concatenation
DINOv2
1.534 ± 1.304
3.002 ± 2.861
22.60%
73.64%
2
✓
✓
2D
harmonic ( nh=6 )
DINOv2
0.244 ± 0.279
3.429 ± 2.490
39.84%
83.12%
3
✓
✗
2D
harmonic ( nh=6 )
DINOv2
1.111 ± 1.055
15.56 ± 8.714
0.60%
6.12%
4
✓
✓
3D
harmonic ( nh=0 )
DINOv2
1.297 ± 1.197
3.199 ± 1.591
20.12%
76.88%
5
✓
✓
3D
harmonic ( nh=2 )
DINOv2
0.728 ± 0.893
2.845 ± 3.767
40.76%
89.16%
TABLE II: Ablation on KITTI at 10∘50cm (Mean ± Standard Deviation)
Accurate camera-LiDAR calibration is a prerequisite for robust multi-modal perception in robotics. Recent target-less approaches based on deep point correspondences achieve remarkable performance for extrinsic calibration but assume rectified images with known intrinsics. In this work, we overcome this limitation and present the first fully target-less pipeline that jointly estimates camera intrinsics (pinhole model with radial-tangential distortion) and camera-LiDAR extrinsics with deep pixel-point correspondences. Our approach extends deep correspondence-based calibration by (i) automatic intrinsic initialization via structure-from-motion, (ii) generalizing camera-LiDAR matching to raw images with unknown intrinsics including distortion, and (iii) tightly coupling correspondence estimation with joint nonlinear optimization over both intrinsics and extrinsics. We evaluate our method on the KITTI dataset with unseen camera-LiDAR pairs and demonstrate that joint calibration achieves improved extrinsic accuracy while additionally recovering accurate intrinsics.
Simon Bultmann, Daniele Cattaneo, Abhinav Valada
Department of Computer Science, University of Freiburg, Germany
Target-based LiDAR-camera extrinsic calibration is a prerequisite for multi-sensor fusion in robotics. However, in the widely adopted four-hole pipeline, \rt{calibration accuracy is limited by hole-center extraction in the LiDAR side, where sparse angular coverage and mixed-pixel returns displace the estimated centers}. This paper presents P2Calib, which exploits pattern priors, geometric constraints specified by the CAD model of the target board, to improve calibration accuracy. First, we incorporate the known hole radius as a fitting constraint to prevent center estimates from degrading under sparse angular coverage. Building on the improved hole estimates, we further enforce the rigid rectangular layout of the four holes as a global consistency constraint to correct residual errors across holes. Both priors are integrated into an \rt{interactive tool that runs the pipeline from target detection to the final extrinsic}. Experiments on simulated and real datasets show that P2Calib reduces the joint registration residual by 90% and 82% and the held-out reprojection error by 96% and 77% over the baseline. \rt{We will release the code and data\codelink{} to facilitate future research.
Xiangcheng Hu
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong SAR, China
Most learning-based camera-LiDAR calibration methods treat each camera-LiDAR pair independently, ignoring the rigid geometric coupling in multi-camera platforms. As a result, per-camera estimates may be individually accurate yet inconsistent at the system level. We present a two-stage framework for joint multi-camera LiDAR extrinsic calibration that combines learned pairwise matching with geometric refinement. First, CMRNext is applied independently to each camera to produce initial extrinsic estimates and dense 2D-3D correspondences. These predictions are then jointly refined through a multi-frame bundle adjustment with reprojection, per-camera prior, and relative-pose prior terms. This approach converts pairwise predictions into a globally consistent multi-camera calibration. Experiments on KITTI (in-domain for CMRNext) and Walkley (out-of-domain) datasets show improved per-camera accuracy and inter-camera consistency. On KITTI, the method achieves 0.89 cm translation error and 0.038 rotation error. On Walkley, it reduces translation error from 108.6 cm to 3.1 cm, highlighting the benefit of explicit multi-camera coupling when single-camera predictions are less reliable.
Aziz Al-Najjar, Marzieh Amini, James R. Green +1
Department of System and Computer Engineering, Carleton University, Ottawa, Canada