Online Target-less Radar-LiDAR-Camera Extrinsic Calibration via Joint Optimization
Authors: Gunhee Shin, Yunsoo Kim, Chanhyuk Lee, Wanhee Kim, Minwoo Lee, Sungwoo Han, Jeongwoo Woo, Hyuntai Chin, Minha Park, Hyun Myung
Organizations: School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon, 34141, Korea · Kookmin University, Seoul, 02707, Korea · LG Innotek, Seoul, 07796, Korea
Fusing radar, LiDAR, and camera enables robust perception in diverse and adverse conditions, but the fusion performance critically depends on accurate extrinsic calibration among the three sensors. In this paper, we address the problem of online target-less extrinsic calibration for the radar-LiDAR-camera system. Existing target-less methods are mostly designed for a single sensor pair, and composing the pairwise results does not guarantee consistency across the three sensors. Moreover, the sparse and noisy radar measurements make the radar-involving pairs unreliable. To tackle these challenges, we propose a joint calibration framework that constructs residuals for each sensor pair and optimizes the extrinsics of all pairs together to minimize the overall residual. Furthermore, we introduce an adaptive radar noise filter that rejects spurious radar returns using a range-dependent margin, and a correspondence accumulation strategy that aggregates sparse radar correspondences over frames. We validate our method on an in-house radar-LiDAR-camera dataset covering diverse urban environments, where it reduces calibration errors across all sensor pairs over a state-of-the-art camera-LiDAR baseline.
Figures & tables
Figure 1: Overview of the proposed joint radar-LiDAR-camera calibration pipeline. Time-synchronized measurements are processed by two correspondence matchers: a point cloud and image matcher (CMRNext) for the camera-involving pairs and a point-to-plane matcher for the radar-LiDAR pair, where the adaptive radar noise filter rejects spurious radar correspondences. The correspondences are accumulated over frames, and the joint optimization estimates the extrinsics of all three sensor pairs.
Sequence
Route 1
Route 2
Route 3
Sensor pair
R2C
L2C
R2L
R2C
L2C
R2L
R2C
L2C
R2L
Error metric
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
CMRNext
2.56
0.47
0.62
0.34
2.71
0.63
1.39
0.53
0.45
0.36
1.42
0.68
2.06
0.56
0.74
0.36
2.05
0.68
Ours
0.64
0.41
0.56
0.36
0.47
0.23
0.50
0.34
0.40
0.36
0.31
0.15
0.88
0.36
0.76
0.34
0.64
0.14
Table 1: Quantitative comparison of the proposed method against the CMRNext baseline across three routes of our in-house dataset. R [°] and T [m] indicate rotation and translation errors. Best results per column are highlighted in bold.
Figure 2: Distribution of rotation (left) and translation (right) errors of CMRNext and the proposed method over 10 random perturbation samples for the radar-camera (R2C), LiDAR-camera (L2C), and radar-LiDAR (R2L) sensor pairs. The proposed method achieves a lower median and a tighter spread across all three pairs.
Sensor pair
R2C
L2C
R2L
Error metric
R [°]
T [m]
R [°]
T [m]
R [°]
T [m]
CMRNext
2.00
0.52
0.60
0.35
2.06
0.66
Ours (ARNF)
1.71
0.46
0.60
0.35
1.79
0.57
Ours (ARNF+CA)
1.39
0.60
0.58
0.36
1.50
0.69
Ours (ARNF+CA+JO)
0.68
0.37
0.58
0.35
0.47
0.17
Table 2: Ablation study of the proposed components (ARNF, CA, JO) against the CMRNext baseline, averaged over all three test routes of our in-house dataset. R [°] and T [m] indicate rotation and translation errors. Best results per column are highlighted in bold.
Method
Accum. frames per update
Time per update [s]
Per-frame [s]
CMRNext
1
0.67
0.67
Ours (ARNF)
1
0.68
0.68
Ours (ARNF+CA)
10
0.67
0.07
Ours (ARNF+CA+JO)
10
10.64
1.06
Table 3: Runtime comparison of the proposed components against the CMRNext baseline. Configurations with CA accumulate correspondences over 10 frames before each calibration update; we therefore report both the per-update cost and the amortized per-frame cost.
In this paper, we address extrinsic calibration for camera, lidar, and 4D radar sensors. Accurate extrinsic calibration of radar remains a challenge due to the sparsity of its data. We propose CLRNet, a novel, multi-modal end-to-end deep learning (DL) calibration network capable of addressing joint camera-lidar-radar calibration, or pairwise calibration between any two of these sensors. We incorporate equirectangular projection, camera-based depth image prediction, additional radar channels, and leverage lidar with a shared feature space and loop closure loss. In extensive experiments using the View-of-Delft and Dual-Radar datasets, we demonstrate superior calibration accuracy compared to existing state-of-the-art methods, reducing both median translational and rotational calibration errors by at least 50%. Finally, we examine the domain transfer capabilities of the proposed network and baselines, when evaluating across datasets. The code will be made publicly available upon acceptance at: https://github.com/tudelft-iv.
Marcell Kegl, Andras Palffy, Csaba Benedek +1
Hungarian Research Network Institute for Computer Science and Control (HUN-REN SZTAKI), 1111 Budapest, Kende utca 13-17 · Pázmány Péter Catholic University, Faculty of Information Technology and Bionics, 1083 Budapest, Práter utca 50/A · Perciv AI, Delft, the Netherlands +1
Accurate camera-LiDAR calibration is a prerequisite for robust multi-modal perception in robotics. Recent target-less approaches based on deep point correspondences achieve remarkable performance for extrinsic calibration but assume rectified images with known intrinsics. In this work, we overcome this limitation and present the first fully target-less pipeline that jointly estimates camera intrinsics (pinhole model with radial-tangential distortion) and camera-LiDAR extrinsics with deep pixel-point correspondences. Our approach extends deep correspondence-based calibration by (i) automatic intrinsic initialization via structure-from-motion, (ii) generalizing camera-LiDAR matching to raw images with unknown intrinsics including distortion, and (iii) tightly coupling correspondence estimation with joint nonlinear optimization over both intrinsics and extrinsics. We evaluate our method on the KITTI dataset with unseen camera-LiDAR pairs and demonstrate that joint calibration achieves improved extrinsic accuracy while additionally recovering accurate intrinsics.
Simon Bultmann, Daniele Cattaneo, Abhinav Valada
Department of Computer Science, University of Freiburg, Germany
Most learning-based camera-LiDAR calibration methods treat each camera-LiDAR pair independently, ignoring the rigid geometric coupling in multi-camera platforms. As a result, per-camera estimates may be individually accurate yet inconsistent at the system level. We present a two-stage framework for joint multi-camera LiDAR extrinsic calibration that combines learned pairwise matching with geometric refinement. First, CMRNext is applied independently to each camera to produce initial extrinsic estimates and dense 2D-3D correspondences. These predictions are then jointly refined through a multi-frame bundle adjustment with reprojection, per-camera prior, and relative-pose prior terms. This approach converts pairwise predictions into a globally consistent multi-camera calibration. Experiments on KITTI (in-domain for CMRNext) and Walkley (out-of-domain) datasets show improved per-camera accuracy and inter-camera consistency. On KITTI, the method achieves 0.89 cm translation error and 0.038 rotation error. On Walkley, it reduces translation error from 108.6 cm to 3.1 cm, highlighting the benefit of explicit multi-camera coupling when single-camera predictions are less reliable.
Aziz Al-Najjar, Marzieh Amini, James R. Green +1
Department of System and Computer Engineering, Carleton University, Ottawa, Canada