LiREC-Net: A Target-Free and Learning-Based Network for LiDAR, RGB, and Event Calibration

Aditya Ranjan Dash; Ramy Battrawy; René Schuster; Didier Stricker

cs.CVFeb 25, 2026

LiREC-Net: A Target-Free and Learning-Based Network for LiDAR, RGB, and Event Calibration

Authors: Aditya Ranjan Dash, Ramy Battrawy, René Schuster, Didier Stricker

Organizations: RPTU – University Kaiserslautern-Landau · DFKI – German Research Center for Artificial Intelligence

Abstract

Advanced autonomous systems rely on multi-sensor fusion for safer and more robust perception. To enable effective fusion, calibrating directly from natural driving scenes (i.e., target-free) with high accuracy is crucial for precise multi-sensor alignment. Existing learning-based calibration methods are typically designed for only a single pair of sensor modalities (i.e., a bi-modal setup). Unlike these methods, we propose LiREC-Net, a target-free, learning-based calibration network that jointly calibrates multiple sensor modality pairs, including LiDAR, RGB, and event data, within a unified framework. To reduce redundant computation and improve efficiency, we introduce a shared LiDAR representation that leverages features from both its 3D nature and projected depth map, ensuring better consistency across modalities. Trained and evaluated on established datasets, such as KITTI and DSEC, our LiREC-Net achieves competitive performance to bi-modal models and sets a new strong baseline for the tri-modal use case.

Figures & tables

Figure 1: Our LiREC-Net takes miscalibrated tri-modal inputs and learns to produce spatially aligned outputs. The top row shows the raw miscalibrated overlays, while the bottom row illustrates the calibrated LiDAR projected onto both RGB and event frames.

Figure 2: Overview of LiREC-Net. A miscalibrated LiDAR point cloud $P$ is processed by two LiDAR encoders (point- and depth-based). Point features are projected to the image plane using known intrinsics $\mathbf{K}_{\mathrm{RGB}}$ and $\mathbf{K}_{\mathrm{Ev}}$ , and then fused with depth features to form a unified LiDAR embedding. In parallel, the RGB image $I$ and event representation $E$ are encoded by their respective encoders. The unified LiDAR embedding is combined with the corresponding RGB/event features to build two pair-wise cost volumes, which are refined by context modules and passed to prediction heads that output the LiDAR - RGB and LiDAR - Event extrinsics $\hat{\mathbf{T}}^{\mathrm{Li}-\mathrm{RGB}}$ and $\hat{\mathbf{T}}^{\mathrm{Li}-\mathrm{Ev}}$ .

Network	LiDAR - RGB	LiDAR - Event
Network	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]
RegNet [ 18 ]	6.00 / 0.28	— / —
CalibNet [ 11 ]	4.34 / 0.41	— / —
LCCNet [ 16 ]	1.59 / 0.16	— / —
PseudoCal [ 2 ]	1.18 / 0.05	— / —
LiREC-Net (Ours)	1.80 / 0.11	1.82 / 0.12

Table 1: Calibration results on the KITTI dataset when LiREC-Net is trained and evaluated with maximum perturbation of

\pm(20^{\circ}/150\,\text{cm})

Network	LiDAR - RGB	LiDAR - Event
Network	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]
MULiEv $\triangle$ [ 3 ]	— / —	0.81 / 0.10
LiREC-Net $\triangle$ (Ours)	2.62 / 0.30	2.05 / 0.25
LiREC-Net * (Ours)	2.51 / 0.14	1.18 / 0.07

Table 2: Calibration results on the DSEC dataset. [*] Training/evaluation with 5 stages at

\pm(20^{\circ}/150\,\text{cm})

, [

\triangle

] with 2 stages at

\pm(10^{\circ}/100\,\text{cm})

Panel of Figure 3 — Figure 3: Qualitative results on KITTI [ 6 ] for both LiDAR - RGB and LiDAR - Event pairs.

Panel of Figure 4 — Figure 4: Qualitative results on DSEC [ 5 ] for both LiDAR - RGB and LiDAR - Event pairs.

Network	LiDAR - RGB	LiDAR - Event	T	P	M
Network	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	[s]	[ $\times 10^{9}$ ]	[GiB]
KITTI [ 6 ]
Bi-modal *	2.08 / 0.11	1.90 / 0.09	0.51	1.9	14.6
Tri-modal *	1.80 / 0.11	1.82 / 0.12	0.33	1.7	11.1
DSEC [ 5 ]
Bi-modal $\triangle$	2.55 / 0.28	3.43 / 0.18	0.22	0.8	9.7

Table 3: Calibration result comparison when LiREC-Net is trained and evaluated in tri-modal and bi-modal settings. [*] Training/evaluation with 5 stages at

\pm(20^{\circ}/150\,\text{cm})

, [

\triangle

] with 2 stages at

\pm(10^{\circ}/100\,\text{cm})

. T is the average inference time, P is the total number of model parameters, and M is the GPU memory used for inference.

Point	Depth	LiDAR - RGB	LiDAR - Event
Features	Features	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]
✓		14.43 / 0.70	14.05 / 0.64
	✓	2.97 / 0.70	2.16 / 0.60
✓	✓	2.51 / 0.14	1.18 / 0.07

Table 4: Ablation showing the effect of LiDAR’s point and depth feature fusion on LiDAR - RGB and LiDAR - Event calibration, conducted on the DSEC dataset.

SDP	SFP	LiDAR - RGB	LiDAR - Event
SDP	SFP	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]
		3.37 / 0.34	3.35 / 0.30
✓		2.95 / 0.15	1.45 / 0.07
	✓	4.18 / 0.18	3.72 / 0.16
✓	✓	2.51 / 0.14	1.18 / 0.07

Table 5: Ablation showing the effect of scaled projections (SDP and SFP) on LiDAR - RGB and LiDAR - Event calibration, conducted on DSEC dataset.

Encoder	LiDAR - RGB	LiDAR - Event
Encoder	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]	$e_{t}$ [cm] / $e_{r}$ [ ∘ ]
ResNet [ 8 ]	3.04 / 0.19	2.45 / 0.16
MViTV2 [ 15 ]	2.51 / 0.14	1.18 / 0.07

Table 6: Ablation showing the effect of MViTV2 and ResNet on LiDAR - RGB and LiDAR - Event calibration, conducted on DSEC dataset.

Split	Sequences	No. of samples
Training	01–21	39011
Evaluation	00	0 4541

Table 7: Training and evaluation splits for the KITTI dataset [ 6 ] .

Split	Sequences	No. of samples
Training	interlaken_00_c–g	52727
	thun_00_a
	zurich_city_00_a–b
	zurich_city_01_a–f
	zurich_city_02_a–e
	zurich_city_03_a

Table 8: Training and evaluation splits for the DSEC dataset [ 5 ] .

Dataset	Stage	Epochs	Decay Milestones
KITTI [ 6 ]	Stage 1	120	20, 50, 70, 100
KITTI [ 6 ]	Other Stages	50	15, 30, 40
DSEC [ 5 ]	Stage 1	150	100, 110, 120, 130
DSEC [ 5 ]	Other Stages	70	45, 50, 55, 60

Table 9: Training schedule with epochs and learning rate (LR) milestones per stage.

Parameter(s)	Description	Value(s)
$H\times W$	Image size (KITTI)	0 $376\times 1240$
$H\times W$	Image size (DSEC)	$1080\times 1440$
$H^{\prime}\times W^{\prime}$	Model input size	$256\times 512$
$H^{\prime\prime}\times W^{\prime\prime}$	Feature size	$16\times 32$
$d$	Correlation radius	4
$\mu_{RGB}$	Mean value (KITTI)	[0.485, 0.456, 0.406]

Table 10: Hyperparamters and corresponding values of LiREC-Net.

Network	Translation				Rotations
Network	$e_{X}$ [cm]	$e_{Y}$ [cm]	$e_{Z}$ [cm]	$\frac{e_{X},e_{Y},e_{Z}}{3}$ [cm]	$e_{R}$ [ ∘ ]	$e_{P}$ [ ∘ ]	$e_{Y}$ [ ∘ ]	$\frac{e_{R},e_{P},e_{Y}}{3}$ [ ∘ ]
LCCRAFT [ 14 ]	0.821	0.771	1.631	1.074	0.029	0.117	0.041	0.062
LiREC-Net	1.348	0.523	0.747	0.873	0.043	0.092	0.030	0.055

Table 11: Comparison between LCCRAFT [ 14 ] and our LiREC-Net on KITTI. Errors are reported per translation axis (

e_{X}

e_{Y}

e_{Z}

in cm) and rotation (

e_{R}

e_{P}

e_{Y}

in [ ∘ ]).

Table 12: Detailed per-stage results of LIREC-Net.

Panel of Figure 5 — Figure 5: Per-stage visualization for LiDAR - RGB calibration on KITTI. The exact translation and rotation error is provided for the perturbed input and each stage. For miscalibrated input, there are no LiDAR points inside the RGB image boundary because of the large perturbation.

Panel of Figure 6 — Figure 6: Per-stage visualization for LiDAR - Event calibration on KITTI. The exact translation and rotation error is provided for the perturbed input and each stage. For miscalibrated input, there are no LiDAR points inside the RGB image boundary because of the large perturbation.

Panel of Figure 7 — Figure 7: Per-stage visualization for LiDAR - RGB calibration on DSEC. The exact translation and rotation error is provided for the perturbed input and each stage.

Panel of Figure 8 — Figure 8: Per-stage visualization for LiDAR - Event calibration on DSEC. The exact translation and rotation error is provided for the perturbed input and each stage.

Lidar-Camera Extrinsic Calibration

See all 17 papers

CalibBEV: LiDAR-Camera Calibration via BEV Alignment
Aug 3, 2026Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace +3Cross-Modal Alignment Camera-LiDAR Fusion
University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA. +1
CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning
Mar 16, 2026Marcell Kegl, Andras Palffy, Csaba Benedek +1Lidar-Camera Extrinsic Calibration Camera Calibration
Hungarian Research Network Institute for Computer Science and Control (HUN-REN SZTAKI), 1111 Budapest, Kende utca 13-17 · Pázmány Péter Catholic University, Faculty of Information Technology and Bionics, 1083 Budapest, Práter utca 50/A · Perciv AI, Delft, the Netherlands +1
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
Mar 31, 2026Ni Ou, Zhuo Chen, Xinru Zhang +1Lidar-Camera Extrinsic Calibration Camera Calibration
Ni Ou, Junzheng Wang are with the School of Automation, Beijing Institute of Technology, Beijing, 100081, China. · Zhuo Chen is with the Robot Perception Lab, Centre for Robotics Research, Department of Engineering, King’s College London, London WC2R 2LS, United Kingdom. · Xinru Zhang is with the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, 100081, China.

Aug 3, 2026J/K move · Enter open · S save

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace, Emanuele Ghelfi+2

University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA.+1

Cross-Modal Alignment Camera-LiDAR Fusion Lidar-Camera Extrinsic Calibration

Open paper AskPDF arXiv

We present CalibBEV, a novel Bird's Eye View (BEV) alignment approach for LiDAR-camera calibration. Our method unifies LiDAR and camera data into a shared 3D spatial representation, enabling accurate and robust cross-modal calibration. CalibBEV extracts sensor-wise BEV features from each modality using domain-specific architectures and estimates the calibration matrix through a two-step alignment process. First, we perform an implicit alignment by regressing a coarse calibration matrix directly from the BEV features. To ease this alignment, we enforce semantic consistency between BEV representations across modalities using a contrastive loss inspired by CLIP, guiding both networks toward a unified feature space. In the second step, we leverage our BEV formulation to explicitly align the features of one modality with the other, refining the initial coarse estimate into a final, more accurate calibration matrix. CalibBEV significantly outperforms prior point-to-pixel matching methods, achieving state-of-the-art calibration accuracy. On the KITTI and nuScenes benchmarks, our method reduces the Relative Rotation Error (RRE) by 51% and 68%, and the Relative Translation Error (RTE) by 80% and 91%, respectively, compared to previous methods.

Abstract

Lidar-Camera Extrinsic Calibration

Explore similar work

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning

Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations