Simplified Cross-Modal Calibration for Heterogeneous Event-RGB Stereo Systems

Nico Hessenthaler; Adam T. Müller; Nicolaj C. Stache

cs.CVAug 24, 2026

Simplified Cross-Modal Calibration for Heterogeneous Event-RGB Stereo Systems

Authors: Nico Hessenthaler, Adam T. Müller, Nicolaj C. Stache

Organizations: Center for Machine Learning Heilbronn University of Applied Sciences Heilbronn, Germany

Abstract

Accurate extrinsic calibration between event-based and frame-based cameras remains a practical bottleneck for heterogeneous stereo systems. Existing approaches often require sensor or target motion, precise synchronization, or computationally expensive event-to-image reconstruction. We propose a simple, motion-free cross-modal calibration framework that uses a temporally modulated, blended ChArUco target presented on standard consumer displays. By alternating between the original pattern and a partially blended version, the target reliably triggers events while remaining continuously observable to a frame-based camera, avoiding blank frames and reducing synchronization constraints to a coarse, trigger-based alignment. We discretize events into frames coarsely aligned with the RGB images, apply lightweight denoising, and perform ChArUco-based intrinsic and stereo extrinsic calibration. Extensive experiments assess robustness to blending opacity, display brightness, external illumination, viewing angle, and handheld acquisition. Compared to the strongest motion-based reference (E2Calib + Kalibr) and a non-motion-based reference (Plasberg et al.), our approach reduces the mean reprojection error by $44\%$ and $6\%$ , respectively, while substantially simplifying the calibration procedure. Finally, we demonstrate practical utility in a robotic eye-to-hand calibration case study, showing consistent transformations and stable downstream geometric measurements even under partial occlusions. Code is publicly available at https://github.com/nhessenthaler/simple-evrgb-cal.

Figures & tables

Figure 1: Pipeline for cross‑modal stereo calibration using a temporally modulated ChArUco pattern. A pattern of blended images is generated via linear interpolation and modulated over time, enabling robust detection in both frame‑based and event cameras.

Figure 2: Detection rate and stereo reprojection error as a function of the blending opacity $\alpha$ . Frame-based detection remains close to $100\%$ across most opacity levels, with a slight degradation for $\alpha\rightarrow 1$ due to the sporadic invisibility of the pattern in the absence of temporal display synchronization. In contrast, event-based detection increases with $\alpha$ and saturates for $\alpha\geq 0.4$ . The stereo reprojection error reaches a minimum at $\alpha=0.6$ , indicating an optimal trade-off between detection robustness and calibration accuracy.

Brightness [%]	$L_{\text{white}}$ [cd/m 2 ]	$L_{\text{black}}$ [cd/m 2 ]	$u_{c}$ [%] $\uparrow$	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
0	35.0	0.03	15.3	–
20	78.0	0.06	98.5	0.442
40	121.0	0.10	100.0	0.418
60	164.0	0.13	100.0	0.405
80	236.0	0.19	100.0	0.395
100	391.0	0.32	100.0	0.384

Table 1: Influence of display brightness on stereo calibration performance for a single experimental run, evaluated across brightness levels from

0\%

100\%

. The table reports the measured luminance of the white (

L_{\text{white}}

) and black (

L_{\text{black}}

) display states, the combined successful corner detection rate (

u_{c}

), and the stereo reprojection error (

\varepsilon_{\text{rep}}

), with arrows (

\uparrow,\downarrow

) indicating the direction of improved performance.

Sensor		$u_{c}$ [%] $\uparrow$	$f_{x}$ [px]	$f_{y}$ [px]	$c_{x}$ [px]	$c_{y}$ [px]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
RGB	( $\mu$ )	100.0	2771.5	2772.6	800.1	450.8	0.142
	( $\sigma$ )	0.0	3.5	3.4	1.8	1.4	0.006
Event	( $\mu$ )	100.0	2571.1	2572.3	609.3	378.2	0.449
	( $\sigma$ )	0.0	26.4	26.3	10.7	9.1	0.013

Table 2: Statistical robustness evaluation of the intrinsic parameters over 20 independent calibration runs using a robotic setup. The table reports the mean (

\mu

) and standard deviation (

\sigma

) for the corner detection success rate (

u_{c}

), focal lengths (

f_{x},f_{y}

), principal point (

c_{x},c_{y}

), and reprojection error (

\varepsilon_{\text{rep}}

Sensor		$u_{c}$ [%] $\uparrow$	$\\|\mathbf{r}\\|$ [°]	$\\|\mathbf{t}\\|$ [m]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
RGB + Event	( $\mu$ )	99.8	1.077	0.0346	0.383
	( $\sigma$ )	0.5	0.193	0.00017	0.012

Table 3: Statistical robustness evaluation of the extrinsic parameters over 20 runs. The table reports the successful corner detection rate (

u_{c}

), the magnitude of the estimated rotation (

\|\mathbf{r}\|

) and translation (

\|\mathbf{t}\|

), and the reprojection error (

\varepsilon_{\text{rep}}

Sensor		$u_{c}$ [%] $\uparrow$	$\\|\mathbf{r}\\|$ [°]	$\\|\mathbf{t}\\|$ [m]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
RGB + Event	( $\mu$ )	92.8	1.252	0.0345	0.402
	( $\sigma$ )	10.8	0.266	0.00012	0.017

Table 4: Illumination robustness evaluation of the extrinsic parameters over 5 runs with varying spotlight positions. The table reports the successful corner detection rate (

u_{c}

), the magnitude of the estimated rotation (

\|\mathbf{r}\|

) and translation (

\|\mathbf{t}\|

), and the reprojection error (

\varepsilon_{\text{rep}}

	Vertical (V)		Horizontal (H)		Diagonal (D)
Angle	$u_{c}$	$\varepsilon_{\text{rep}}$	$u_{c}$	$\varepsilon_{\text{rep}}$	$u_{c}$	$\varepsilon_{\text{rep}}$
	[%] $\downarrow$	[px] $\uparrow$	[%] $\downarrow$	[px] $\uparrow$	[%] $\downarrow$	[px] $\uparrow$
30 ∘	100.0	0.322	100.0	0.320	100.0	0.313
40 ∘	100.0	0.363	100.0	0.342	100.0	0.388
50 ∘	100.0	0.374	100.0	0.359	89.2	0.482
55 ∘	100.0	0.436	56.7	0.374	48.3	0.926

Table 5: Viewing angle stability evaluation across viewing angles from

30^{\circ}

70^{\circ}

, evaluated along the vertical (V), horizontal (H), and diagonal (D) axes. The table reports the combined corner detection rate (

u_{c}

) and reprojection error (

\varepsilon_{\text{rep}}

), averaged over 5 runs per direction.

Sensor		$u_{c}$ [%] $\uparrow$	$\\|\mathbf{r}\\|$ [°]	$\\|\mathbf{t}\\|$ [m]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
RGB + Event	( $\mu$ )	100.0	0.9435	0.0343	0.361
	( $\sigma$ )	0.0	0.285	0.00027	0.041

Table 6: Handheld calibration evaluation of the extrinsic parameters over 5 runs. The table reports the successful corner detection rate (

u_{c}

), the magnitude of the estimated rotation (

\|\mathbf{r}\|

) and translation (

\|\mathbf{t}\|

), and the reprojection error (

\varepsilon_{\text{rep}}

Panel of Figure 3 — Figure 3: E2VID reconstructed intensity frames used for the original E2Calib baseline (checkerboard) and our adapted ChArUco target-based variant.

Method		$u_{c}$ [%] $\uparrow$	$\\|\mathbf{r}\\|$ [°]	$\\|\mathbf{t}\\|$ [m]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
E2Calib	( $\mu$ )	47.7	1.437	0.0339	1.235
	( $\sigma$ )	5.6	0.386	0.0004	0.156
E2Calib + ChArUco	( $\mu$ )	67.7	1.791	0.0338	1.161
	( $\sigma$ )	3.7	0.381	0.0003	0.108
E2Calib + Kalibr	( $\mu$ )		1.581	0.0360	0.687
	( $\sigma$ )		0.178	0.0002	0.038

Table 7: Extrinsic calibration performance of the motion-requiring baseline (E2Calib), its ChArUco-based adaptation (E2Calib + ChArUco), a variant using Kalibr instead of OpenCV for calibration (E2Calib + Kalibr), and the static baseline method of Plasberg et al., each evaluated over 5 runs. The table reports the successful corner detection rate (

u_{c}

), the magnitude of the estimated rotation (

\|\mathbf{r}\|

) and translation (

\|\mathbf{t}\|

), and the reprojection error (

\varepsilon_{\text{rep}}

Figure 4: Geometric transformation pipeline for eye-to-hand calibration, illustrating the mapping of a target point $\mathbf{p}_{\mathrm{t}}$ from the ChArUco target frame ( $\mathrm{t}$ ) to the event camera frame ( $\mathrm{e}$ ) via ${}^{\mathrm{e}}\mathbf{T}_{\mathrm{t}}$ , and subsequently to the robot base frame ( $\mathcal{B}$ ) using ${}^{\mathcal{B}}\mathbf{T}_{\mathrm{e}}$ , with the final relation to the TCP defined through the robot kinematics ${}^{\mathrm{tcp}}\mathbf{T}_{\mathcal{B}}$ .

Statistic		$t_{x}$ [m]	$t_{y}$ [m]	$t_{z}$ [m]	$\\|\mathbf{t}\\|$ [m]	$\theta$ [°]
${}^{\mathcal{B}}\mathbf{T}_{\mathrm{e}}$	( $\mu$ )	0.1258	-0.5678	0.3839	0.6970	173.76
	( $\sigma$ )	0.0093	0.0076	0.0061	0.0056	2.80

Table 8: Hand-eye calibration results aggregated over 20 valid runs. The table reports the mean (

\mu

) and standard deviation (

\sigma

) for the estimated translation vector (

\mathbf{t}=(t_{x},t_{y},t_{z})

) between the event camera and robot base, its Euclidean norm (

\|\mathbf{t}\|

), and the rotation magnitude (

\theta

) obtained from the rotation matrix

\mathbf{R}

via axis-angle representation.

Evaluation Mode	$\overline{d}$ [mm]	$\sigma_{d}$ [mm] $\downarrow$	$d_{\mathrm{gt}}$ [mm]	$\mathrm{MAE}$ [mm] $\downarrow$
Target-to-TCP	67.09	1.86	67.80	0.71
Base-Frame Distance	Scene dep.	0.66	Scene dep.	1.40

Table 9: Spatial consistency evaluation results averaged over 5 runs, with each run consisting of 22 diverse poses for the target-to-TCP evaluation and 6 trajectories for the base-frame distance evaluation. The table reports the mean Euclidean distance (

\overline{d}

) between the transformed calibration point (ID9) and the TCP, its standard deviation (

\sigma_{d}

), the ground truth distance (

d_{\mathrm{gt}}

), and the mean absolute error (

\mathrm{MAE}

) with respect to the ground truth.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The hybrid frame-event stereo rig is mounted on a UR5e robotic arm to ensure precise trajectory repeatability. The Dell S2722QC monitor displays the calibration pattern, serving as the cross-modal target for both the frame-based and event sensors.

Parameter UI-5250CP	Value
Exposure Time	$100.0\,\mathrm{ms}$
Gain	100
Frame Rate	$10.0\,\mathrm{fps}$
Gamma	100

Appendix

Table 10: IDS uEye frame-based camera configuration used in all experiments. The gamma value is specified in the camera parameterization scheme, where a value of 100 corresponds to a linear gamma of

\gamma=1.0

. All parameters not explicitly listed were kept at their hardware default settings.

Parameter EVK4	Value
Event Trail Filter	Enabled
Filtering Type	STC_CUT_TRAIL
Trail Threshold	100000
bias_diff_off	$-15$
bias_diff_on	$-35$
bias_fo	$-35$

Appendix

Table 11: Prophesee EVK4 sensor configuration used in all experiments. Bias values denote relative offsets from the hardware defaults and were tuned to optimize the signal-to-noise ratio for flickering pattern detection. All parameters not explicitly listed were kept at their hardware default settings.

Panel of Figure 6 — Figure 6: Qualitative example from the illumination robustness experiment. Both images depict the same scene. Differences in perspective are due to the stereo baseline offset between the frame-based and event camera sensors. (a) RGB image frame captured under strong external spotlight interference, causing overexposure on the calibration target. (b) Corresponding event frame recorded under the same conditions, illustrating the effect of intense illumination on event generation and pattern visibility.

Sensor		$u_{c}$ [%] $\uparrow$	$f_{x}$ [px]	$f_{y}$ [px]	$c_{x}$ [px]	$c_{y}$ [px]	$\varepsilon_{\text{rep}}$ [px] $\downarrow$
RGB	( $\mu$ )	100.0	2764.8	2766.1	797.1	449.5	0.140
	( $\sigma$ )	0.0	2.4	2.3	5.0	1.8	0.003
Event	( $\mu$ )	98.5	2568.9	2569.6	600.5	367.8	0.456
	( $\sigma$ )	3.4	11.4	11.6	9.3	10.2	0.007

Appendix

Table 12: Illumination robustness evaluation of the intrinsic parameters with our approach over 5 runs with varying spotlight positions. The table reports the corner detection success rate (

u_{c}

), focal lengths (

f_{x},f_{y}

), principal point (

c_{x},c_{y}

), and reprojection error (

\varepsilon_{\text{rep}}

Panel of Figure 7 — Figure 7: Robustness of the proposed method under occlusion during eye-to-hand calibration. Even in the presence of partial occlusions caused by the robot actuator, salient ChArUco corners (highlighted in red) are reliably detected, enabling accurate calibration.

Camera Calibration

Camera calibration estimates the geometric and optical parameters that connect image measurements to real-world positions and other sensors.

See all 56 papers

Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo
Apr 16, 2026Ninghui Xu, Fabio Tosi, Lihui Wang +5Event-Based Vision Cross-Modal Alignment
School of Instrument Science and Engineering, Southeast University, State Key Lab of Comprehensive PNT Network and Equipment Technology, Key Lab of Micro-Inertial Instrument and Advanced Navigation Technology, MOE · University of Bologna · Department of Computer Science and Engineering, University of Bologna +2
Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation
Jun 24, 2026Gwanhyeong Song, Chaehyeon Song, Ayoung KimMulti-View Geometry Camera Pose Estimation
Dept. of Mechanical Engineering, SNU, Seoul, S. Korea
StereoGeo: an end-to-end stereo camera calibration method
Jun 12, 2026Imane Meddour, Andréa Macario Barros, Cédric Gouy-PaillerStereo Vision Camera Calibration
Universite Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Apr 16, 2026J/K move · Enter open · S save

Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo

Ninghui Xu, Fabio Tosi, Lihui Wang, Jiawei Han+4

School of Instrument Science and Engineering, Southeast University, State Key Lab of Comprehensive PNT Network and Equipment Technology, Key Lab of Micro-Inertial Instrument and Advanced Navigation Technology, MOE · University of Bologna · Department of Computer Science and Engineering, University of Bologna+2

Event-Based Vision Cross-Modal Alignment Stereo Matching

Open paper AskPDF arXiv

Conventional frame-based cameras capture rich contextual information but suffer from limited temporal resolution and motion blur in dynamic scenes. Event cameras offer an alternative visual representation with higher dynamic range free from such limitations. The complementary characteristics of the two modalities make event-frame asymmetric stereo promising for reliable 3D perception under fast motion and challenging illumination. However, the modality gap often leads to marginalization of domain-specific cues essential for cross-modal stereo matching. In this paper, we introduce Bi-CMPStereo, a novel bidirectional cross-modal prompting framework that fully exploits semantic and structural features from both domains for robust matching. Our approach learns finely aligned stereo representations within a target canonical space and integrates complementary representations by projecting each modality into both event and frame domains. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods in accuracy and generalization.

Abstract

Camera Calibration

Explore similar work

Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo

Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation

StereoGeo: an end-to-end stereo camera calibration method