XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration

Aurelien Godet; Gabriel Jobert; Mauro Dalla Mura

cs.CVSep 18, 2026

XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration

Authors: Aurelien Godet, Gabriel Jobert, Mauro Dalla Mura

Organizations: Univ. Grenoble Alpes, CNRS, Grenoble INP–UGA, GIPSA-lab, 38000 Grenoble, France · Lynred, 364 Avenue de Valence, 38113 Veurey-Voroize, France · Institut Universitaire de France (IUF), France

Abstract

Image registration is a vital preprocessing step in multimodal perception tasks, including image fusion, object detection, and semantic segmentation. In Advanced Driver- Assistance Systems (ADAS), spatial misalignment between visible (RGB) and infrared (IR) cameras -caused by non-coincident optical axes and field-of-view differences- introduces non-uniform parallax and visual ghosting. Classical keypoint-based methods are restricted to global homographies that fail under dynamic depth, while unconstrained dense flow algorithms lack structural regularization and suffer from temporal instability. In this paper, we propose XCalib, an unsupervised dense thermal-visible registration framework that bridges this gap. Rather than serving as an absolute metric calibration tool, XCalib leverages virtual pinhole camera parameterization strictly as a geometric constraint space. By optimizing effective relative pose and intrinsics alongside predicted monocular metric depth, XCalib restricts the search space of spatial displacements to physically valid projection geometries. Our key contributions are: (1) a novel registration paradigm that uses camera parameterization as an implicit regularizer for dense cross-modal warping; (2) Normalized Edges Correlation (NEC), a robust structural similarity metric tailored to cross- spectral alignment; and (3) extensive quantitative and qualitative evaluations across public ADAS datasets, demonstrating superior temporal stability and alignment accuracy over unconstrained dense flow baselines.

Figures & tables

Fig. 1: Samples from the Lynred Mobility Dataset (left column) and the FLIR ADAS v2 dataset (right column). Top: superimposed visible and infrared images without registration. Bottom: superimposed visible and warped infrared images using the dense mapping estimated by our method.

Fig. 2: The proposed two-stage pipeline for thermal–visible image registration. The first stage involves unsupervised calibration of the camera rig to estimate intrinsic and extrinsic parameters using monocular depth estimation. The second stage uses these parameters, along with metric depth predictions, to compute dense pixel-wise mappings between thermal and visible images.

Fig. 3: Samples from the selected datasets. First row: the rgb images, Second row: the infrared images

Dataset*	Vis**	Focal diff	Baseline	Resolution RGB	Nb frames
Lynred day	high	low	low	960x1280	1800
Lynred night	med	low	low	960x1280	1800
FLIR day	high	med	low	1024x1280	672
FLIR night	med	high	low	1600x1800	1040
RoadScene	high	Aligned	Aligned	variable	221

TABLE I: Datasets used for the experiments. *The color rates the difficulty of the alignment: easy , medium , high . **Visibility: refers to the illumation or the weather conditions

Panel of Figure 5 — Fig. 4: Experiment 1: Metrics (divided by per v[ $\delta=0$ ]) evolution with increasing disparity ( $\delta$ ) between the infrared and visible registered images. We expect metrics suitable for cross-modality registration quality assessment to consistently decrease as the disparity increases.

Dataset	Metric	Source	RIFT [ 9 ]	IMF [ 17 ]	SuperFusion [ 19 ]	CrossRAFT [ 20 ]	PGMR [ 29 ]	XCalib (ours)
Lynred day	NEC $\uparrow$	0.45	0.54 (+21.2%)	0.46 (+2.8%)	0.60 (+34.3%)	0.60 (+34.2%)	0.56 (+26.8%)	0.57 (+28.8%)
	GC $\uparrow$	0.19	0.24 (+28.3%)	0.22 (+14.4%)	0.38 (+103.0%)	0.35 (+87.0%)	0.34 (+79.3%)	0.33 (+74.1%)
	SSIM $\uparrow$	0.48	0.50 (+4.2%)	0.49 (+2.6%)	0.52 (+7.7%)	0.51 (+7.1%)	0.51 (+6.6%)	0.52 (+7.9%)
	NCC $\uparrow$	0.14	0.15 (+12.3%)	0.14 (+4.5%)	0.16 (+18.0%)	0.16 (+17.2%)	0.15 (+14.3%)	0.16 (+17.2%)
Lynred night	NEC $\uparrow$	0.39	0.43 (+9.7%)	0.39 (+1.3%)	0.47 (+21.0%)	0.46 (+19.1%)	0.44 (+13.3%)	0.46 (+17.2%)
	GC $\uparrow$	0.05	0.07 (+33.7%)	0.06 (+31.6%)	0.12 (+145.0%)	0.10 (+111.6%)	0.14 (+179.1%)	0.13 (+173.1%)

TABLE II: Experiment 2. Quantitative evaluation of image alignment. The Source column reports the metric values computed on the images before registration. All other columns report the corresponding values after registration. The top three results for each metric are highlighted from first to third using 1.GREEN , 2.BLUE , and 3.CYAN .

Dataset	Method	EPE $\downarrow$	$<1$ px $\uparrow$	$<3$ px $\uparrow$	$<5$ px $\uparrow$
Lynred day	SFusion	0.86	0.75	0.95	0.98
	RIFT	5.19	0.28	0.69	0.83
	C-RAFT	0.70	0.83	0.97	0.99
	PGMR	0.85	0.74	0.97	0.99
	Ours	0.53	0.90	0.98	0.99
Lynred night	SFusion	1.68	0.51	0.84	0.94

TABLE III: Flow evaluation metrics. The best performances per metric and dataset is highlighted in green .

Panel of Figure 8 — Fig. 5: Superimposed images with details for each methods (columns) and each datasets (rows). ’Source’ are the raw images superimposed without registration. In each fused image the infrared image is only colormapped with the colormap ’twilight’ from Matplotlib. For the second row, the visible registered image and the visible ground truth are superimposed.

Fig. 6: Comparison of the convergence behavior of the optimization with different translation parameterizations. The combined loss is the weighted sum of the flow and image losses described in the following paragraph. The transparent shaded area around each curve represents the standard deviation across 10 randomly initialized runs, while the solid line represents the mean loss.

Cross-Modal Image Registration

Cross-modal image registration aligns images from different sensors or modalities into a shared spatial frame.

See all 69 papers

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration
Aug 11, 2026Qi Ming, Yuyang Wang, Mingjing Zhao +7Multispectral Object Detection Cross-Modal Alignment
Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4
FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration
Mar 8, 2026Congcong Bian, Haolong Ma, Hui Li +4Infrared-Visible Image Fusion Cross-Modal Image Registration
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Jul 27, 2026Minchong Chen, Xiaoyun Yuan, Minyu Cao +4Diffusion Models for Image Restoration Multi-Source Image Fusion
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China

Aug 11, 2026J/K move · Enter open · S save

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Qi Ming, Yuyang Wang, Mingjing Zhao, Yifan Xiao+6

Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute+4

Multispectral Object Detection Cross-Modal Alignment Infrared Object Detection Cross-Modal Image Registration Multimodal Sensor Fusion

Open paper AskPDF arXiv

Visible-infrared object detection relies on complementary RGB and thermal cues, but its performance is often degraded by cross-modal spatial misalignment. Most existing methods rely on implicit feature adaptation to handle weakly misaligned scenarios, while large-offset geometric discrepancies remain insufficiently addressed. In this paper, we propose a Joint Feature-domain Registration and Detection network (JFRDet), an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies. JFRDet introduces a Cross-Modal Affine Alignment (CMAA) module to estimate an image-level affine transformation for explicit multi-level feature alignment. Note that illumination changes directly affect the reliability of RGB cues, an Illumination-Guided Complementary Fusion (IGCF) module adaptively exploits modality reliability under varying illumination conditions for cross-modal fusion. Then, an Alignment Quality-Consistency Gating (AQCG) strategy stabilizes joint optimization by modulating detection supervision according to alignment reliability and gradient consistency. We further construct DroneVehicle Misaligned (DVMA), a benchmark for evaluating visible-infrared oriented object detection under severe cross-modal geometric misalignment. The proposed JFRDet achieves 69.7%

\mathrm{mAP}_{50}

on DVMA, which represents state-of-the-art (SOTA) performance. The code and dataset will be available on GitHub.

Abstract

Cross-Modal Image Registration

Explore similar work

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion