Organizations: School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China
Spatial registration across different visual modalities is a critical but formidable step in multi-modality image fusion for real-world perception. Although several methods are proposed to address this issue, the existing registration-based fusion methods typically require extensive pre-registration operations, limiting their efficiency. To overcome these limitations, a general cross-modality registration method guided by visual priors is proposed for infrared and visible image fusion task, termed FusionRegister. Firstly, FusionRegister achieves robustness by learning cross-modality misregistration representations rather than forcing alignment of all differences, ensuring stable outputs even under challenging input conditions. Moreover, FusionRegister demonstrates strong generality by operating directly on fused results, where misregistration is explicitly represented and effectively handled, enabling seamless integration with diverse fusion methods while preserving their intrinsic properties. In addition, its efficiency is further enhanced by serving the backbone fusion method as a natural visual prior provider, which guides the registration process to focus only on mismatch regions, thereby avoiding redundant operations. Extensive experiments on three datasets demonstrate that FusionRegister not only inherits the fusion quality of state-of-the-art methods, but also delivers superior detail alignment and robustness, making it highly suitable for infrared and visible image fusion method. The code will be available at https://github.com/bociic/FusionRegister.
Figures & tables
Figure 1: Existing registration-based fusion methods face three key limitations: (I) Depend on artificial deformations; (II) Cannot interact with fusion methods; (III) Require extensive pre-operations and global registration. We propose a visual prior-based post-registration method that leverages fusion results to preserve fusion quality while significantly enhancing structural accuracy, achieving efficiency and robustness.
Figure 2: The fusion results show that even with spatial deformation in the infrared image, misregistration appears only in modality-shared regions, where structural similarity with the fine registered result is significantly reduced.
Figure 3: The proposed FusionRegister consists of two core components: post-registration of misregistration regions via bi-directional warping, and a modality retainment block (MRB). Note that ML is misregistration location operation and e-w denotes element-wise.
Figure 4: Detailed architecture of the Modality Retainment Block (MRB).
Figure 5: Detailed architecture of the Correlation Layer.
Figure 6: Image quality evaluation of FusionRegister under two scenarios. The method with “+” denotes the version aligned via FusionRegister.
Figure 7: As a general framework, FusionRegister successfully completes post-registration for different fusion methods and scenarios, and it completely retains the high-quality fusion performance of the original methods. More qualitative experiments are provided in the supplementary material.
Figure 8: A comparative experiment between FusionRegister and existing registration-based fusion methods. More visual comparisons are provided in the supplementary material.
MSRS
M 3 FD
LLVIP
Method
IoU
PR
IoU
PR
IoU
PR
Registration-based Fusion Methods
MURF [ 52 ]
58.3
73.3
61.3
75.1
67.3
78.4
SemLA [ 50 ]
78.7
88.4
75.0
85.9
81.0
89.7
C 2 RF [ 44 ]
70.5
81.5
69.4
82.6
75.4
86.1
IVFWSR [ 17 ]
64.6
79.4
70.7
83.2
75.6
86.1
Table 1: Segmentation quantitative results on MSRS, M 3 FD and LLVIP datasets represent registration accuracy. The bold and red represent the best and second-best performance. +FR denote the enhanced versions with FusionRegister.
Reg
Image Quality
Efficiency
Method
IoU
PR
EN
SF
AG
SD
T(s)
P(M)
w/o MRB
85.6
91.9
6.92
11.46
3.95
42.71
0.012
2.83
1-d Warping
85.5
92.4
7.02
11.58
4.04
43.57
0.019
2.94
MRB w/ dt
86.0
92.3
7.03
11.53
3.96
44.15
0.025
3.21
MRB w/ dc
84.8
92.1
6.98
11.41
3.98
42.52
0.014
3.25
More Layers
86.3
92.9
7.03
11.65
4.05
43.79
0.021
7.32
Table 2: Quantitative analysis of registration accuracy, image quality and complexity from the ablation experiments on MSRS. The bold and red represent the best and second-best
Figure 9: The MRB enables FusionRegister to preserve rich, fine-grained details.
Figure 10: Bi-directional warping enables FusionRegister to achieve more precise registration.
Figure 11: Implementing modality retainment with gMLP leads to the extraction of more comprehensive and abundant features.
SemLA
MURF
IVFWSR
IMF
CAP
C 2 RF
Ours
Params (M)
28.03
13.4
53.8
52.6
1.5
10.53
2.94
MSRS (S)
0.75
2.23
0.008
0.188
4.5
0.139
0.019
M 3 FD (S)
2.39
4.85
0.013
0.465
10.57
0.349
0.057
Table 3: Comparison of Resource Consumption and Parameters. The bold and red represent the best and second-best performance. The blue represents the WORST performance
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China