Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion

Yanglin Deng; Tianyang Xu; Chunyang Cheng; Hui Li; Xiao-jun Wu; Josef Kittler

cs.CVMar 23, 2026

Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion

Authors: Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler

Organizations: School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey

Abstract

Infrared and visible image fusion(IVIF) combines complementary modalities while preserving natural textures and salient thermal signatures. Existing solutions predominantly rely on extensive sets of rigidly aligned image pairs for training. However, acquiring such data is often impractical due to the costly and labour-intensive alignment process. Besides, maintaining a rigid pairing setting during training restricts the volume of cross-modal relationships, thereby limiting generalisation performance. To this end, this work challenges the necessity of Strictly Paired Training Paradigm (SPTP) by systematically investigating UnPaired and Arbitrarily Paired Training Paradigms (UPTP and APTP) for high-performance IVIF. We establish a theoretical objective of APTP, reflecting the complementary nature between UPTP and SPTP. More importantly, we develop a practical framework capable of significantly enriching cross-modal relationships even with severely limited and unaligned training data. To validate our propositions, three end-to-end lightweight baselines, alongside a set of innovative loss functions, are designed to cover three classic frameworks (CNN, Transformer, GAN). Comprehensive experiments demonstrate that the proposed APTP and UPTP are feasible and capable of training models on a severely limited and content-inconsistent infrared and visible dataset, achieving performance comparable to that of a dataset 100 $\times$ larger in SPTP. This finding fundamentally alleviates the cost and difficulty of data collection while enhancing model robustness from the data perspective, delivering a feasible solution for IVIF studies. The code is available at blue{https://github.com/yanglinDeng/IVIF_unpair}.

Figures & tables

Figure 1: Motivation of proposed two training paradigms. (a) Compared with the strictly-paired paradigm, the two proposed paradigms lift the requirement for spatiotemporally aligned pairs, expanding the trainable pairs to $N\times(N-1)$ and $N^{2}$ , respectively, reducing data-collection costs and improving generalisation. (b) Performance improvement pattern: greater enhancement from $N^{2}$ for Smaller $N$ .

Figure 2: Training convergence and performance comparison between SPTP and UPTP on three baselines.

Figure 3: Evolution of fusion results during CNN baseline training (SPTP v.s. UPTP)

Figure 4: Fused images from baselines trained on SPTP and APTP

Figure 5: Explanation of relationship diversity.

Figure 6: Qualitative comparison of three baselines (from top to bottom: CNN, Transformer, and GAN, respectively) under different training data settings on the LLVIP, M3FD, and MSRS test sets.

Datasets	Mode	Quantity			CNN					Transformer					GAN
Datasets	Mode	Paired Base	w/o Expanded	Trainable	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM
MSRS	SPTP	150	$\times$	150	6.54	2.57	0.88	0.60	0.96	6.53	2.44	0.86	0.59	0.98	6.52	2.17	0.75	0.51	0.90
	UPTP	150	$\times$	150	6.59	2.62	0.90	0.61	0.97	6.55	2.49	0.88	0.60	0.98	6.60	2.09	0.75	0.51	0.91
	UPTP	150	$\checkmark$	7500	6.56	2.69	0.92	0.65	0.98	6.56	2.69	0.92	0.65	0.98	6.58	2.52	0.90	0.62	0.99
	UPTP	150	$\checkmark$	15000	6.57	2.70	0.93	0.65	0.99	6.55	2.70	0.92	0.65	0.98	6.58	2.62	0.92	0.63	0.98
	SPTP	15000	$\times$	15000	6.59	2.70	0.93	0.65	0.98	6.55	2.78	0.93	0.66	0.97	6.57	2.60	0.92	0.64	0.99

Table 1: Qualitative generalisation results on the LLVIP (250 pairs), MSRS (361 pairs) and M 3 FD (420 pairs) test sets for three baseline models trained on MSRS with different training settings. Bold fonts indicate higher than their corresponding SPTP baseline groups.

Figure 7: Visualisation of fused images: cross-dataset unpaired training v.s. single-dataset strictly-paired training models.

Mode	Quantity			Infrared	Visible	EN	MI	VIF	Qabf	SSIM
Mode	PB	w/o E	T	Infrared	Visible	EN	MI	VIF	Qabf	SSIM
SPTP	150	$\times$	150	MSRS	MSRS	6.54	2.57	0.88	0.60	0.96
	150	$\times$	150	M3FD	M3FD	6.52	2.37	0.84	0.58	0.90
	150	$\times$	150	LLVIP	LLVIP	6.63	2.43	0.90	0.61	0.99
UPTP	150	$\checkmark$	7500	M3FD	MSRS	6.57	2.66	0.93	0.66	1.01
	150	$\checkmark$	7500	M3FD	LLVIP	6.59	2.63	0.93	0.65	1.01

Table 2: Qualitative results on MSRS of CNN baseline trained on different training sets. Bold indicates that the model performs better than the one trained on its original paired data.

Mode	Quantity			EN	MI	VIF	Qabf	SSIM	Avg. Imp.
Mode	PB	w/o E	T	EN	MI	VIF	Qabf	SSIM	Avg. Imp.
SPTP	15	$\times$	15	6.29	2.25	0.74	0.52	0.65	$-$
APTP	15	$\checkmark$	225	6.45	2.56	0.80	0.56	0.73	8.89%
SPTP	30	$\times$	30	6.48	2.48	0.80	0.59	0.93	$-$
APTP	30	$\checkmark$	900	6.55	2.62	0.90	0.61	0.96	5.17%
SPTP	60	$\times$	60	6.54	2.57	0.88	0.60	0.96	$-$

Table 3: Quantitative results on MSRS of comparing the improvement of APTP on different basic strictly-paired images.

	Venue	MB	LLVIP					MSRS					RoadScene					TNO
	Venue	MB	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM
DeF [ 21 ]	22’ ECCV	30.04	7.25	2.33	0.74	0.43	0.83	3.34	2.15	0.75	0.51	0.93	6.80	2.08	0.50	0.38	0.86	6.94	1.81	0.53	0.34	0.91
LRR [ 20 ]	23’ TPAMI	0.19	6.67	1.65	0.56	0.47	0.66	6.19	2.03	0.54	0.45	0.71	7.12	2.09	0.51	0.38	0.71	6.90	1.74	0.56	0.37	0.83
Meta [ 50 ]	23’ CVPR	3.10	7.14	1.16	0.61	0.29	0.60	6.37	1.16	0.71	0.48	0.78	7.17	1.60	0.51	0.36	0.75	7.09	1.12	0.57	0.35	0.80
DDF [ 53 ]	23’ ICCV	2108	7.17	1.78	0.58	0.23	0.58	6.17	1.89	0.74	0.47	0.91	7.42	2.13	0.57	0.42	0.88	6.83	1.46	0.56	0.36	0.90
Cross [ 19 ]	24’ Inf.	79.40	6.47	1.79	0.69	0.44	0.71	6.49	2.17	0.84	0.56	0.88	7.25	2.13	0.58	0.39	067	6.83	2.10	0.74	0.45	0.88

Table 4: Qualitative results of three baseline models and several SOTA methods. Bold shows the best value.

Explore similar work

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Aug 13, 2026Yi Shi, Huichao Xie, Yuqing Wang +7Multi-Source Image Fusion Infrared-Visible Image Fusion
Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1
Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared
Mar 9, 2026Yafei Zhang, Meng Ma, Huafeng Li +1Infrared-Visible Image Fusion Multi-Source Image Fusion
Faculty of Information Engineering and Automation, Kunming University of Science and Technology · Department of Biomedical Engineering, Hefei University of Technology
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Jul 27, 2026Minchong Chen, Xiaoyun Yuan, Minyu Cao +4Diffusion Models for Image Restoration Multi-Source Image Fusion
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China

Aug 13, 2026J/K move · Enter open · S save

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

Yi Shi, Huichao Xie, Yuqing Wang, Mingyu Wang+6

Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China+1

Multi-Source Image Fusion Infrared-Visible Image Fusion Multimodal Fusion

Open paper AskPDF arXiv

Infrared-visible image fusion (IVIF) is pivotal for multimodal perception, yet reconciling the inherent information disparity between thermal and textural features remains a fundamental challenge. Existing prior-guided methods often rely on static constraints that induce optimization conflicts or utilize extrinsic semantic priors from large-scale foundation models (e.g., CLIP/DINO), which frequently fail to exploit the intrinsic modality characteristics essential for high-fidelity fusion. To address these issues, we propose P2Fusion, a prior-guided distillation-based framework that reformulates IVIF via dual intrinsic prompts. Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators. Specifically, a Teach-to-Fuse mechanism provides dual-granularity progressive guidance, coupled with a Gated Dynamic Expert Recalibration (GDER) module for decoupled feature refinement. This design enables the network to adaptively mediate modal competition through expert specialization. Extensive experiments demonstrate that P2Fusion achieves state-of-the-art performance across five mainstream datasets. Notably, our framework demonstrates consistent performance advantages in fusion quality, achieving state-of-the-art results in 14 out of 20 key evaluation metrics across 5 benchmarks. Furthermore, it effectively contributes to the robustness of downstream perception, such as +3.2% mAP on MSRS, +0.5% mAP on M3FD and +0.9% mAP on DroneVehicle for object detection. Our code will be available at https://github.com/YiShi99/P2Fusion

Abstract

Explore similar work

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion