Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion

Yanglin Deng; Tianyang Xu; Chunyang Cheng; Hui Li; Xiao-jun Wu; Josef Kittler

cs.CVMar 23, 2026

Beyond Strict Pairing: Arbitrarily Paired Training for High-Performance Infrared and Visible Image Fusion

Authors: Yanglin Deng, Tianyang Xu, Chunyang Cheng, Hui Li, Xiao-jun Wu, Josef Kittler

Organizations: School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey

Abstract

Infrared and visible image fusion(IVIF) combines complementary modalities while preserving natural textures and salient thermal signatures. Existing solutions predominantly rely on extensive sets of rigidly aligned image pairs for training. However, acquiring such data is often impractical due to the costly and labour-intensive alignment process. Besides, maintaining a rigid pairing setting during training restricts the volume of cross-modal relationships, thereby limiting generalisation performance. To this end, this work challenges the necessity of Strictly Paired Training Paradigm (SPTP) by systematically investigating UnPaired and Arbitrarily Paired Training Paradigms (UPTP and APTP) for high-performance IVIF. We establish a theoretical objective of APTP, reflecting the complementary nature between UPTP and SPTP. More importantly, we develop a practical framework capable of significantly enriching cross-modal relationships even with severely limited and unaligned training data. To validate our propositions, three end-to-end lightweight baselines, alongside a set of innovative loss functions, are designed to cover three classic frameworks (CNN, Transformer, GAN). Comprehensive experiments demonstrate that the proposed APTP and UPTP are feasible and capable of training models on a severely limited and content-inconsistent infrared and visible dataset, achieving performance comparable to that of a dataset 100 $\times$ larger in SPTP. This finding fundamentally alleviates the cost and difficulty of data collection while enhancing model robustness from the data perspective, delivering a feasible solution for IVIF studies. The code is available at blue{https://github.com/yanglinDeng/IVIF_unpair}.

Figures & tables

Figure 1: Motivation of proposed two training paradigms. (a) Compared with the strictly-paired paradigm, the two proposed paradigms lift the requirement for spatiotemporally aligned pairs, expanding the trainable pairs to $N\times(N-1)$ and $N^{2}$ , respectively, reducing data-collection costs and improving generalisation. (b) Performance improvement pattern: greater enhancement from $N^{2}$ for Smaller $N$ .

Figure 2: Training convergence and performance comparison between SPTP and UPTP on three baselines.

Figure 3: Evolution of fusion results during CNN baseline training (SPTP v.s. UPTP)

Figure 4: Fused images from baselines trained on SPTP and APTP

Figure 5: Explanation of relationship diversity.

Figure 6: Qualitative comparison of three baselines (from top to bottom: CNN, Transformer, and GAN, respectively) under different training data settings on the LLVIP, M3FD, and MSRS test sets.

Datasets	Mode	Quantity			CNN					Transformer					GAN
Datasets	Mode	Paired Base	w/o Expanded	Trainable	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM
MSRS	SPTP	150	$\times$	150	6.54	2.57	0.88	0.60	0.96	6.53	2.44	0.86	0.59	0.98	6.52	2.17	0.75	0.51	0.90
	UPTP	150	$\times$	150	6.59	2.62	0.90	0.61	0.97	6.55	2.49	0.88	0.60	0.98	6.60	2.09	0.75	0.51	0.91
	UPTP	150	$\checkmark$	7500	6.56	2.69	0.92	0.65	0.98	6.56	2.69	0.92	0.65	0.98	6.58	2.52	0.90	0.62	0.99
	UPTP	150	$\checkmark$	15000	6.57	2.70	0.93	0.65	0.99	6.55	2.70	0.92	0.65	0.98	6.58	2.62	0.92	0.63	0.98
	SPTP	15000	$\times$	15000	6.59	2.70	0.93	0.65	0.98	6.55	2.78	0.93	0.66	0.97	6.57	2.60	0.92	0.64	0.99

Table 1: Qualitative generalisation results on the LLVIP (250 pairs), MSRS (361 pairs) and M 3 FD (420 pairs) test sets for three baseline models trained on MSRS with different training settings. Bold fonts indicate higher than their corresponding SPTP baseline groups.

Figure 7: Visualisation of fused images: cross-dataset unpaired training v.s. single-dataset strictly-paired training models.

Mode	Quantity			Infrared	Visible	EN	MI	VIF	Qabf	SSIM
Mode	PB	w/o E	T	Infrared	Visible	EN	MI	VIF	Qabf	SSIM
SPTP	150	$\times$	150	MSRS	MSRS	6.54	2.57	0.88	0.60	0.96
	150	$\times$	150	M3FD	M3FD	6.52	2.37	0.84	0.58	0.90
	150	$\times$	150	LLVIP	LLVIP	6.63	2.43	0.90	0.61	0.99
UPTP	150	$\checkmark$	7500	M3FD	MSRS	6.57	2.66	0.93	0.66	1.01
	150	$\checkmark$	7500	M3FD	LLVIP	6.59	2.63	0.93	0.65	1.01

Table 2: Qualitative results on MSRS of CNN baseline trained on different training sets. Bold indicates that the model performs better than the one trained on its original paired data.

Mode	Quantity			EN	MI	VIF	Qabf	SSIM	Avg. Imp.
Mode	PB	w/o E	T	EN	MI	VIF	Qabf	SSIM	Avg. Imp.
SPTP	15	$\times$	15	6.29	2.25	0.74	0.52	0.65	$-$
APTP	15	$\checkmark$	225	6.45	2.56	0.80	0.56	0.73	8.89%
SPTP	30	$\times$	30	6.48	2.48	0.80	0.59	0.93	$-$
APTP	30	$\checkmark$	900	6.55	2.62	0.90	0.61	0.96	5.17%
SPTP	60	$\times$	60	6.54	2.57	0.88	0.60	0.96	$-$

Table 3: Quantitative results on MSRS of comparing the improvement of APTP on different basic strictly-paired images.

	Venue	MB	LLVIP					MSRS					RoadScene					TNO
	Venue	MB	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM	EN	MI	VIF	Qabf	SSIM
DeF [ 21 ]	22’ ECCV	30.04	7.25	2.33	0.74	0.43	0.83	3.34	2.15	0.75	0.51	0.93	6.80	2.08	0.50	0.38	0.86	6.94	1.81	0.53	0.34	0.91
LRR [ 20 ]	23’ TPAMI	0.19	6.67	1.65	0.56	0.47	0.66	6.19	2.03	0.54	0.45	0.71	7.12	2.09	0.51	0.38	0.71	6.90	1.74	0.56	0.37	0.83
Meta [ 50 ]	23’ CVPR	3.10	7.14	1.16	0.61	0.29	0.60	6.37	1.16	0.71	0.48	0.78	7.17	1.60	0.51	0.36	0.75	7.09	1.12	0.57	0.35	0.80
DDF [ 53 ]	23’ ICCV	2108	7.17	1.78	0.58	0.23	0.58	6.17	1.89	0.74	0.47	0.91	7.42	2.13	0.57	0.42	0.88	6.83	1.46	0.56	0.36	0.90
Cross [ 19 ]	24’ Inf.	79.40	6.47	1.79	0.69	0.44	0.71	6.49	2.17	0.84	0.56	0.88	7.25	2.13	0.58	0.39	067	6.83	2.10	0.74	0.45	0.88

Table 4: Qualitative results of three baseline models and several SOTA methods. Bold shows the best value.

Explore similar work

Aug 13, 2026

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

Infrared-visible image fusion (IVIF) is pivotal for multimodal perception, yet reconciling the inherent information disparity between thermal and textural features remains a fundamental challenge. Existing prior-guided methods often rely on static constraints that induce optimization conflicts or utilize extrinsic semantic priors from large-scale foundation models (e.g., CLIP/DINO), which frequently fail to exploit the intrinsic modality characteristics essential for high-fidelity fusion. To address these issues, we propose P2Fusion, a prior-guided distillation-based framework that reformulates IVIF via dual intrinsic prompts. Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators. Specifically, a Teach-to-Fuse mechanism provides dual-granularity progressive guidance, coupled with a Gated Dynamic Expert Recalibration (GDER) module for decoupled feature refinement. This design enables the network to adaptively mediate modal competition through expert specialization. Extensive experiments demonstrate that P2Fusion achieves state-of-the-art performance across five mainstream datasets. Notably, our framework demonstrates consistent performance advantages in fusion quality, achieving state-of-the-art results in 14 out of 20 key evaluation metrics across 5 benchmarks. Furthermore, it effectively contributes to the robustness of downstream perception, such as +3.2% mAP on MSRS, +0.5% mAP on M3FD and +0.9% mAP on DroneVehicle for object detection. Our code will be available at https://github.com/YiShi99/P2Fusion

Yi Shi, Huichao Xie, Yuqing Wang +7

Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1

Multi-Source Image Fusion Infrared-Visible Image Fusion Multimodal Fusion

Ask PDF

Mar 9, 2026

Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

Infrared-visible (IR-VIS) image fusion is vital for perception and security, yet most methods rely on the availability of both modalities during training and inference. When the infrared modality is absent, pixel-space generative substitutes become hard to control and inherently lack interpretability. We address missing-IR fusion by proposing a dictionary-guided, coefficient-domain framework built upon a shared convolutional dictionary. The pipeline comprises three key components: (1) Joint Shared-dictionary Representation Learning (JSRL) learns a unified and interpretable atom space shared by both IR and VIS modalities; (2) VIS-Guided IR Inference (VGII) transfers VIS coefficients to pseudo-IR coefficients in the coefficient domain and performs a one-step closed-loop refinement guided by a frozen large language model as a weak semantic prior; and (3) Adaptive Fusion via Representation Inference (AFRI) merges VIS structures and inferred IR cues at the atom level through window attention and convolutional mixing, followed by reconstruction with the shared dictionary. This encode-transfer-fuse-reconstruct pipeline avoids uncontrolled pixel-space generation while ensuring prior preservation within interpretable dictionary-coefficient representation. Experiments under missing-IR settings demonstrate consistent improvements in perceptual quality and downstream detection performance. To our knowledge, this represents the first framework that jointly learns a shared dictionary and performs coefficient-domain inference-fusion to tackle missing-IR fusion. The source code is publicly available at https://github.com/harukiv/DCMIF.

Yafei Zhang, Meng Ma, Huafeng Li +1

Faculty of Information Engineering and Automation, Kunming University of Science and Technology · Department of Biomedical Engineering, Hefei University of Technology

Infrared-Visible Image Fusion Multi-Source Image Fusion Dictionary Learning

Ask PDF

Jul 27, 2026

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion

Mobile infrared-visible imaging typically pairs a compact infrared sensor with a high-resolution visible camera for complementary perception. While cross-sensor misalignment caused by different optics, viewpoints, fields of view, and exposure timings hinders practical deployment. In this paper, we propose BeyondFusion, a unified latent diffusion framework for calibration-free visible-guided infrared super-resolution and infrared-visible fusion tasks. The proposed framework supports both task-specific training and joint training where two tasks are optimized and executed as two readouts of the same generative process. Instead of relying on explicit registration or geometric warping, BeyondFusion introduces a cross-modal self-aligning (CMSA) module into the denoising U-Net. CMSA reorganizes infrared and visible latent tokens into a shared attention space to learn content-adaptive cross-modal correspondence during the denoising process. Together with misalignment augmentation module, the model is facilitated to exploit visible structural and semantic cues while preserving thermal consistency, enabling high-frequency infrared reconstruction and informative fused-image generation under uncalibrated conditions. Extensive experiments on public benchmarks and a mobile infrared-visible imaging system show strong performance across aligned inputs, low-resolution infrared observations, synthetic misalignments, and real mobile captures with unsynchronized sensors. Ablation studies, unified training analysis, and downstream pedestrian detection further validate the effectiveness of BeyondFusion for calibration-free multimodal imaging.

Minchong Chen, Xiaoyun Yuan, Minyu Cao +4

MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China

Diffusion Models for Image Restoration Multi-Source Image Fusion Infrared-Visible Image Fusion Cross-Modal Alignment Image Super-Resolution

Ask PDF