Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

Yafei Zhang; Meng Ma; Huafeng Li; Yu Liu

cs.CVMar 9, 2026

Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

Authors: Yafei Zhang, Meng Ma, Huafeng Li, Yu Liu

Organizations: Faculty of Information Engineering and Automation, Kunming University of Science and Technology · Department of Biomedical Engineering, Hefei University of Technology

Abstract

Infrared-visible (IR-VIS) image fusion is vital for perception and security, yet most methods rely on the availability of both modalities during training and inference. When the infrared modality is absent, pixel-space generative substitutes become hard to control and inherently lack interpretability. We address missing-IR fusion by proposing a dictionary-guided, coefficient-domain framework built upon a shared convolutional dictionary. The pipeline comprises three key components: (1) Joint Shared-dictionary Representation Learning (JSRL) learns a unified and interpretable atom space shared by both IR and VIS modalities; (2) VIS-Guided IR Inference (VGII) transfers VIS coefficients to pseudo-IR coefficients in the coefficient domain and performs a one-step closed-loop refinement guided by a frozen large language model as a weak semantic prior; and (3) Adaptive Fusion via Representation Inference (AFRI) merges VIS structures and inferred IR cues at the atom level through window attention and convolutional mixing, followed by reconstruction with the shared dictionary. This encode-transfer-fuse-reconstruct pipeline avoids uncontrolled pixel-space generation while ensuring prior preservation within interpretable dictionary-coefficient representation. Experiments under missing-IR settings demonstrate consistent improvements in perceptual quality and downstream detection performance. To our knowledge, this represents the first framework that jointly learns a shared dictionary and performs coefficient-domain inference-fusion to tackle missing-IR fusion. The source code is publicly available at https://github.com/harukiv/DCMIF.

Figures & tables

Figure 1: Comparison between existing methods and ours. For infrared-missing cross-modal fusion, (a) existing methods adopt a multi-stage framework that generates infrared images before fusion, while (b) we propose a single-stage framework that directly infers infrared features from visible images without generating infrared images.

Figure 2: Architecture of the proposed framework. It integrates three modules (JSRL, VGII, and AFRI) to form a closed-loop pipeline of encoding, inference, fusion, and reconstruction. JSRL learns a shared dictionary for cross-modal alignment, VGII predicts latent infrared coefficients from visible ones, and AFRI fuses both modalities in the coefficient domain to reconstruct consistent, high-quality images under missing-IR conditions.

Figure 4: Visual comparison results on the FLIR, MSRS, and KAIST datasets for fused image quality.

Methods	MSRS						FLIR						KAIST
Methods	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
U2Fusion	2.308	0.840	24.692	6.145	0.483	6.288	4.645	1.353	48.215	6.916	0.410	11.302	3.262	1.626	34.797	6.092	0.505	9.399
TarDAL	2.189	2.183	23.251	6.059	0.417	6.379	3.722	0.643	39.761	7.393	0.420	9.392	1.815	1.273	19.393	6.282	0.431	5.355
CDDFuse	4.818	1.609	51.368	7.321	0.565	14.250	5.079	0.726	52.633	6.766	0.454	13.346	3.167	1.298	34.043	7.122	0.497	9.194
LRRNet	3.621	2.837	38.519	6.930	0.461	10.731	4.648	1.798	49.145	7.063	0.481	12.449	2.939	1.475	31.707	7.032	0.463	8.213
IVFWSR	3.360	1.337	36.254	6.798	0.457	9.241	3.386	1.266	36.392	7.365	0.400	8.144	2.672	0.868	28.868	6.959	0.443	7.743

Table 1: Quantitative comparison of the fusion results of our method and existing methods on the MSRS, FLIR, and KAIST datasets. The best and second-best performances are highlighted with Red and Blue backgrounds, respectively.

Figure 5: Qualitative comparison of object detection and semantic segmentation on the $\text{M}^{3}$ FD and FMB datasets, respectively.

Methods	$\text{M}^{3}$ FD							FMB
Methods	people	car	bus	motor	truck	lamp	mAP	wall	building	sky	floor	tree	road	mIoU
VIS	0.864	0.964	0.990	0.926	0.908	0.970	0.931	89.461	58.723	85.666	33.918	67.091	95.984	61.794
U2Fusion	0.841	0.959	0.989	0.918	0.878	0.970	0.926	89.523	57.843	85.234	33.852	65.076	94.965	60.829
TarDAL	0.896	0.976	0.993	0.951	0.913	0.983	0.952	87.462	52.695	77.013	31.397	53.117	90.756	56.194
CDDFuse	0.883	0.974	0.991	0.943	0.903	0.976	0.945	89.811	57.901	87.383	32.988	67.748	96.055	61.649
LRRNet	0.898	0.978	0.993	0.958	0.904	0.986	0.953	89.705	59.081	86.685	34.596	66.974	95.678	62.942

Table 2: Quantitative comparison of the object detection and semantic segmentation results of our method and existing methods on the

\text{M}^{3}

FD and FMB datasets, respectively. Best results are highlighted in red and second-best in blue .

Methods	Dictionary	LLM	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
Model $\mathrm{I}$	✗	✗	3.320	1.452	45.531	6.058	0.396	9.238
Model $\mathrm{II}$	✓	✗	4.363	1.046	48.351	6.578	0.434	11.936
Model $\mathrm{III}$	✗	✓	4.256	0.619	48.154	6.423	0.418	11.175
Ours	✓	✓	4.518	0.596	48.784	6.639	0.435	12.554

Table 3: Quantitative comparison between our model and its ablated variants. The best result for each metric is highlighted in Red .

Figure 6: Qualitative comparison between our model and its ablated variants.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Kernel size	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
$3\times 3$	4.156	1.357	44.882	6.545	0.430	11.195
$5\times 5$	4.518	0.596	48.784	6.639	0.435	12.554
$7\times 7$	4.499	1.298	48.560	6.627	0.356	12.300

Appendix

Table 4: Influence of the dictionary kernel size on model perfomance. The best performance for each metric is marked with Red .

N	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	GFLOPS
1	4.518	0.596	48.784	6.639	0.435	12.554	63.023
2	4.529	0.565	49.096	6.598	0.432	12.671	120.795
3	4.485	0.679	48.641	6.705	0.432	12.265	178.567

Appendix

Table 5: Influence of the number of IV-DLBs on model perfomance. The best performance for each metric is marked with Red .

Prompt	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
No Prompt	4.183	1.016	47.166	6.215	0.401	11.694
Simplified Prompt	4.431	0.610	48.687	6.613	0.419	12.186
Our Prompt	4.518	0.596	48.784	6.639	0.435	12.554

Appendix

Table 6: Influence of the textual prompt on model perfomance. The best performance for each metric is marked with Red .

Figure 7: Two different types of text prompts. (a) is our prompt used in the experiment, while (b) is the simplified version.

Model	Param (M)	FLOPs (G)	Time (s)	Qcb $\uparrow$
E+U2Fusion	48.064	710.385	0.103	0.368
E+TarDAL	46.892	96.768	0.052	0.304
E+CDDFuse	47.784	194.176	0.059	0.429
E+LRRNet	46.644	80.347	0.026	0.390
E+IVFSWR	60.098	936.753	0.176	0.422
E+CoCoNet	55.709	118.866	0.075	0.409

Appendix

Table 7: Complexity analysis of fusion models with missing infrared modality. All parameters listed in the table represent the learnable parameters of the models. The FLOPs and inference time for each model are calculated using input images of size

256\times 256

. ’E’ denotes the EGGAN method, while ’P’ denotes the PID method.

Figure 8: Visual comparison of different fusion methods on the FLIR, MSRS, and KAIST datasets, where the infrared images are generated from visible images using the EGGAN method.

Figure 9: Visual comparison of different fusion methods on the FLIR, MSRS, and KAIST datasets, where the infrared images are generated from visible images using the PID method.

Datasets	MSRS						FLIR						KAIST
Methods	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
U2Fusion	3.289	3.309	34.980	5.447	0.327	9.310	2.320	1.453	24.107	5.045	0.368	6.517	2.425	2.734	25.699	5.489	0.388	6.259
TarDAL	3.205	3.458	34.550	6.640	0.363	8.602	2.341	2.756	25.123	6.536	0.304	6.898	1.916	1.728	20.399	6.197	0.382	5.807
CDDFuse	3.644	2.839	38.606	6.513	0.318	11.581	3.004	0.515	31.810	6.406	0.429	8.746	2.944	4.315	31.419	6.225	0.374	8.835
LRRNet	3.979	4.924	42.225	6.617	0.340	12.258	3.905	1.993	41.323	6.866	0.390	11.946	3.086	2.443	33.250	6.642	0.418	8.862
IVFWSR	3.229	4.218	34.902	6.461	0.378	8.608	3.000	3.704	32.530	6.349	0.422	7.818	2.635	2.631	28.661	6.462	0.414	6.959

Appendix

Table 8: Quantitative comparison of different fusion methods on the FLIR, MSRS, and KAIST datasets, where the infrared images are generated from visible images using the EGGAN method. The best and second-best performances are highlighted with Red and Blue backgrounds, respectively.

Datasets	MSRS						FLIR						KAIST
Methods	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$	AG $\uparrow$	CE $\downarrow$	EI $\uparrow$	EN $\uparrow$	Qcb $\uparrow$	SF $\uparrow$
U2Fusion	3.074	1.668	32.810	6.047	0.428	9.168	3.277	1.353	33.900	6.576	0.388	8.329	2.533	0.828	27.040	6.180	0.459	6.911
TarDAL	1.938	2.342	20.524	6.030	0.378	5.654	2.516	0.643	26.795	7.055	0.282	6.860	1.937	1.268	20.667	6.302	0.427	5.766
CDDFuse	4.821	1.651	51.500	7.191	0.466	14.142	3.837	0.726	39.804	6.517	0.403	10.254	3.349	1.361	35.909	7.064	0.488	9.859
LRRNet	3.815	2.837	40.509	7.049	0.445	11.150	3.970	1.798	42.038	6.955	0.408	11.154	3.125	1.432	33.685	7.017	0.439	8.740
IVFWSR	3.994	1.342	43.035	7.046	0.470	11.054	2.976	1.266	31.999	7.180	0.377	7.546	2.878	0.857	31.010	6.962	0.442	8.313

Appendix

Table 9: Quantitative comparison of different fusion methods on the FLIR, MSRS, and KAIST datasets, where the infrared images are generated from visible images using the PID method. The best and second-best performances are highlighted with Red and Blue backgrounds, respectively.

Explore similar work

FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration
Mar 8, 2026Congcong Bian, Haolong Ma, Hui Li +4Infrared-Visible Image Fusion Cross-Modal Image Registration
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China
P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Aug 13, 2026Yi Shi, Huichao Xie, Yuqing Wang +7Multi-Source Image Fusion Infrared-Visible Image Fusion
Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Jul 27, 2026Minchong Chen, Xiaoyun Yuan, Minyu Cao +4Diffusion Models for Image Restoration Multi-Source Image Fusion
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China

Mar 8, 2026J/K move · Enter open · S save

FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration

Congcong Bian, Haolong Ma, Hui Li, Zhongwei Shen+3

School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China

Infrared-Visible Image Fusion Cross-Modal Image Registration

Open paper AskPDF arXiv

Spatial registration across different visual modalities is a critical but formidable step in multi-modality image fusion for real-world perception. Although several methods are proposed to address this issue, the existing registration-based fusion methods typically require extensive pre-registration operations, limiting their efficiency. To overcome these limitations, a general cross-modality registration method guided by visual priors is proposed for infrared and visible image fusion task, termed FusionRegister. Firstly, FusionRegister achieves robustness by learning cross-modality misregistration representations rather than forcing alignment of all differences, ensuring stable outputs even under challenging input conditions. Moreover, FusionRegister demonstrates strong generality by operating directly on fused results, where misregistration is explicitly represented and effectively handled, enabling seamless integration with diverse fusion methods while preserving their intrinsic properties. In addition, its efficiency is further enhanced by serving the backbone fusion method as a natural visual prior provider, which guides the registration process to focus only on mismatch regions, thereby avoiding redundant operations. Extensive experiments on three datasets demonstrate that FusionRegister not only inherits the fusion quality of state-of-the-art methods, but also delivers superior detail alignment and robustness, making it highly suitable for infrared and visible image fusion method. The code will be available at https://github.com/bociic/FusionRegister.

Abstract

Explore similar work

FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion