Bridging Human Evaluation to Infrared and Visible Image Fusion

Jinyuan Liu; Xingyuan Li; Qingyun Mei; Haoyuan Xu; Zhiying Jiang; Long Ma; Risheng Liu; Xin Fan

cs.CVMar 4, 2026

Bridging Human Evaluation to Infrared and Visible Image Fusion

Authors: Jinyuan Liu, Xingyuan Li, Qingyun Mei, Haoyuan Xu, Zhiying Jiang, Long Ma, Risheng Liu, Xin Fan

Organizations: School of Mechanical Engineering, Dalian University of Technology · College of Computer Science and Technology, Zhejiang University · School of Software Technology & DUT-RU International School of ISE, Dalian University of Technology · College of Information Science and Technology, Dalian Maritime University

Abstract

Infrared and visible image fusion (IVIF) integrates complementary modalities to enhance scene perception. Current methods predominantly focus on optimizing handcrafted losses and objective metrics, often resulting in fusion outcomes that do not align with human visual preferences. This challenge is further exacerbated by the ill-posed nature of IVIF, which severely limits its effectiveness in human perceptual environments such as security surveillance and driver assistance systems. To address these limitations, we propose a feedback reinforcement framework that bridges human evaluation to infrared and visible image fusion. To address the lack of human-centric evaluation metrics and data, we introduce the first large-scale human feedback dataset for IVIF, containing multidimensional subjective scores and artifact annotations, and enriched by a fine-tuned large language model with expert review. Based on this dataset, we design a domain-specific reward function and train a reward model to quantify perceptual quality. Guided by this reward, we fine-tune the fusion network through Group Relative Policy Optimization, achieving state-of-the-art performance that better aligns fused images with human aesthetics. Code is available at https://github.com/ALKA-Wind/EVAFusion.

Figures & tables

Figure 1: Illustration of our pipeline. The fine-tuning module is based on RLHF, integrating the ViT-based fusion-oriented reward model trained with the Human Feedback Dataset. A segmentation-assisted mechanism, based on GRPO, is introduced for fine-tuning, aiming to improve the quality of the fused images.

Figure 2: Workflow of the dataset collection. The dataset is processed and clustered to optimize representative images, followed by expert screening. Use fusion models to fuse $I_{vis}$ and $I_{inf}$ images, with experts scoring and annotating a subset as prior knowledge for aligning the GPT model. After GPT annotates all images, experts review the results, resulting in the human feedback IVIF dataset.

Figure 3: Overview of our collected dataset. (a) Data diversity: samples from multiple benchmark datasets; (b) Label diversity: each sample contains fine-grained scores across four quality dimensions and artifact heatmaps; (c) Scene diversity: covering key semantic categories including people, cars, buildings, roads, and more.

Figure 4: Qualitative comparison of our method with existing image fusion methods. From top to bottom: RoadScene, M 3 FD, TNO.

	TNO				RoadScene				M 3 FD
Method	CC $\uparrow$	PSNR $\uparrow$	Qabf $\uparrow$	SSIM $\uparrow$	CC $\uparrow$	PSNR $\uparrow$	Qabf $\uparrow$	SSIM $\uparrow$	CC $\uparrow$	PSNR $\uparrow$	Qabf $\uparrow$	SSIM $\uparrow$
CDDFuse [ 68 ]	0.47	63.49	0.49	0.54	0.52	59.84	0.43	0.45	0.62	63.14	0.61	0.51
DDFM [ 70 ]	0.44	63.40	0.17	0.03	0.41	59.40	0.15	0.03	0.65	64.82	0.48	0.50
EMMA [ 69 ]	0.45	63.10	0.42	0.49	0.50	59.71	0.42	0.44	0.58	63.35	0.57	0.46
LRRNet [ 17 ]	0.41	62.38	0.31	0.41	0.53	59.56	0.39	0.36	0.63	64.18	0.48	0.38
MRFS [ 64 ]	0.49	61.78	0.41	0.47	0.50	61.46	0.39	0.45	0.46	61.24	0.54	0.44

Table 1: Quantitative comparison of IVIF between our method and state-of-the-art methods on TNO, RoadScene, and M 3 FD datasets. Red and blue represent the best and second-best results.

Figure 5: Preference ranking heatmaps on three datasets. Yellow represents the most preferred, and purple represents the least preferred.

	TNO		RoadScene		M 3 FD
Method	NIQE $\downarrow$	BRI* $\downarrow$	NIQE $\downarrow$	BRI* $\downarrow$	NIQE $\downarrow$	BRI* $\downarrow$
CDDFuse [ 68 ]	7.43	37.62	3.26	24.74	4.37	32.88
DDFM [ 70 ]	7.54	34.28	3.32	27.19	5.05	38.78
EMMA [ 69 ]	6.33	26.08	4.93	22.20	5.45	35.16
LRRNet [ 17 ]	9.31	38.68	4.42	29.45	4.33	37.68
MRFS [ 64 ]	7.89	25.95	3.81	27.07	4.37	34.52

Table 2: Quantitative comparison of IVIF between our method and other methods using no-reference image quality assessment metrics. Red and blue represent the best and second-best results.

Figure 6: Qualitative comparison of our method with the fusion images generated by different fusion methods on the FMB dataset. Our approach performs the best segmentation results.

Figure 7: Qualitative comparison of our method with the fusion images generated by different fusion methods on the M 3 FD dataset. Our method achieves the best detection results.

	FMB							M 3 FD
Method	Person	Car	Sky	Bus	Motor	Pole	mIoU	People	Car	Bus	Light	Motor	Trunk	mAP
CDDFuse [ 68 ]	63.38	81.59	93.68	56.60	26.05	41.04	55.39	73.06	82.32	67.64	40.03	31.61	55.07	58.29
DDFM [ 70 ]	63.26	81.64	94.61	63.34	25.71	42.60	56.64	71.84	83.42	76.42	45.61	34.19	55.47	61.16
EMMA [ 69 ]	63.10	80.95	93.85	62.90	26.96	40.92	56.00	72.11	82.32	75.65	36.24	33.33	54.11	58.96
LRRNet [ 17 ]	63.58	81.62	94.06	64.28	27.47	40.59	56.37	71.27	83.56	76.41	38.89	36.15	54.25	60.09
MRFS [ 64 ]	63.79	81.13	94.01	56.76	27.33	41.35	54.61	68.95	81.53	65.84	35.80	32.35	55.84	56.72

Table 3: Quantitative comparison of our method and existing image fusion methods on downstream segmentation and detection tasks on FMB and M 3 FD. Red and blue represent the best and second-best results.

Figure 8: Comparison between w/o score, w/o heatmap, w/o SAM, and our method.

	TNO		RoadScene		M ${}^{\scalebox{0.7}{3}}$ FD
Method	CC $\uparrow$	PSNR $\uparrow$	CC $\uparrow$	PSNR $\uparrow$	CC $\uparrow$	PSNR $\uparrow$
w/o Score	0.50	64.21	0.51	60.97	0.58	64.81
w/o Heatmap	0.50	65.17	0.54	61.21	0.60	64.93
w/o SAM	0.48	65.03	0.52	60.92	0.57	63.02
Ours	0.51	65.43	0.56	61.84	0.65	65.09

Table 4: Comparison of ablation studies removed key components.

Figure 9: Comparison between the baseline, DPO [ 44 ] , PPO [ 45 ] , and our method.

	TNO		RoadScene		M ${}^{\scalebox{0.7}{3}}$ FD
Method	CC $\uparrow$	PSNR $\uparrow$	CC $\uparrow$	PSNR $\uparrow$	CC $\uparrow$	PSNR $\uparrow$
Baseline	0.48	63.83	0.49	59.66	0.55	62.87
DPO	0.50	63.98	0.49	61.50	0.57	63.74
PPO	0.51	63.59	0.53	61.17	0.59	64.32
Ours	0.51	65.43	0.56	61.84	0.65	65.09

Table 5: Comparison of ablation studies with different fine-tuning methods.

Explore similar work

EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment
Apr 3, 2026Chunyang Cheng, Tianyang Xu, Xiao-Jun Wu +4Image Quality Assessment Infrared-Visible Image Fusion
C. Cheng and Z. Tang are with the Wuxi School of Medicine and also with the School of Food Science and Technology, Jiangnan University, Wuxi 214122, P.R. China. · T. Xu, X.-J. Wu, T. Zhou and H. Li are with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, P.R. China. · Josef Kittler is with the Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford GU2 7XH, U.K.
Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment
May 7, 2026Yuchen Guo, Junli Gong, Yao Lu +3Infrared-Visible Image Fusion Learning to Rank
Northwestern University · Northeastern University · University of Washington +2
P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Aug 13, 2026Yi Shi, Huichao Xie, Yuqing Wang +7Multi-Source Image Fusion Infrared-Visible Image Fusion
Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1

Apr 3, 2026J/K move · Enter open · S save

EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment

Chunyang Cheng, Tianyang Xu, Xiao-Jun Wu, Tao Zhou+3

C. Cheng and Z. Tang are with the Wuxi School of Medicine and also with the School of Food Science and Technology, Jiangnan University, Wuxi 214122, P.R. China. · T. Xu, X.-J. Wu, T. Zhou and H. Li are with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, P.R. China. · Josef Kittler is with the Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford GU2 7XH, U.K.

Image Quality Assessment Infrared-Visible Image Fusion

Open paper AskPDF arXiv

Evaluation is essential in image fusion research, yet most existing metrics are directly borrowed from other vision tasks without proper adaptation. These traditional metrics, often based on complex image transformations, not only fail to capture the true quality of the fusion results but also are computationally demanding. To address these issues, we propose a unified evaluation framework specifically tailored for image fusion. At its core is a lightweight network designed efficiently to approximate widely used metrics, following a divide-and-conquer strategy. Unlike conventional approaches that directly assess similarity between fused and source images, we first decompose the fusion result into infrared and visible components. The evaluation model is then used to measure the degree of information preservation in these separated components, effectively disentangling the fusion evaluation process. During training, we incorporate a contrastive learning strategy and inform our evaluation model by perceptual scene assessment provided by a large language model. Last, we propose the first consistency evaluation framework, which measures the alignment between image fusion metrics and human visual perception, using both independent no-reference scores and downstream tasks performance as objective references. Extensive experiments show that our learning-based evaluation paradigm delivers both superior efficiency (up to 1,000 times faster) and greater consistency across a range of standard image fusion benchmarks. Our code will be publicly available at https://github.com/AWCXV/EvaNet.

Abstract

Explore similar work

EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment

Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation