EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment

Chunyang Cheng; Tianyang Xu; Xiao-Jun Wu; Tao Zhou; Hui Li; Zhangyong Tang; Josef Kittler

Fig. 1: An illustration of the consistency and the efficiency issues (sub-figure (a) and (b)) of existing image fusion metrics. Image fusion evaluation relies heavily on traditional signal processing techniques ( e.g. , discrete cosine transform (DCT) and wavelet transform) or metrics adapted from other vision tasks. Without appropriate adjustment, these metrics lack consistency, i.e. , better visualisation cannot always correspond higher metric values. Moreover, the evaluation phase in current image fusion settings incurs high processing time costs. The proposed EvaNet addresses these issues effectively by significantly reducing the evaluation time and enhancing consistency.

Fig. 2: The speed (milliseconds per image) imbalance between inference and evaluation in image fusion. Traditional metrics rely on separate, computationally intensive procedures to perform different assessments, significantly slowing down the evaluation phase. In contrast, the proposed EvaNet generates multiple evaluation results simultaneously within a single forward pass, offering acceleration by a factor of up to 1000, and as will be demonstrated, also considerably improved metric consistency.

Category	IVIF Algorithm	IVIF Metric	MFIF Metric
Traditional	[ 19 ] , [ 20 ] , [ 21 ] …	[ 22 ] , Qabf, VIF…	SSIM, PSNR, …
CNN	[ 8 ] , [ 23 ] , [ 24 ] …	Unexplored	2024’ TPAMI [ 18 ]
Transformer	[ 25 ] , [ 26 ] , [ 27 ] …	-	-
Mamba	[ 9 ] , [ 28 ] …	-	-
Diffusion	[ 29 ] , [ 30 ] …	-	-

TABLE I: Statistics of recent Infrared and Visible Image Fusion (IVIF) and Multi-Focus Image Fusion (MFIF) algorithms and metrics.

Fig. 3: Overview of the proposed EvaNet framework. Our method replaces traditional image fusion assessment processes by a lightweight learning-based network to significantly improve evaluation efficiency. In addition, a divide-and-conquer strategy is used to disentangle and independently measure the information preserved from each source modality. The environment branch, as part of the three-branch design, introduces an adaptive penalty mechanism to mitigate modality imbalance in challenging fusion scenarios.

Fig. 4: The network architecture of the proposed EvaNet (see Sec. III-B ). The model consists of two main components. The left part shows the modality-specific decomposition, implemented using two lightweight information probes [ 11 ] . The right part illustrates the surrogate metric prediction process, consisting of three branches: two modality branches correspond to the infrared and visible inputs, while the environment branch, guided by a large language model (LLM), predicts an adaptive weighting factor (

ENV

) to eliminate any modality imbalance during evaluation.

Fig. 5: An illustration of perceptual scene environment assessment. Left: some examples showing that the LLVIP-train dataset contains multiple image pairs with limited scene diversity, where foreground objects vary but the background remains consistent. Right: environment label generation process using a Generative Pre-trained Transformer (GPT) [ 45 ] , which estimates scene conditions such as illumination and obscuration to guide the training of the environment branch in EvaNet.

Fig. 6: An overview of the training strategy based on contrastive learning. The decomposed infrared or visible image (obtained from the fusion result, e.g. ,

\hat{I}^{u}_{\textrm{vis}}

) and its corresponding original input (

I^{u}_{\textrm{vis}}

) form a positive sample pair, guiding the model to learn the similarity range of relevant images. In contrast, the original image paired with an unrelated randomly selected image (

I^{v}_{\textrm{vis}}

) forms a negative sample, encouraging the network to suppress metric predictions for unrelated content.

Setting	Traditional Metrics	EvaNet (Ours)
Platform	MATLAB	PyTorch
CPU	Intel i7-6850K	Intel i7-6850K
GPU	–	NVIDIA RTX 3090
Batch size	1	1
Image resolution	640 $\times$ 512	640 $\times$ 512
I/O time included	No	No

TABLE II: Comparison of runtime evaluation settings between traditional image fusion metrics and the proposed EvaNet. All experiments are conducted under an identical workstation.

Dataset	PSNR	CC	Qabf	VIF	SSIM	$\textrm{FMI}_{\textrm{pixel}}$	$\textrm{FMI}_{\textrm{dct}}$	$\textrm{FMI}_{\textrm{w}}$	Sum (8 metrics)	EvaNet (8 metrics)
LLVIP-test (part)	0.2s	1.0s	27.2s	330.4s	9.6s	516.1s	2770.4s	4039.4s	7694.3s ( $>$ 2 hours)	1.6s ( $<$ 1 min)
LLVIP-test (all)	2.9s	13.5s	341.3s	4560.6s	111.5s	8121.4s	38571.3s	51094.5s	102817.0s ( $>$ 24 hours)	10.8s ( $<$ 1 min)

TABLE III: The speed (in seconds) of generating the assessment scores using eight different metrics and our EvaNet, measured across part (250 pairs) and all fused images (3463 pairs) from the LLVIP dataset. ( Bold : Best performance)

Method	Setting	Manner	Efficiency	Model Size
Vanilla (N metrics)	Traditional	N-Exe.	30 s / img	-
TextAttn (N metrics)	Traditional	2N-Exe.	60 s / img	1552.67 MB
EvaNet (N metrics)	Leaning-based	1 Exe.	6 ms / img	1.15 MB

TABLE IV: A summary of the computational characteristics of existing image fusion evaluation methods when delivering

N

assessment metrics.

Fig. 7: Typical and zero-input decomposition results produced by the Information Probe. (a) and (b) show successful disentanglement of modality-specific content from two representative fused images with different degrees of thermal information preservation. (c) presents the zero-decomposition case, where an all-zero input yields two clean blank outputs without artefacts, verifying the robustness of the probe and its non-reliance on memorised patterns. At the right side of (a), we present intuitive prediction results of our EvaNet for 8 image fusion assessments, i.e. , VIF, Qabf, SSIM, CC, PSNR,

\textrm{FMI}_{\textrm{dct}}

,

\textrm{FMI}_{\textrm{wave}}

, and

\textrm{FMI}_{\textrm{pixel}}

, for the decomposed image and the source input.

Setting	$\Delta$ vs. (a)	$\textrm{VIF}^{*}$	$\textrm{Qabf}^{*}$	$\textrm{SSIM}^{*}$	$\textrm{CC}^{*}$	$\textrm{PSNR}^{*}$	$\textrm{FMI}_{\textrm{d}}^{*}$	$\textrm{FMI}_{\textrm{w}}^{*}$	$\textrm{FMI}_{\textrm{p}}^{*}$
(a) Traditional	-	0.728	0.685	0.650	0.635	0.611	0.619	0.603	0.655
(b) Dec. only	2.27%	0.673	0.620	0.623	0.664	0.727	0.713	0.647	0.614
(c) Con. only	-0.33%	0.725	0.642	0.648	0.615	0.619	0.633	0.618	0.664
(d) Env. only	-1.37%	0.655	0.692	0.657	0.658	0.622	0.564	0.617	0.643
(e) w/o Env.	7.47%	0.710	0.682	0.717	0.724	0.716	0.691	0.677	0.639
(f) w/o Con.	-0.80%	0.587	0.605	0.683	0.665	0.730	0.625	0.608	0.614

TABLE V: Ablation experiments of the different components of the proposed EvaNet when applied to 8 different image fusion metrics. “

\Delta

vs. (a) ” indicates the average metric performance change compared to the setting a.

Fig. 8: The fusion results of two representative approaches A (SDNet) and B (ReCoNet). Below the fused images, we present the ablation results, highlighting the impact of the environment perception branch within our EvaNet. “X→Y” denotes the change in the corresponding value after enabling environment-aware evaluation. The proposed environment-aware design can effectively encourage and penalise results with or without an appropriate illumination condition. The environmental condition predicted by the perception branch is shown in each source image.

Fig. 9: Metric consistency scores obtained using environment labels generated by different large language models and a human-labeled baseline. More advanced LLMs yield higher overall consistency, while all LLM-based supervisions outperform human annotation.

Dataset	LLVIP Dataset			RoadScene Dataset			TNO Dataset
Metric	Vanilla	+TextAttn	+EvaNet	Vanilla	+TextAttn	+EvaNet	Vanilla	+TextAttn	+EvaNet
$\textrm{MC}_{\textrm{deep}}$ (VIF)	0.634	0.659	0.724	0.753	0.736	0.629	0.621	0.622	0.638
$\textrm{MC}_{\textrm{deep}}$ (Qabf)	0.611	0.607	0.717	0.542	0.582	0.709	0.609	0.620	0.644
$\textrm{MC}_{\textrm{deep}}$ (SSIM)	0.649	0.560	0.726	0.584	0.603	0.655	0.603	0.601	0.645
$\textrm{MC}_{\textrm{deep}}$ (CC)	0.696	0.608	0.724	0.590	0.589	0.630	0.673	0.614	0.655
$\textrm{MC}_{\textrm{deep}}$ (PSNR)	0.609	0.559	0.737	0.601	0.612	0.633	0.667	0.663	0.645

TABLE VI: The quantitative results of

\textrm{MC}_{\textrm{deep}}

on different image fusion datasets. ( Bold : Better performance compared to the vanilla metrics)

Fig. 10: Metric-wise attribution visualisation of “Conv3” for two fusion results on the LLVIP dataset. Brighter regions indicate higher contribution to the predicted metric. Different metrics attend to distinct image regions in accordance with their underlying definitions.

Fig. 11: The qualitative results produced by the EvaNet when predicting representative fusion results on the LLVIP dataset. From top to bottom, MetaFusion [ 33 ] , ReCoNet [ 50 ] , and FusionBooster [ 11 ] .

Fig. 12: The qualitative results produced by the EvaNet when predicting the representative fusion results on the RoadScene dataset. From top to bottom, MetaFusion, TarDAL [ 7 ] , and CoCoNet [ 53 ] .

Fig. 13: The qualitative results produced by the EvaNet when predicting representative fusion results on the TNO dataset. From top to bottom, EMMA [ 52 ] , IFCNN [ 46 ] , and DDFM [ 29 ] .

Fig. 14: The quantitative results of the ablation study of the number of predicted metrics, evaluated using

\textrm{MC}_{\textrm{deep}}

. For the first five independent metrics, increasing the number of simultaneous predictions does not significantly compromise consistency. This suggests that EvaNet maintains reliable performance even with a more comprehensive output configuration.

Fig. 15: Detection results visualisation of two representative image fusion methods (LRRNet [ 19 ] and CoCoNet [ 53 ] ) and the corresponding ranking difference relating to PSNR with or without EvaNet.

Fig. 16: Segmentation results visualisation of two representative image fusion methods (EMMA [ 52 ] and SDNet [ 47 ] ) and the corresponding ranking difference in Qabf with or without EvaNet.

Fig. 17: The quantitative results reporting metric consistency using CLIP-IQA as the reference model, i.e. ,

\textrm{MC}_{\textrm{clip}}

values for different metrics.

Task	Detection			Segmentation
Method	Vanilla	+TextAtt	+EvaNet	Vanilla	+TextAtt	+EvaNet
$\textrm{MC}_{\textrm{ds}}(\textrm{VIF})$	0.590	0.575	0.752	0.591	-	0.682
$\textrm{MC}_{\textrm{ds}}(\textrm{Qabf})$	0.604	0.615	0.720	0.595	-	0.632
$\textrm{MC}_{\textrm{ds}}(\textrm{SSIM})$	0.707	0.607	0.724	0.689	-	0.737
$\textrm{MC}_{\textrm{ds}}(\textrm{FMI}_{\textrm{d}})$	0.740	0.634	0.654	0.666	-	0.563
$\textrm{MC}_{\textrm{ds}}(\textrm{FMI}_{\textrm{w}})$	0.642	0.607	0.658	0.695	-	0.554

TABLE VII: The quantitative results of MC-DS obtained for different downstream vision tasks. ( Bold : Better performance compared to the vanilla metrics;’-’ suggests that the metric does not support this dataset)

Fig. 18: The results of comparison of the proposed EvaNet and the original image quality assessments on the MSRS dataset.

Fig. 19: A comparison of the EN values of the source images and the two state-of-the-art (Text-IF [ 51 ] and GIFNet [ 8 ] ) image fusion results. Without an appropriate adaptation, the vanilla version of EN indicates that simply outputting the infrared modality is better than fusing two modalities together.

Fig. 20: Qualitative results obtained for the daytime scene and nighttime scene by fusion method A (U2Fusion [ 60 ] ) and fusion method B (FBooster [ 11 ] ) on the MSRS dataset. The ranking change of the mutual information pixel level feature distributions is presented below these fused images.

Metric	EN	SD	EI	SF
$\textrm{MC}_{\textrm{deep}}$	0.659→ 0.679	0.715→0.621	0.643→0.625	0.622→ 0.635
$\textrm{MC}_{\textrm{clip}}$	0.676→0.617	0.628→ 0.726	0.585→ 0.695	0.676→ 0.702

TABLE VIII: The MC value results produced by our EvaNet for reference-free image quality assessments.

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment

Abstract

Figures & tables

Explore similar work

Bridging Human Evaluation to Infrared and Visible Image Fusion

Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment

Ranking Infrared-Visible Fusion the Way Humans Do: A Learned Pairwise Preference Measure