Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

Hao Zhang; Xuhui Zuo; Jiayi Ma

cs.CVOct 9, 2026

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

Authors: Hao Zhang, Xuhui Zuo, Jiayi Ma

Organizations: Electronic Information School, Wuhan University, Wuhan 430072, China

Abstract

Existing multi-modal image fusion (MMIF) methods generally face an optimization gap between the visual and semantic domains, limiting the enhancement of perception capabilities. To overcome this long-standing barrier, this work proposes W2F, a new visual-semantic mutual reinforcing paradigm for MMIF. First, we reinforce the foundation for visual-semantic collaboration by amplifying feature complementarity through two targeted designs. Specifically, the segmentation-guided fusion loss introduces object-aware structural guidance into visual fusion, while the embedded information restoration prior strengthens the ability of visual fusion to provide reliable perceptual evidence for the semantic branch under degraded conditions. Then, we introduce a strong multi-modal information interaction mechanism composed of interactive gated mixed attention (IGM-Att) and progressive coupling attention (PC-Att). These modules ensure comprehensive scene representation from both local and global perspectives, providing informational support for visual-semantic collaboration. Finally, we construct a collaborative mechanism to couple visual fusion and semantic fusion. It employs a feature search module to extract complementary key features from both tasks, enriching the representational potential of visual and semantic information. Meanwhile, a gradient correction strategy ensures collaboration from the optimization perspective, enhancing performance for both visual and semantic fusion through consistency learning. Extensive experiments demonstrate that W2F surpasses state-of-the-art methods in both visual appeal and decision accuracy. The code is publicly available at https://github.com/HaoZhang1018/W2F.

Figures & tables

Figure 1: Illustration of three optimization paradigms for the interaction between visual and semantic fusion. (a) Independent optimization paradigm; (b) sequential iterative optimization paradigm; (c) our proposed mutual reinforcing paradigm; (d) comparison of results from different paradigms, where IF-FILM Zhao et al. (2024b) and CMNeXt Zhang et al. (2023b) represent the independent optimization paradigm, and SegMiF Liu et al. (2023a) represents the sequential iterative optimization paradigm. The source images are a paired visible and infrared image sample from the FMB dataset Liu et al. (2023a) .

Figure 2: Overall framework of our W2F for mutually reinforcing visual fusion and segmentation via visual-semantic collaboration.

Figure 3: Illustration of amplifying the complementarity between visual fusion and semantic segmentation, which strengthens the basis for their collaboration.

Figure 4: New high-quality datasets constructed using Adobe Photoshop Lightroom, containing $1,569$ and $1,500$ spatially aligned clean visible and infrared reference image pairs for MFNet Ha et al. (2017) and FMB Liu et al. (2023a) , respectively, to provide the desired information restoration prior.

Figure 5: Architecture of our proposed IGM-Att module.

Figure 6: Architecture of our proposed PC-Att module.

Figure 7: Optimization illustration of the proposed feature search module. The two curved surfaces denote the optimization landscapes of the visual fusion and semantic segmentation losses, respectively. The red trajectories represent the independent optimization processes leading to the Independently Optimized State , where further performance improvement is constrained by the lack of cross-task complementarity. The white trajectories represent the subsequent collaborative optimization that progressively exploits complementary cross-task representations to break this performance bottleneck, leading to the Collaboratively Optimized State .

Figure 8: Architecture of our proposed feature search module.

Figure 9: Illustrating the gradient correction strategy.

Figure 10: Qualitative results of multi-modal semantic segmentation on the MFNet dataset.

MFNet	Car	Person	Bike	Curve	Car Stop	Guard.	Cone	Bump	mIoU
SeAFusion	84.2	71.1	58.7	33.1	20.1	0.0	40.4	33.9	48.8
SuperFusion	82.9	69.1	58.8	31.3	27.1	0.0	40.0	37.2	49.3
SegMiF	87.8	71.4	63.2	47.5	31.1	0.0	48.9	50.3	56.1
EAEFNet	87.6	72.6	63.8	48.6	35.0	14.2	52.4	58.3	58.9
CMX	90.1	75.2	64.5	50.2	35.3	8.5	54.2	60.6	59.7
CMNeXt	90.3	74.5	64.5	46.6	39.2	15.1	52.6	59.8	60.1

Table 1: Quantitative results of multi-modal semantic segmentation on the MFNet dataset.

Figure 11: Qualitative results of multi-modal semantic segmentation on the FMB dataset.

FMB	Car	Person	Truck	T-Lamp	T-Sign	Buil.	Vege.	Pole	mIoU
SeAFusion	76.2	59.6	15.1	34.4	68.0	80.1	83.5	38.4	48.4
SuperFusion	75.8	55.7	0.06	27.3	69.0	76.9	83.2	36.7	46.3
SegMiF	78.7	65.5	42.4	35.6	71.7	80.1	85.1	35.7	54.8
EAEFNet	79.7	61.6	22.5	34.3	74.6	82.3	86.6	46.2	54.1
CMX	79.2	69.1	39.7	49.7	76.5	84.5	87.2	54.1	57.6
CMNeXt	79.4	62.3	32.7	40.9	74.4	81.6	86.0	49.4	56.3

Table 2: Quantitative results of multi-modal semantic segmentation on the FMB dataset.

Figure 12: Qualitative results of multi-modal visual fusion on the MFNet dataset.

MFNet	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.720	0.531	0.565	0.702	0.718	0.434	0.719	0.716	0.742
SCD $\uparrow$	1.057	0.831	1.091	1.021	1.169	0.697	0.981	1.068	1.180
SSIM $\uparrow$	0.746	0.673	0.677	0.717	0.744	0.724	0.667	0.715	0.777
$N^{AB/F}$ $\downarrow$	0.077	0.035	0.138	0.166	0.023	0.041	0.056	0.045	0.023

Table 3: Quantitative results of multi-modal visual fusion on the MFNet dataset.

Figure 13: Qualitative results of multi-modal visual fusion on the FMB dataset.

FMB	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.624	0.453	0.556	0.474	0.624	0.545	0.639	0.632	0.702
SCD $\uparrow$	0.835	0.741	0.824	0.353	0.987	0.715	0.695	0.704	1.063
SSIM $\uparrow$	0.675	0.571	0.704	0.661	0.741	0.694	0.674	0.690	0.757
$N^{AB/F}$ $\downarrow$	0.094	0.153	0.109	0.101	0.020	0.110	0.096	0.053	0.052

Table 4: Quantitative results of multi-modal visual fusion on the FMB dataset.

Figure 14: Qualitative results of semantic segmentation retraining on the MFNet dataset.

Figure 15: Qualitative results of visual fusion retraining on the MFNet dataset.

MFNet	Car	Person	Bike	Curve	Car Stop	Guard.	Cone	Bump	mIoU
SeAFusion	81.9	66.4	58.1	32.4	11.4	0.0	44.0	48.4	48.9
SuperFusion	83.1	69.4	58.2	32.2	28.4	0.0	40.2	42.3	50.2
SegMiF	87.2	71.0	61.7	39.7	27.5	9.8	56.8	47.6	55.5
EAEFNet	86.8	71.2	64.8	49.6	36.5	11.4	52.6	53.7	58.3
CMX	90.0	74.8	64.8	48.3	35.3	8.9	54.4	59.5	59.4
CMNeXt	90.3	73.7	64.0	44.1	39.8	17.0	53.3	56.4	59.6

Table 5: Quantitative results of semantic segmentation retraining on the MFNet dataset.

MFNet	SeAF.	SuperF.	SegMiF	MURF	MRFS	Ours
VIF $\uparrow$	0.676	0.342	0.663	0.716	0.767	0.742
SCD $\uparrow$	1.010	0.814	1.004	1.126	1.160	1.180
SSIM $\uparrow$	0.737	0.571	0.727	0.748	0.737	0.777
$N^{AB/F}$ $\downarrow$	0.073	0.392	0.084	0.070	0.073	0.023

Table 6: Quantitative results of visual fusion retraining on the MFNet dataset.

Figure 16: Qualitative results of generalization to the M3SVD dataset on semantic segmentation task.

Figure 17: Qualitative results of generalization to the M3SVD dataset on visual fusion task.

M3SVD	SeAF.	SuperF.	SegMiF	EAEF.	CMX	CMNeXt	CAINet	MRFS	Ours
Car	21.5	22.2	39.3	16.5	70.8	75.4	53.2	69.1	73.7
Person	65.4	61.3	73.3	78.0	79.5	77.5	72.4	77.1	82.3
mIoU	43.5	41.8	56.3	47.3	75.2	76.5	62.8	73.1	77.8

Table 7: Quantitative results of generalization to the M3SVD dataset on multi-modal semantic segmentation task.

Figure 18: Qualitative results of generalization to the RoadScene dataset.

RoadScene	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.978	0.601	1.016	0.635	0.881	0.506	0.916	0.915	0.799
SCD $\uparrow$	1.352	1.472	1.330	1.318	1.597	1.245	0.801	1.238	1.648
SSIM $\uparrow$	0.599	0.579	0.599	0.600	0.643	0.584	0.613	0.639	0.656
$N^{AB/F}$ $\downarrow$	0.352	0.109	0.399	0.209	0.106	0.149	0.047	0.238	0.104

Table 8: Quantitative results of generalization to the M3SVD dataset on visual fusion task.

M3SVD	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.692	0.668	0.801	0.785	0.822	0.769	0.760	0.751	0.690
SCD $\uparrow$	1.487	1.015	1.702	1.070	1.714	1.650	1.298	1.267	1.726
SSIM $\uparrow$	0.614	0.642	0.659	0.685	0.708	0.696	0.609	0.679	0.748
$N^{AB/F}$ $\downarrow$	0.386	0.292	0.368	0.257	0.242	0.304	0.383	0.252	0.209

Table 9: Quantitative results of generalization to the RoadScene dataset.

Figure 19: Qualitative results of application to the polarization image fusion task.

Figure 20: Qualitative results of application to the near-infrared and visible image fusion task.

Polarization	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.860	0.451	0.951	0.800	0.848	0.412	0.937	0.833	0.772
SCD $\uparrow$	1.417	1.170	1.392	1.098	1.576	1.043	1.343	1.132	1.585
SSIM $\uparrow$	0.660	0.606	0.635	0.599	0.656	0.624	0.659	0.619	0.661
$N^{AB/F}$ $\downarrow$	0.149	0.103	0.207	0.252	0.109	0.254	0.180	0.126	0.095

Table 10: Quantitative results of application to the polarization visual fusion task.

Figure 21: Qualitative results of application to the medical image fusion task.

NIR-VIS	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	1.246	0.940	1.318	1.047	1.319	0.994	1.315	1.162	1.331
SCD $\uparrow$	1.181	1.000	1.522	0.780	1.310	0.562	1.303	1.184	0.596
SSIM $\uparrow$	0.801	0.738	0.779	0.781	0.830	0.788	0.820	0.775	0.878
$N^{AB/F}$ $\downarrow$	0.186	0.149	0.185	0.100	0.053	0.181	0.166	0.128	0.091

Table 11: Quantitative results of application to the near-infrared and visible image fusion task.

Medical	SeAF.	SuperF.	SegMiF	TIMF.	DDFM	MURF	IF-F.	MRFS	Ours
VIF $\uparrow$	0.619	0.448	0.635	0.568	0.664	0.700	0.611	0.598	0.788
SCD $\uparrow$	0.794	0.238	0.695	0.740	0.219	0.735	0.983	1.006	0.515
SSIM $\uparrow$	0.304	0.236	0.723	0.250	0.753	0.718	0.444	0.268	0.770
$N^{AB/F}$ $\downarrow$	0.234	0.234	0.110	0.366	0.048	0.184	0.231	0.484	0.077

Table 12: Quantitative results of application to the medical image fusion task.

Figure 22: Feature visualization of visual fusion and semantic fusion before and after interaction.

Figure 23: Qualitative impact of each ablation variant on semantic segmentation.

Model	Car	Person	Bike	Curve	Car Stop	Guard.	Cone	Bump	mIoU
I	89.7	74.3	66.0	48.3	37.1	7.8	54.5	51.4	58.6
II	90.3	74.9	65.6	46.7	44.6	3.4	54.3	55.7	59.3
III	89.4	75.1	65.8	47.6	40.6	12.7	59.1	57.1	60.6
IV	87.9	72.2	63.7	46.9	44.3	15.1	51.1	57.9	59.7
V (W2F)	89.9	74.7	66.5	47.1	53.3	15.8	54.3	53.3	61.5
VI	89.2	74.6	64.1	48.2	41.3	10.6	56.9	55.3	59.8

Table 13: Quantitative segmentation of ablation studies.

Figure 24: Qualitative impact of each ablation variant on visual fusion.

Figure 25: Quantitative impact of each ablation variant on visual fusion.

Figure 26: Qualitative results of semantic verification with the object detection task.

	Precision	Recall	[email protected]	[email protected]	mAP@[0.5:0.95]
VIS	97.0	94.1	96.3	73.4	63.8
IR	97.6	97.3	98.9	88.1	72.8
SeAFusion	97.7	99.1	99.1	91.1	73.4
SuperFusion	96.8	98.5	99.1	91.6	73.8
SegMiF	98.2	97.3	98.8	91.0	73.4
TIMFusion	97.1	97.6	98.7	90.8	72.5

Table 14: Quantitative semantic verification with object detection.

Method	EAEFNet	CMX	CMNeXt	CAINet	DDFM	MURF	IF-FILM	SeAFusion	SuperFusion	SegMiF	TIMFusion	MRFS	Ours
Segmentation	✔	✔	✔	✔	✘	✘	✘	✔	✔	✔	✘	✔	✔
Visual Fusion	✘	✘	✘	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
Parameters (M)	147.21	139.86	116.53	9.46	552.66	4.92	2.07	13.06	14.86	45.60	1.23	134.97	268.25
FLOPs (G)	316.49	211.35	207.19	35.50	8700.83	111.78	140.64	102.53	142.65	526.20	337.35	219.16	394.09
Time (s)	0.16	0.29	0.09	0.03	60.03	0.12	0.03	0.38	0.14	0.57	0.02	0.12	0.15

Table 15: Computational analysis of multi-modal visual fusion and segmentation methods. Note: TIMFusion only reports visual fusion results because its semantic segmentation code is unavailable.

Explore similar work

AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion
Oct 9, 2026Hao Zhang, Zhiqian Zheng, Linfeng Tang +2Multi-Source Image Fusion
Electronic Information School, Wuhan University, Wuhan, 430072, China · Suzhou Institute, Wuhan University, Suzhou, 215123, China · School of Robotics, Wuhan University, Wuhan, 430072, China
SGPDFuse: Semantically-Guided Physics-Disentanglement General Multi-Modal Image Fusion
Aug 29, 2026Haozhen Wei, Chengjun Jiang, Yutong Guo +4Multi-Source Image Fusion Infrared-Visible Image Fusion
Dalian University of Technology Dalian, Liaoning, China · City University of Hong Kong Hong Kong, China · Zhejiang University Hangzhou, Zhejiang, China
Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction
Jun 25, 2026Xilai Li, Xiaosong Li, Haishu Tan +3Multi-Source Image Fusion Cross-Modal Attention
Foshan University, Foshan, China · China University of Mining and Technology, Beijing, China · Kunming University of Science and Technology, Kunming, China

Oct 9, 2026J/K move · Enter open · S save

AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion

Hao Zhang, Zhiqian Zheng, Linfeng Tang, Xinyu Xiang+1

Electronic Information School, Wuhan University, Wuhan, 430072, China · Suzhou Institute, Wuhan University, Suzhou, 215123, China · School of Robotics, Wuhan University, Wuhan, 430072, China

Multi-Source Image Fusion

Open paper AskPDF arXiv

Existing image fusion methods struggle to balance visual quality and semantic attributes due to the inherent conflict between their optimization objectives. To address this challenge, we propose an attack-defense semantic reinforcement framework for multi-modal image fusion, termed AD-SRF. By decoupling the optimization of these two aspects and then losslessly reintegrating them, this framework injects rich semantic attributes into the fused image without compromising its visual quality. First, we develop a transformer-based restorative visual fusion network as a strong visual baseline. Equipped with the cross-modal retrieval module and the information restoration-fusion collaborative loss, it produces highly visible fused images even under degraded conditions. Second, we introduce an attack-defense game mechanism to strengthen the semantic attributes of the fused images. The attacker learns perturbation patterns that impair downstream segmentation, whereas the defender restores missing semantics from the original inputs and reintegrates them into the fused image through an imperceptible semantic map. Through iterative adversarial interaction, this mechanism drives the defender to continually strengthen the fused image's semantic attributes, boosting downstream segmentation performance without compromising visual quality. Extensive experiments demonstrate that AD-SRF achieves a win-win performance in both visual quality and semantic attributes. The code is publicly available at https://github.com/HaoZhang1018/AD-SRF.

Abstract

Explore similar work

AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion

SGPDFuse: Semantically-Guided Physics-Disentanglement General Multi-Modal Image Fusion

Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction