AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion

Hao Zhang; Zhiqian Zheng; Linfeng Tang; Xinyu Xiang; Jiayi Ma

cs.CVOct 9, 2026

AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion

Authors: Hao Zhang, Zhiqian Zheng, Linfeng Tang, Xinyu Xiang, Jiayi Ma

Organizations: Electronic Information School, Wuhan University, Wuhan, 430072, China · Suzhou Institute, Wuhan University, Suzhou, 215123, China · School of Robotics, Wuhan University, Wuhan, 430072, China

Abstract

Existing image fusion methods struggle to balance visual quality and semantic attributes due to the inherent conflict between their optimization objectives. To address this challenge, we propose an attack-defense semantic reinforcement framework for multi-modal image fusion, termed AD-SRF. By decoupling the optimization of these two aspects and then losslessly reintegrating them, this framework injects rich semantic attributes into the fused image without compromising its visual quality. First, we develop a transformer-based restorative visual fusion network as a strong visual baseline. Equipped with the cross-modal retrieval module and the information restoration-fusion collaborative loss, it produces highly visible fused images even under degraded conditions. Second, we introduce an attack-defense game mechanism to strengthen the semantic attributes of the fused images. The attacker learns perturbation patterns that impair downstream segmentation, whereas the defender restores missing semantics from the original inputs and reintegrates them into the fused image through an imperceptible semantic map. Through iterative adversarial interaction, this mechanism drives the defender to continually strengthen the fused image's semantic attributes, boosting downstream segmentation performance without compromising visual quality. Extensive experiments demonstrate that AD-SRF achieves a win-win performance in both visual quality and semantic attributes. The code is publicly available at https://github.com/HaoZhang1018/AD-SRF.

Figures & tables

Fig. 1: Comparative analysis of existing coupled and our proposed decoupled paradigms. Our AD-SRF achieves both high-fidelity visual representations and accurate semantic decision performance under adverse conditions, whereas the SOTA competitor suffers from weakened visualization and missed detections.

Fig. 2: The overall pipeline of our proposed attack-defense semantic reinforcement framework for multi-modal image fusion (AD-SRF). It starts from a strong visual baseline and leverages an attack-defense game to learn imperceptible semantic maps, improving the visual quality of fused images and their semantic affinity for the downstream segmentation task under degraded conditions.

Fig. 3: An Optimization Perspective on ADG: Principles and Visualizations. We demonstrate the principles of ADG from an optimization perspective, complemented by confidence map visualizations that verify the stability and precision of the reinforced features.

Fig. 4: Qualitative comparison of our method and existing image fusion methods under standard setting.

Methods	MFNet						FMB
Methods	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$
SeAFusion [ 9 ]	63.8112	3.6301	6.4217	0.3979	0.4700	0.0225	59.9176	4.2748	6.7538	0.2829	0.2685	0.0107
MRFS [ 16 ]	62.7057	3.2083	7.0467	0.3595	0.4061	0.0207	60.5819	3.6545	6.7777	0.2593	0.2219	0.0122
EMMA [ 29 ]	63.3372	4.0151	6.6118	0.3881	0.4896	0.0219	60.8850	4.6881	6.7729	0.2968	0.3333	0.0110
KDFuse [ 30 ]	63.7641	3.7307	6.4990	0.3840	0.5067	0.0224	60.2450	4.2054	6.6484	0.2676	0.2662	0.0114
Text-IF [ 31 ]	63.2369	4.0099	6.7773	0.4119	0.5598	0.0205	60.8934	4.4739	6.7402	0.2784	0.3085	0.0097

TABLE I: Quantitative vision results on MFNet and FMB datasets under standard setting.

Fig. 5: Qualitative comparison of our method and competitive methods on semantic segmentation under standard setting.

Methods	Car	Pers.	Bike	Curve	C.S.	Guar.	Cone	Bump	mIoU
SeAFusion [ 9 ]	89.7	74.1	64.5	46.4	37.6	4.7	51.5	51.4	57.6
MRFS [ 16 ]	89.4	75.4	65.1	49.0	37.2	5.4	53.1	58.8	59.1
EMMA [ 29 ]	89.2	72.6	63.4	47.6	29.4	6.3	54.6	59.8	57.9
KDFuse [ 30 ]	90.1	73.9	65.8	48.3	27.2	4.7	59.0	56.2	58.2
Text-IF [ 31 ]	90.1	72.9	65.2	47.2	34.7	5.4	55.6	54.1	58.2
A 2 RNet [ 21 ]	90.1	72.9	65.2	47.2	34.7	5.4	55.6	54.1	58.2

TABLE II: Quantitative segmentation results on MFNet dataset under standard setting (mIoU is averaged over all 9 categories).

Methods	BG.	Road	Side.	Build.	Lamp	Sign	Veg.	Sky	Pers.	Car	Truck	Bus	Motor.	Bike	Pole	mIoU
SeAFusion [ 9 ]	15.75	86.71	52.08	83.53	49.29	75.23	86.30	94.61	69.38	83.56	54.59	74.40	39.79	-	50.93	61.08
MRFS [ 16 ]	17.04	86.94	46.28	85.39	50.07	75.75	86.98	95.11	71.31	76.16	34.44	28.63	49.03	-	53.64	57.12
EMMA [ 29 ]	16.77	87.67	49.80	83.30	47.99	75.61	86.62	94.04	67.94	83.35	57.49	75.08	45.09	-	50.99	61.45
KDFuse [ 30 ]	17.16	88.19	49.17	83.79	45.09	76.70	86.87	94.32	69.67	83.41	56.71	74.24	47.73	-	54.33	61.82
Text-IF [ 31 ]	16.37	87.65	49.99	85.17	46.48	75.37	85.75	94.08	68.94	83.48	57.97	74.90	48.07	-	54.25	61.90
A 2 RNet [ 21 ]	18.42	88.02	49.68	83.29	40.77	77.06	86.55	93.85	68.38	82.37	43.37	73.05	41.03	-	52.59	59.89

TABLE III: Quantitative segmentation results on FMB dataset under standard setting (mIoU is averaged over all 15 categories). Test set lacks bike class.

Methods	FMB
Methods	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$
SeAFusion † [ 9 ]	57.1016	5.3778	6.5741	0.2784	0.3469	0.0073
MRFS † [ 16 ]	60.4472	3.5689	6.8023	0.2535	0.2128	0.0122
KDFuse † [ 30 ]	59.9691	4.9620	6.6208	0.2917	0.3674	0.0090
Text-IF † [ 31 ]	60.5321	4.7523	7.1092	0.3117	0.3147	0.0080
A 2 RNet † [ 21 ]	62.0232	5.6666	6.9115	0.2325	0.3742	0.0096

TABLE IV: Quantitative results on FMB dataset under retraining enhancement setting.

Methods	BG.	Road	Side.	Build.	Lamp	Sign	Veg.	Sky	Pers.	Car	Truck	Bus	Motor.	Bike	Pole	mIoU
SeAFusion † [ 9 ]	17.50	88.83	52.80	84.07	49.22	76.46	87.07	94.36	69.12	82.20	55.23	59.43	42.49	-	55.03	60.92
MRFS † [ 16 ]	16.99	86.84	46.15	85.35	48.83	75.85	86.88	95.11	71.18	76.18	34.75	28.52	48.50	-	53.35	56.96
KDFuse † [ 30 ]	17.87	88.05	52.17	82.51	46.84	76.50	86.47	94.14	68.76	83.24	58.57	61.98	44.94	-	53.01	61.00
Text-IF † [ 31 ]	17.09	88.03	49.92	83.85	48.86	75.65	86.97	94.08	70.66	83.19	57.40	68.84	42.57	-	54.80	61.46
A 2 RNet † [ 21 ]	15.85	85.49	48.44	82.53	40.66	74.88	85.86	94.47	66.11	81.83	59.47	65.29	47.12	-	52.68	60.04
TDFusion † [ 32 ]	17.02	88.22	51.38	85.21	47.38	75.05	87.07	94.37	69.70	83.98	58.87	76.50	44.72	-	53.19	62.18

TABLE V: Quantitative segmentation results on FMB dataset under retraining enhancement setting (mIoU is averaged over all 15 categories). Test set lacks bike class.

Fig. 6: Qualitative comparison of our method and existing image fusion methods under retraining enhancement setting.

Fig. 7: Qualitative comparison of our method and existing image fusion methods under retraining enhancement setting.

Methods	MFNet		FMB
Methods	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
$I_{f}^{H}{+}\zeta_{A}$	48.72	0.9949	51.34	0.9948
$I_{f}^{H}{+}\zeta_{D}$	47.02	0.9910	50.96	0.9942
$I_{f}^{H}{+}\zeta_{D}{+}\zeta_{A}$	45.64	0.9883	49.31	0.9913

TABLE VI: Visual fidelity validation of ADG paradigm.

Fig. 8: Qualitative validation of ADG paradigm: visual fidelity and semantic interaction.

Datasets	$\Psi(I_{f}^{H})$	$\Psi(I_{f}^{H}{+}\zeta_{A})$	$\Psi(I_{f}^{H}{+}\zeta_{D})$	$\Psi(I_{f}^{H}{+}\zeta_{D}{+}\zeta_{A})$
MFNet	59.61	57.77	61.65	61.01
FMB	62.27	60.39	63.87	63.72

TABLE VII: Semantic interaction validation of ADG paradigm.

Fig. 9: Qualitative applications to the task of polarization image fusion.

Fig. 10: Qualitative applications to the task of near-infrared and visible image fusion.

Methods	Polarization
Methods	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$
SeAFusion [ 9 ]	62.0774	2.2667	6.5333	0.2711	0.2265	0.1661
MRFS [ 16 ]	61.5999	1.4195	6.7700	0.2224	0.1628	0.1412
EMMA [ 29 ]	61.7656	2.1649	6.7912	0.2624	0.2210	0.1501
KDFuse [ 30 ]	62.2455	2.1002	6.5313	0.2589	0.2197	0.1738
Text-IF [ 31 ]	62.0376	2.4223	6.7537	0.3094	0.2423	0.1662

TABLE VIII: Quantitative results on Polarization image fusion.

Methods	NIR-VIS
Methods	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$
SeAFusion [ 9 ]	65.1848	5.5793	7.2054	0.5399	0.3544	0.0460
MRFS [ 16 ]	64.2899	3.9187	7.1277	0.4324	0.2219	0.0570
EMMA [ 29 ]	65.1122	5.6443	7.2679	0.5326	0.3594	0.0479
KDFuse [ 30 ]	64.8639	4.9950	7.1430	0.4913	0.2981	0.0554
Text-IF [ 31 ]	64.7539	5.3033	7.2012	0.5240	0.3321	0.0495

TABLE IX: Quantitative results on NIR-VIS image fusion.

Fig. 11: Qualitative results of ablations on visual components.

Fig. 12: Qualitative results of ablations on semantic components.

Ablations	PSNR $\uparrow$	AG $\uparrow$	EN $\uparrow$	SSIM $\uparrow$	$\bm{Q^{AB/F}}$ $\uparrow$	$\bm{N^{AB/F}}$ $\downarrow$
Model I	62.0865	5.4873	7.0925	0.3293	0.4028	0.0093
Model II	61.1906	5.4129	6.9576	0.3257	0.3766	0.0083
Model III	59.3522	5.9259	7.0201	0.3359	0.4408	0.0087
Full Model	61.2501	5.8770	7.1544	0.3368	0.4219	0.0081

TABLE X: Quantitative results of ablations on visual components.

Ablations	Road	Side.	Build.	Sign	Veg.	Sky	Pers.	Car	mIoU
Model I	89.09	53.50	84.66	76.60	87.81	94.53	70.50	85.58	62.92
Model II	87.85	48.26	85.01	77.39	87.10	94.54	69.91	83.28	62.27
Model III	90.51	55.61	84.79	76.40	87.05	94.86	69.99	85.86	63.00
Model IV	89.57	56.80	85.11	77.59	87.44	94.79	71.51	85.82	63.33
Full Model	90.14	57.36	85.44	77.75	87.81	94.92	71.73	86.15	63.87

TABLE XI: Quantitative results of ablations on semantic components.

Methods	Road	Side.	Build.	Sign	Veg.	Sky	Pers.	Car	mIoU
KDFuse	88.19	49.17	83.79	76.70	86.87	94.32	69.67	83.41	61.82
KDFuse ∗	90.43	54.50	83.98	76.84	87.49	94.12	70.06	84.74	63.06
Text-IF	87.65	49.99	85.17	75.37	85.75	94.08	68.94	83.48	61.90
Text-IF ∗	89.21	56.22	83.70	76.64	87.17	93.47	70.76	84.89	63.10
ControlFusion	88.52	49.54	84.34	76.57	86.27	94.16	71.00	84.20	61.67
ControlFusion ∗	89.11	52.07	84.21	77.00	87.14	93.96	71.47	85.73	63.22

TABLE XII: Quantitative evaluation of ADG’s generalization performance across various fusion methods.

Fig. 13: Consistent semantic gain with ADG across multiple fusion methods.

Explore similar work

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration
Oct 9, 2026Hao Zhang, Xuhui Zuo, Jiayi MaMulti-Source Image Fusion
Electronic Information School, Wuhan University, Wuhan 430072, China
SGPDFuse: Semantically-Guided Physics-Disentanglement General Multi-Modal Image Fusion
Aug 29, 2026Haozhen Wei, Chengjun Jiang, Yutong Guo +4Multi-Source Image Fusion Infrared-Visible Image Fusion
Dalian University of Technology Dalian, Liaoning, China · City University of Hong Kong Hong Kong, China · Zhejiang University Hangzhou, Zhejiang, China
Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction
Jun 25, 2026Xilai Li, Xiaosong Li, Haishu Tan +3Multi-Source Image Fusion Cross-Modal Attention
Foshan University, Foshan, China · China University of Mining and Technology, Beijing, China · Kunming University of Science and Technology, Kunming, China

Oct 9, 2026J/K move · Enter open · S save

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

Hao Zhang, Xuhui Zuo, Jiayi Ma

Electronic Information School, Wuhan University, Wuhan 430072, China

Multi-Source Image Fusion

Open paper AskPDF arXiv

Existing multi-modal image fusion (MMIF) methods generally face an optimization gap between the visual and semantic domains, limiting the enhancement of perception capabilities. To overcome this long-standing barrier, this work proposes W2F, a new visual-semantic mutual reinforcing paradigm for MMIF. First, we reinforce the foundation for visual-semantic collaboration by amplifying feature complementarity through two targeted designs. Specifically, the segmentation-guided fusion loss introduces object-aware structural guidance into visual fusion, while the embedded information restoration prior strengthens the ability of visual fusion to provide reliable perceptual evidence for the semantic branch under degraded conditions. Then, we introduce a strong multi-modal information interaction mechanism composed of interactive gated mixed attention (IGM-Att) and progressive coupling attention (PC-Att). These modules ensure comprehensive scene representation from both local and global perspectives, providing informational support for visual-semantic collaboration. Finally, we construct a collaborative mechanism to couple visual fusion and semantic fusion. It employs a feature search module to extract complementary key features from both tasks, enriching the representational potential of visual and semantic information. Meanwhile, a gradient correction strategy ensures collaboration from the optimization perspective, enhancing performance for both visual and semantic fusion through consistency learning. Extensive experiments demonstrate that W2F surpasses state-of-the-art methods in both visual appeal and decision accuracy. The code is publicly available at https://github.com/HaoZhang1018/W2F.

Abstract

Explore similar work

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

SGPDFuse: Semantically-Guided Physics-Disentanglement General Multi-Modal Image Fusion

Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction