Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion

Zeyu Wang; Mingyu Ge; Haiyu Song; Haoran Duan

cs.CVSep 30, 2026

Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion

Authors: Zeyu Wang, Mingyu Ge, Haiyu Song, Haoran Duan

Organizations: College of Computer Science and Engineering Dalian Minzu University · Department of Automation Tsinghua University

Abstract

Multi-modal image fusion (MMIF) aims to form a single image by integrating shared information, preserving complementary cues, and coordinating cross-modal conflicts across modalities. However, due to the absence of ground-truth fused images, existing MMIF supervision commonly uses spatial-domain sources or gradient variants as surrogate ground truth, making the supervision mechanism inherently misaligned with the goal of MMIF and causing pixel-level compromise or modality bias. To address this, we propose a relation-constrained supervision paradigm that moves fusion supervision from the spatial domain to a learned relation space. Rather than relying solely on direct source approximation, we further leverage frozen pretrained representation models as information providers and design a learnable feature adapter to align heterogeneous DINO and CLIP features into a unified supervision space. The adapter infers three relation parameters, namely sharedness, dominance, and coordination radius, which define three losses corresponding to the MMIF's goal. To make this space reliable, we devise a self-supervised contrastive ranking objective tailored to the adapter and couple it with the fusion network through alternating optimization. Extensive experiments show that the proposed supervision space yields significant gains regardless of which mainstream backbone the fusion network adopts, offering a supervision paradigm better aligned with the goal of MMIF. Code: github.com/GMY628/RCS-Fusion.

Figures & tables

Figure 1: Vanilla MMIF Supervision Paradigm vs Our Paradigm. Our method construct a relation-constraint supervision space tailored for MMIF, leading to better modality balance and fusion quality.

Figure 2: Overview of the Learnable Feature Adapter and Relation-Constrained Supervision.

Figure 3: Overview of the Alternating Optimization Strategy.

Figure 4: Qualitative comparison for IVF (top two rows) and MIF (bottom two rows).

IVF Task		RoadScene							M 3 FD							MSRS
Methods	Pub./Year	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$
EMMA	CVPR 24	4.779	49.127	46.777	2.809	0.436	0.432	0.355	5.871	57.569	42.377	3.771	0.589	0.512	0.460	3.788	40.539	40.590	4.164	0.541	0.760	0.496
Text-Difuse	NIPS 24	3.962	42.863	48.966	1.961	0.140	0.288	0.052	1.861	20.580	40.568	2.054	0.066	0.240	0.332	2.792	30.723	39.679	1.310	0.118	0.288	0.038
Tc-MoA	CVPR 24	4.192	44.883	42.610	1.043	0.413	0.556	0.403	5.180	54.156	39.030	3.333	0.651	0.891	0.513	3.618	38.414	42.335	3.493	0.630	1.090	0.483
ReFusion	IJCV 24	5.357	59.336	48.229	2.726	0.437	0.566	0.397	5.338	57.331	40.803	3.548	0.637	0.731	0.489	3.813	40.641	42.890	4.102	0.442	1.085	0.576
C2RF	IJCV 25	4.989	52.270	45.576	2.720	0.519	0.565	0.388	4.779	50.329	39.348	2.498	0.223	0.287	0.399	3.474	37.153	41.348	1.729	0.185	0.330	0.092

Table 1: Quantitative comparison for IVF and MIF task. The best, second best, and third best are highlighted in red , blue , and green , respectively. “Ours(CNN/Mamba/Transformer)” denotes CNN, Mamba, and Transformer fusion backbones trained under our relation-constrained supervision.

Group	ID	Description	IVF on RoadScene							MIF on MRI-PET
Group	ID	Description	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$
(a)	(1)	w/o LFA	5.247	58.932	47.286	2.861	0.506	0.589	0.372	8.721	94.863	72.418	2.321	0.552	0.196	0.414
	(2)	w/o FAM	5.516	60.217	49.103	3.041	0.537	0.618	0.401	9.184	97.326	75.861	2.512	0.594	0.218	0.452
	(3)	w/o RRM	5.438	59.846	48.624	2.982	0.529	0.611	0.394	9.062	96.814	75.104	2.463	0.586	0.213	0.444
	(4)	w/o RCS	5.196	58.714	47.635	2.803	0.497	0.581	0.363	8.604	94.217	71.936	2.276	0.541	0.189	0.405
(b)	(1)	w/o $\mathcal{L}_{\mathrm{shr}}$	5.642	60.384	49.427	3.086	0.545	0.626	0.410	9.346	97.942	76.412	2.574	0.607	0.224	0.461

Table 3: Ablation study of the proposed method on the IVF RoadScene and MIF MRI-PET datasets. The best results are highlighted in bold . More ablation variants are provided in the appendix.

Figure 6: Qualitative results on downstream tasks (Object detection and medical image segmentation).

Table 9

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{\mathrm{ssr}}=\frac{1}{2}\Big(\max(0,\delta_{s}-s^{+}+s_{1}^{-})+\max(0,\delta_{s}-s^{+}+s_{2}^{-})\Big).

(22)

Appendix

Algorithm 2 AdapterObjective

(A,B,A^{-},B^{-},F;\Phi,G_{\mathrm{shr}})

Figure 7: Full qualitative comparison for IVF task on M 3 FD, RoadScene and MSRS.

Figure 8: Full qualitative comparison for MIF task on CT-MRI, MRI-PET and MRI-SPECT.

Method	$AG\!\uparrow$	$EN\!\uparrow$	$SD\!\uparrow$	$MI\!\uparrow$	$Q_{AB/F}\!\uparrow$	$Q_{M}\!\uparrow$	$Q_{P}\!\uparrow$
ReFusion Bai et al. (2025b)	5.357	59.336	48.229	2.726	0.437	0.566	0.397
ReFusion*	5.692	60.108	49.356	2.916	0.523	0.612	0.418
Gain	6.3% $\uparrow$	1.3% $\uparrow$	2.3% $\uparrow$	7.0% $\uparrow$	19.7% $\uparrow$	8.1% $\uparrow$	5.3% $\uparrow$
EMMA Zhao et al. (2024a)	4.779	49.127	46.777	2.809	0.436	0.432	0.355
EMMA*	5.514	58.742	49.108	2.943	0.518	0.589	0.407
Gain	15.4% $\uparrow$	19.6% $\uparrow$	5.0% $\uparrow$	4.8% $\uparrow$	18.8% $\uparrow$	36.3% $\uparrow$	14.6% $\uparrow$

Appendix

Table 6: Refinement effects of our paradigm on the RoadScene dataset. Original vs. upgraded models (* denotes our paradigm inserted).

Group	ID	Description	IVF on RoadScene							MIF on MRI-PET
Group	ID	Description	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$	$AG\uparrow$	$EN\uparrow$	$SD\uparrow$	$MI\uparrow$	$Q_{AB/F}\uparrow$	$Q_{M}\uparrow$	$Q_{P}\uparrow$
(a)	(1)	w/o LFA	5.247	58.932	47.286	2.861	0.506	0.589	0.372	8.721	94.863	72.418	2.321	0.552	0.196	0.414
	(2)	w/o FAM	5.516	60.217	49.103	3.041	0.537	0.618	0.401	9.184	97.326	75.861	2.512	0.594	0.218	0.452
	(3)	w/o RRM	5.438	59.846	48.624	2.982	0.529	0.611	0.394	9.062	96.814	75.104	2.463	0.586	0.213	0.444
	(4)	w/o RCS	5.196	58.714	47.635	2.803	0.497	0.581	0.363	8.604	94.217	71.936	2.276	0.541	0.189	0.405
(b)	(1)	w/o $\mathcal{L}_{\mathrm{shr}}$	5.642	60.384	49.427	3.086	0.545	0.626	0.410	9.346	97.942	76.412	2.574	0.607	0.224	0.461

Appendix

Table 7: Extended ablation study of the proposed method on the IVF RoadScene and MIF MRI-PET datasets. This table includes additional variants for fixed relation parameters and adapter training strategies. The best results are highlighted in bold .

IVF				MIF
Method	Time (s)	GFLOPs (G)	Params (M)	Method	Time (s)	GFLOPs (G)	Params (M)
EMMA	0.058	8.861	1.516	EMMA	0.009	8.861	1.516
Text-Difuse	23.818	2742.5	119.460	Text-Difuse	24.424	2742.5	119.460
Tc-MoA	0.543	61.000	340.580	Tc-MoA	0.161	61.000	340.580
ReFusion	0.026	98.344	1.486	ReFusion	0.012	98.344	1.486
C2RF	0.089	123.869	1.325	C2RF	0.112	123.869	1.325

Appendix

Table 8: Comparison of runtime, computational cost (GFLOPs), and model size (Params) on IVF and MIF tasks. All values are averaged per image.

Figure 9: Schematic diagram of fusion network.

Explore similar work

When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion
Sep 30, 2026Zeyu Wang, Jiayu Wang, Haiyu Song +1Multimodal Fusion Visible Image Fusion
College of Computer Science and Engineering, Dalian Minzu University, Dalian, China · School of Artificial Intelligence and Robotics, Hunan University, Hunan, China · Department of Automation, Tsinghua University, Beijing, China
Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction
Jun 25, 2026Xilai Li, Xiaosong Li, Haishu Tan +3Visible Image Fusion Query-Specific Degradation-Aware Representation
Foshan University, Foshan, China · China University of Mining and Technology, Beijing, China · Kunming University of Science and Technology, Kunming, China
ConFusion: Continuous Fusion Space Learning for Fine-Grained Controllable Infrared and Visible Image Fusion
Jul 26, 2026Guo Yurong, He Yufei, Li Yonghao +3Visible Image Fusion Infrared-Visible Image Fusion
North China Electric Power University Baoding, Hebei, China · Beijing University of Posts and Telecommunications Beijing, China

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion

Zeyu Wang, Jiayu Wang, Haiyu Song, Haoran Duan

College of Computer Science and Engineering, Dalian Minzu University, Dalian, China · School of Artificial Intelligence and Robotics, Hunan University, Hunan, China · Department of Automation, Tsinghua University, Beijing, China

Multimodal Fusion Visible Image Fusion Modalities Integral Structural Constraints

Open paper AskPDF arXiv

Multimodal image fusion (MMIF) aims to integrate complementary information from different modalities into a high-quality fused image and support downstream tasks. Recently, feature decomposition has become an important paradigm by separating source images into common and modality-specific unique features. However, existing methods lack clear supervision because ground-truth (GT) decomposition feature maps are unavailable. They usually combine multiple image-level metrics as losses, which are inherently incomplete and may conflict since each pixel couples attributes such as texture, edge, and contour. To address this, we propose a 1D signal-level self-supervised feature decomposition paradigm. Our core insight is to reformulate feature decomposition from unclear 2D image-level supervision into an integral-driven 1D signal-level optimization problem. This objective-level reformulation uses the 1D signal form to compute the integral constraint. The decomposer is optimized by the integral area between common and original signals, enabling more stable optimization with a clear optimization objective. Our model follows a two-stage SSL framework. Stage I designs dual pretext tasks for integral-driven decomposition at the signal level and structure-preserving reconstruction at the image level. Stage II fuses unique features and combines them with common features to reconstruct the fused image. Experiments on representative MMIF tasks show state-of-the-art (SOTA) performance. Code: github.com/Wangjiayu0512/SIDFusion.

Abstract

Explore similar work

When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion

Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction

ConFusion: Continuous Fusion Space Learning for Fine-Grained Controllable Infrared and Visible Image Fusion