Hybrid Fusion: One-Minute Efficient Training for Zero-Shot Cross-Domain Image Fusion

Ran Zhang; Xuanhua He; Liu Liu

cs.CVFeb 24, 2026

Hybrid Fusion: One-Minute Efficient Training for Zero-Shot Cross-Domain Image Fusion

Authors: Ran Zhang, Xuanhua He, Liu Liu

Organizations: Hefei University of Technology, Hefei, China · University of Science and Technology of China, Hefei, China

Abstract

Image fusion seeks to integrate complementary information from multiple sources into a single, superior image. While traditional methods are fast, they lack adaptability and performance. Conversely, deep learning approaches achieve state-of-the-art (SOTA) results but suffer from critical inefficiencies: their reliance on slow, resource-intensive, patch-based training introduces a significant gap with full-resolution inference. We propose a novel hybrid framework that resolves this trade-off. Our method utilizes a learnable U-Net to generate a dynamic guidance map that directs a classic, fixed Laplacian pyramid fusion kernel. This decoupling of policy learning from pixel synthesis enables remarkably efficient full-resolution training, eliminating the train-inference gap. Consequently, our model achieves SOTA-comparable performance in about one minute on a RTX 4090 or two minutes on a consumer laptop GPU from scratch without any external model and demonstrates powerful zero-shot generalization across diverse tasks, from infrared-visible to medical imaging. By design, the fused output is linearly constructed solely from source information, ensuring high faithfulness for critical applications. The codes are available at https://github.com/Zirconium233/HybridFusion

Figures & tables

Figure 1: A conceptual comparison of image fusion paradigms.

Figure 2: The architecture of our hybrid fusion model. A learnable U-Net (top) takes the concatenated visible luminance (Y) and infrared (Ir) channels to produce a guidance weight map. This map then directs a fixed, non-learnable Laplacian Pyramid fusion kernel (bottom) that operates on the multi-scale decompositions of the source images. The original chrominance (CbCr) is preserved and reapplied to the fused luminance, ensuring color faithfulness in the final output.

Method	MSRS				M3FD				RoadScene				Inference Mode
Method	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	Inference Mode
VisAsFused	6.596	6.865	1.044	0.650	6.806	7.556	1.025	0.688	6.948	7.667	1.040	0.474	-
IrAsFused	5.309	5.668	1.028	0.384	7.130	8.070	1.019	0.367	7.563	8.469	1.061	0.665	-
SuperFusion [ 6 ]	6.587	3.596	0.813	0.557	6.726	4.345	0.664	0.522	6.990	3.562	0.608	0.452	Full
CDDFuse [ 4 ]	6.701	3.657	0.819	0.548	7.070	3.994	0.802	0.613	7.120	3.001	0.610	0.450	Full
IFCNN [ 7 ]	5.975	1.706	0.579	0.479	6.935	2.630	0.685	0.590	7.222	2.842	0.591	0.536	Full

Table 1: Quantitative comparison on MSRS, M3FD, and RoadScene. Total training time for our method is shown in parentheses. Data is sourced from original papers; efficiency details refer to Table 2 .

Method	MSRS Performance		Training Data	Estimated Training Time (Epochs)	Inference Mode	MSRS Inf. Time (361 images)	Parameters (M)
Method	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	Training Data	Estimated Training Time (Epochs)	Inference Mode	MSRS Inf. Time (361 images)	Parameters (M)
SuperFusion [ 6 ]	0.813	0.557	MSRS Patch (256 $\times$ 256)	$\sim$ 10h (1200 ep)	Full	20min (bs. 4)	11.23
CDDFuse [ 4 ]	0.819	0.548	MSRS Patch (128 $\times$ 128)	$\sim$ 1h (120 ep)	Full	$\sim$ 10min (bs. 1)	2.40
IFCNN [ 7 ]	0.579	0.479	MSRS Patch (128 $\times$ 128)	$\sim$ 1h (120 ep)	Full	$\sim$ 40s (bs. 16)	0.08 (+44.5)
U2Fusion [ 9 ]	0.512	0.391	MSRS Patch (64 $\times$ 64)	$\sim$ 1h (120 ep)	Full	$\sim$ 1min (bs. 8)	0.63
SwinFusion [ 11 ]	0.825	0.558	MSRS Patch (128 $\times$ 128)	$\sim$ 2h (120 ep)	Full	$\sim$ 50min (bs. 4)	13.04

Table 2: Comprehensive efficiency and parameter analysis. Previous methods use inconsistent training settings, making direct comparison unfair, but our method still performs best overall. Performance anchors on MSRS are provided; full metrics are in Table 1 .

Metric / Method	Vi (Base)	U2F [ 9 ]	IFCNN [ 7 ]	SwinF [ 11 ]	SuperF [ 6 ]	CDD [ 4 ]	PSLPT [ 16 ]	Text-IF [ 12 ]	DTPF [ 13 ]	Ours
mAP@50	0.7955	0.9280	0.9210	0.9110	0.9210	0.9220	0.9150	0.9236	0.9368	0.9518
mAP@50-95	0.5411	0.6570	0.6510	0.6410	0.6730	0.6620	0.5660	0.7052	0.7058	0.7233

Table 3: Downstream detection performance on MSRS dataset. Unlike potentially deceptive fusion metrics that may favor source-copying, downstream tasks offer a more objective and stable performance benchmark.

Method	1 Epoch (Rapid Convergence)					100 Epochs (Full Training)
Method	Time $\downarrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	Time $\downarrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$
U2Fusion [ 9 ]	28s	5.022	1.981	0.501	0.363	47min	5.236	2.133	0.522	0.389
CDDFuse [ 4 ]	55s	6.643	2.762	0.765	0.552	92min	6.711	2.897	0.811	0.623
Text-IF [ 12 ]	57s	6.541	2.782	0.746	0.523	97min	6.696	4.867	1.019	0.692
DTPF [ 13 ]	103s	6.612	2.661	0.712	0.512	171min	6.724	4.883	1.004	0.682
Ours	39s	6.771	3.499	0.996	0.678	58min	6.766	4.890	1.079	0.721

Table 4: Fair reproduction comparison on the MSRS dataset. All methods were re-trained on the same condition (RTX 4090, 4 batch size). Under the same training settings, our model undoubtedly achieves the best performance. ‘Time’ denotes the training duration. Please refer to Table 5 . for inference details.

Method	Batch Size	Resolution	VRAM Usage(FP32 Inference)	Inference Time	Architecture
CDDFuse	16	640x480	$>$ 48G	OOM	Restormer backbone
CDDFuse	1	640x480	$\sim$ 10G	399.1ms	Restormer backbone
DTPF	16	640x480	$\sim$ 40G	2483ms	XRestormer backbone
U2 Fusion	16	640x480	$\sim$ 42G	315.6ms	Full-size convolution
Ours	16	256x256	$\sim$ 0.8G ( $\sim$ 2.7G BF16 Train)	266.66ms	UNet with downsampling convolution
Ours	16	640x480	$\sim$ 12G ( $\sim$ 19G BF16 Train)	429.93ms	UNet with downsampling convolution

Table 5: Resource comparison on an RTX 4090. Our method’s VRAM usage scales gracefully with resolution. A batch size of 4 can be used in most consumer GPUs.

Figure 3: Qualitative comparison on the MSRS dataset. Our method effectively highlights the pedestrian from the infrared image while preserving the textural details from the visible image. The learned weight map validates this interpretable behavior. For more comparisons, please refer to supplementary materials.

Method	PET-MRI					CT-MRI					SPECT-MRI					Training Data
Method	SSIM $\uparrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	SSIM $\uparrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	SSIM $\uparrow$	EN $\uparrow$	MI $\uparrow$	VIF $\uparrow$	$Q^{AB/F}$ $\uparrow$	Training Data
MRI as Fused	1.251	1.927	3.090	1.017	0.215	1.390	2.559	3.356	1.027	0.543	1.288	3.436	3.709	1.026	0.238	Input
Med as Fused	1.251	4.302	4.831	1.046	0.815	1.390	3.220	4.017	1.025	0.528	1.288	3.515	4.229	1.043	0.784	Input
PSLPT [ 16 ]	0.815	5.492	2.641	0.548	0.373	0.810	4.730	2.392	0.502	0.432	0.933	5.140	2.747	0.359	0.325	Corr.
EMFusion [ 3 ]	1.221	5.646	3.207	0.685	0.783	1.266	4.785	3.116	0.552	0.475	1.212	4.911	3.210	0.665	0.692	Corr.
MSRPAN [ 26 ]	1.182	5.073	4.119	0.581	0.799	1.261	4.202	4.126	0.436	0.455	1.153	4.753	4.334	0.525	0.560	Corr.

Table 6: Quantitative comparison on medical image fusion datasets.

Figure 4: Qualitative zero-shot comparison on medical tasks. Our MSRS-trained model produces faithful fusion results. Competitor methods like DTPF can exhibit textural and color artifacts when generalizing, while our in-domain trained models (e.g., Trained on SPECT) also show strong cross-domain transferability (e.g., to PET).

Training Dataset (Knowledge Source)	Evaluation on SPECT-MRI			Evaluation on PET-MRI
Training Dataset (Knowledge Source)	SSIM $\uparrow$	VIF $\uparrow$	Qabf $\uparrow$	SSIM $\uparrow$	VIF $\uparrow$	Qabf $\uparrow$
SPECT-MRI (In-domain / Zero-shot)	1.270	0.956	0.780	1.259	0.898	0.795
PET-MRI (Zero-shot / In-domain)	1.233	0.861	0.754	1.253	0.937	0.806
MSRS (Zero-shot / Zero-shot)	1.214	0.965	0.776	1.118	1.025	0.772

Table 7: Analysis of zero-shot generalization between medical tasks. Models demonstrate strong cross-domain performance, with the MSRS-trained model remaining highly competitive.

Traditional Methods
Method	MSRS		M3FD		RoadScene
Method	VIF $\uparrow$	Qabf $\uparrow$	VIF $\uparrow$	Qabf $\uparrow$	VIF $\uparrow$	Qabf $\uparrow$
Laplacian	0.862	0.624	0.745	0.580	0.703	0.527
Heuristic weight map (W $\rightarrow$ IR)	0.887	0.622	0.748	0.573	0.714	0.519
Pure Deep Learning (with ours unet)
Ours (Directly Fuse, 200 epochs)	0.981	0.362	OOM	OOM	—	—

Table 8: Ablation study comparing paradigms. The hybrid method significantly outperforms both alternatives.

Figure 5: Hyperparameter grid search for the unsupervised loss function, showing a broad region of optimal performance.

Figure 6: Interpretability of Hybrid method. For more please refer to supplementary materials.

GPU	Environment	BS	VRAM	Throughput	Time/Epoch	Epoch	MSRS Metrics
GPU	Environment		(est.)	(img/s)	(est.)	Used	VIF $\uparrow$	Qabf $\uparrow$
RTX 4090	Ubuntu	16	$\sim$ 19 GB	22.1	$\sim$ 0.5 min	2	0.971	0.668
RTX 4060 Laptop	WSL2 Ubuntu	4	$\sim$ 5 GB	7.8	$\sim$ 2.3 min	1	1.001	0.681
Google Colab T4	Linux	4	$\sim$ 5 GB	3.7	$\sim$ 5.1 min	1	1.016	0.691
RTX 4060 Laptop	Windows*	4	$\sim$ 5 GB	1.0	$\sim$ 18.8 min	1	0.997	0.681

Table 9: Hardware benchmarks. SOTA results are rapidly attainable even on consumer-grade and free-tier hardware. *Performance on Windows was notably slower due to platform-specific overhead.

Model Size	Inference Time For bs. 16 $640\times 480$ (ms)	Epoch	MSRS VIF	MSRS Qabf	Total Training Time (s)
17.264 M	429.93	1	0.9028	0.6238	35
17.264 M	429.93	2	1.0061	0.6798	69
608.642 K	329.17	1	0.7022	0.4306	25
		2	0.7621	0.5583	48
		6	0.8094	0.5965	140
80.558 K	272.23	1	0.6926	0.3930	25

Table 10: Model scaling analysis. Larger models converge much faster with a minimal increase in inference time, making them more efficient.

Explore similar work

UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation
Mar 15, 2026Xingyuan Li, Songcheng Du, Yang Zou +3Multi-Source Image Fusion
Zhejiang University · Northwest Polytechnical University · Dalian University of Technology +1
Degradation-Robust Fusion: An Efficient Degradation-Aware Diffusion Framework for Multimodal Image Fusion in Arbitrary Degradation Scenarios
Apr 10, 2026Yu Shi, Yu Liu, Zhong-Cheng Wu +3Multi-Source Image Fusion Diffusion Models
Department of Biomedical Engineering, Hefei University of Technology · Faculty of Information Engineering and Automation, Kunming University of Science and Technology · School of Information Science and Technology, University of Science and Technology of China
Toward Real-World Adoption of Portrait Relighting via Hybrid Domain Knowledge Fusion
Apr 25, 2026Qian Huang, Mayoore Selvarasa Jaiswal, Zhen Zhong +2Synthetic-to-Real Domain Adaptation Image Relighting
NVIDIA

Mar 15, 2026J/K move · Enter open · S save

UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation

Xingyuan Li, Songcheng Du, Yang Zou, HaoYuan Xu+2

Zhejiang University · Northwest Polytechnical University · Dalian University of Technology+1

Multi-Source Image Fusion

Open paper AskPDF arXiv

Image fusion aims to integrate complementary information from multiple source images to produce a more informative and visually consistent representation, benefiting both human perception and downstream vision tasks. Despite recent progress, most existing fusion methods are designed for specific tasks (i.e., multi-modal, multi-exposure, or multi-focus fusion) and struggle to effectively preserve source information during the fusion process. This limitation primarily arises from task-specific architectures and the degradation of source information caused by deep-layer propagation. To overcome these issues, we propose UniFusion, a unified image fusion framework designed to achieve cross-task generalization. First, leveraging DINOv3 for modality-consistent feature extraction, UniFusion establishes a shared semantic space for diverse inputs. Second, to preserve the understanding of each source image, we introduce a reconstruction-alignment loss to maintain consistency between fused outputs and inputs. Finally, we employ a bilevel optimization strategy to decouple and jointly optimize reconstruction and fusion objectives, effectively balancing their coupling relationship and ensuring smooth convergence. Extensive experiments across multiple fusion tasks demonstrate UniFusion's superior visual quality, generalization ability, and adaptability to real-world scenarios. Code is available at https://github.com/dusongcheng/UniFusion.

Abstract

Explore similar work

UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation

Degradation-Robust Fusion: An Efficient Degradation-Aware Diffusion Framework for Multimodal Image Fusion in Arbitrary Degradation Scenarios

Toward Real-World Adoption of Portrait Relighting via Hybrid Domain Knowledge Fusion