Image fusion seeks to integrate complementary information from multiple sources into a single, superior image. While traditional methods are fast, they lack adaptability and performance. Conversely, deep learning approaches achieve state-of-the-art (SOTA) results but suffer from critical inefficiencies: their reliance on slow, resource-intensive, patch-based training introduces a significant gap with full-resolution inference. We propose a novel hybrid framework that resolves this trade-off. Our method utilizes a learnable U-Net to generate a dynamic guidance map that directs a classic, fixed Laplacian pyramid fusion kernel. This decoupling of policy learning from pixel synthesis enables remarkably efficient full-resolution training, eliminating the train-inference gap. Consequently, our model achieves SOTA-comparable performance in about one minute on a RTX 4090 or two minutes on a consumer laptop GPU from scratch without any external model and demonstrates powerful zero-shot generalization across diverse tasks, from infrared-visible to medical imaging. By design, the fused output is linearly constructed solely from source information, ensuring high faithfulness for critical applications. The codes are available at https://github.com/Zirconium233/HybridFusion
Figures & tables
Figure 1: A conceptual comparison of image fusion paradigms.
Figure 2: The architecture of our hybrid fusion model. A learnable U-Net (top) takes the concatenated visible luminance (Y) and infrared (Ir) channels to produce a guidance weight map. This map then directs a fixed, non-learnable Laplacian Pyramid fusion kernel (bottom) that operates on the multi-scale decompositions of the source images. The original chrominance (CbCr) is preserved and reapplied to the fused luminance, ensuring color faithfulness in the final output.
Method
MSRS
M3FD
RoadScene
Inference Mode
EN ↑
MI ↑
VIF ↑
QAB/F↑
EN ↑
MI ↑
VIF ↑
QAB/F↑
EN ↑
MI ↑
VIF ↑
QAB/F↑
VisAsFused
6.596
6.865
1.044
0.650
6.806
7.556
1.025
0.688
6.948
7.667
1.040
0.474
-
IrAsFused
5.309
5.668
1.028
0.384
7.130
8.070
1.019
0.367
7.563
8.469
1.061
0.665
-
SuperFusion [ 6 ]
6.587
3.596
0.813
0.557
6.726
4.345
0.664
0.522
6.990
3.562
0.608
0.452
Full
CDDFuse [ 4 ]
6.701
3.657
0.819
0.548
7.070
3.994
0.802
0.613
7.120
3.001
0.610
0.450
Full
IFCNN [ 7 ]
5.975
1.706
0.579
0.479
6.935
2.630
0.685
0.590
7.222
2.842
0.591
0.536
Full
Table 1: Quantitative comparison on MSRS, M3FD, and RoadScene. Total training time for our method is shown in parentheses. Data is sourced from original papers; efficiency details refer to Table 2 .
Method
MSRS Performance
Training Data
Estimated Training Time (Epochs)
Inference Mode
MSRS Inf. Time (361 images)
Parameters (M)
VIF ↑
QAB/F↑
SuperFusion [ 6 ]
0.813
0.557
MSRS Patch (256 × 256)
∼ 10h (1200 ep)
Full
20min (bs. 4)
11.23
CDDFuse [ 4 ]
0.819
0.548
MSRS Patch (128 × 128)
∼ 1h (120 ep)
Full
∼ 10min (bs. 1)
2.40
IFCNN [ 7 ]
0.579
0.479
MSRS Patch (128 × 128)
∼ 1h (120 ep)
Full
∼ 40s (bs. 16)
0.08 (+44.5)
U2Fusion [ 9 ]
0.512
0.391
MSRS Patch (64 × 64)
∼ 1h (120 ep)
Full
∼ 1min (bs. 8)
0.63
SwinFusion [ 11 ]
0.825
0.558
MSRS Patch (128 × 128)
∼ 2h (120 ep)
Full
∼ 50min (bs. 4)
13.04
Table 2: Comprehensive efficiency and parameter analysis. Previous methods use inconsistent training settings, making direct comparison unfair, but our method still performs best overall. Performance anchors on MSRS are provided; full metrics are in Table 1 .
Metric / Method
Vi (Base)
U2F [ 9 ]
IFCNN [ 7 ]
SwinF [ 11 ]
SuperF [ 6 ]
CDD [ 4 ]
PSLPT [ 16 ]
Text-IF [ 12 ]
DTPF [ 13 ]
Ours
mAP@50
0.7955
0.9280
0.9210
0.9110
0.9210
0.9220
0.9150
0.9236
0.9368
0.9518
mAP@50-95
0.5411
0.6570
0.6510
0.6410
0.6730
0.6620
0.5660
0.7052
0.7058
0.7233
Table 3: Downstream detection performance on MSRS dataset. Unlike potentially deceptive fusion metrics that may favor source-copying, downstream tasks offer a more objective and stable performance benchmark.
Method
1 Epoch (Rapid Convergence)
100 Epochs (Full Training)
Time ↓
EN ↑
MI ↑
VIF ↑
QAB/F↑
Time ↓
EN ↑
MI ↑
VIF ↑
QAB/F↑
U2Fusion [ 9 ]
28s
5.022
1.981
0.501
0.363
47min
5.236
2.133
0.522
0.389
CDDFuse [ 4 ]
55s
6.643
2.762
0.765
0.552
92min
6.711
2.897
0.811
0.623
Text-IF [ 12 ]
57s
6.541
2.782
0.746
0.523
97min
6.696
4.867
1.019
0.692
DTPF [ 13 ]
103s
6.612
2.661
0.712
0.512
171min
6.724
4.883
1.004
0.682
Ours
39s
6.771
3.499
0.996
0.678
58min
6.766
4.890
1.079
0.721
Table 4: Fair reproduction comparison on the MSRS dataset. All methods were re-trained on the same condition (RTX 4090, 4 batch size). Under the same training settings, our model undoubtedly achieves the best performance. ‘Time’ denotes the training duration. Please refer to Table 5 . for inference details.
Method
Batch Size
Resolution
VRAM Usage(FP32 Inference)
Inference Time
Architecture
CDDFuse
16
640x480
> 48G
OOM
Restormer backbone
CDDFuse
1
640x480
∼ 10G
399.1ms
Restormer backbone
DTPF
16
640x480
∼ 40G
2483ms
XRestormer backbone
U2 Fusion
16
640x480
∼ 42G
315.6ms
Full-size convolution
Ours
16
256x256
∼ 0.8G ( ∼ 2.7G BF16 Train)
266.66ms
UNet with downsampling convolution
Ours
16
640x480
∼ 12G ( ∼ 19G BF16 Train)
429.93ms
UNet with downsampling convolution
Table 5: Resource comparison on an RTX 4090. Our method’s VRAM usage scales gracefully with resolution. A batch size of 4 can be used in most consumer GPUs.
Figure 3: Qualitative comparison on the MSRS dataset. Our method effectively highlights the pedestrian from the infrared image while preserving the textural details from the visible image. The learned weight map validates this interpretable behavior. For more comparisons, please refer to supplementary materials.
Method
PET-MRI
CT-MRI
SPECT-MRI
Training Data
SSIM ↑
EN ↑
MI ↑
VIF ↑
QAB/F↑
SSIM ↑
EN ↑
MI ↑
VIF ↑
QAB/F↑
SSIM ↑
EN ↑
MI ↑
VIF ↑
QAB/F↑
MRI as Fused
1.251
1.927
3.090
1.017
0.215
1.390
2.559
3.356
1.027
0.543
1.288
3.436
3.709
1.026
0.238
Input
Med as Fused
1.251
4.302
4.831
1.046
0.815
1.390
3.220
4.017
1.025
0.528
1.288
3.515
4.229
1.043
0.784
Input
PSLPT [ 16 ]
0.815
5.492
2.641
0.548
0.373
0.810
4.730
2.392
0.502
0.432
0.933
5.140
2.747
0.359
0.325
Corr.
EMFusion [ 3 ]
1.221
5.646
3.207
0.685
0.783
1.266
4.785
3.116
0.552
0.475
1.212
4.911
3.210
0.665
0.692
Corr.
MSRPAN [ 26 ]
1.182
5.073
4.119
0.581
0.799
1.261
4.202
4.126
0.436
0.455
1.153
4.753
4.334
0.525
0.560
Corr.
Table 6: Quantitative comparison on medical image fusion datasets.
Figure 4: Qualitative zero-shot comparison on medical tasks. Our MSRS-trained model produces faithful fusion results. Competitor methods like DTPF can exhibit textural and color artifacts when generalizing, while our in-domain trained models (e.g., Trained on SPECT) also show strong cross-domain transferability (e.g., to PET).
Training Dataset (Knowledge Source)
Evaluation on SPECT-MRI
Evaluation on PET-MRI
SSIM ↑
VIF ↑
Qabf ↑
SSIM ↑
VIF ↑
Qabf ↑
SPECT-MRI (In-domain / Zero-shot)
1.270
0.956
0.780
1.259
0.898
0.795
PET-MRI (Zero-shot / In-domain)
1.233
0.861
0.754
1.253
0.937
0.806
MSRS (Zero-shot / Zero-shot)
1.214
0.965
0.776
1.118
1.025
0.772
Table 7: Analysis of zero-shot generalization between medical tasks. Models demonstrate strong cross-domain performance, with the MSRS-trained model remaining highly competitive.
Method
MSRS
M3FD
RoadScene
VIF ↑
Qabf ↑
VIF ↑
Qabf ↑
VIF ↑
Qabf ↑
Traditional Methods
Laplacian
0.862
0.624
0.745
0.580
0.703
0.527
Heuristic weight map (W → IR)
0.887
0.622
0.748
0.573
0.714
0.519
Pure Deep Learning (with ours unet)
Ours (Directly Fuse, 200 epochs)
0.981
0.362
OOM
OOM
—
—
Table 8: Ablation study comparing paradigms. The hybrid method significantly outperforms both alternatives.
Figure 5: Hyperparameter grid search for the unsupervised loss function, showing a broad region of optimal performance.
Figure 6: Interpretability of Hybrid method. For more please refer to supplementary materials.
GPU
Environment
BS
VRAM
Throughput
Time/Epoch
Epoch
MSRS Metrics
(est.)
(img/s)
(est.)
Used
VIF ↑
Qabf ↑
RTX 4090
Ubuntu
16
∼ 19 GB
22.1
∼ 0.5 min
2
0.971
0.668
RTX 4060 Laptop
WSL2 Ubuntu
4
∼ 5 GB
7.8
∼ 2.3 min
1
1.001
0.681
Google Colab T4
Linux
4
∼ 5 GB
3.7
∼ 5.1 min
1
1.016
0.691
RTX 4060 Laptop
Windows*
4
∼ 5 GB
1.0
∼ 18.8 min
1
0.997
0.681
Table 9: Hardware benchmarks. SOTA results are rapidly attainable even on consumer-grade and free-tier hardware. *Performance on Windows was notably slower due to platform-specific overhead.
Model Size
Inference Time For bs. 16 640×480 (ms)
Epoch
MSRS VIF
MSRS Qabf
Total Training Time (s)
17.264 M
429.93
1
0.9028
0.6238
35
2
1.0061
0.6798
69
608.642 K
329.17
1
0.7022
0.4306
25
2
0.7621
0.5583
48
6
0.8094
0.5965
140
80.558 K
272.23
1
0.6926
0.3930
25
Table 10: Model scaling analysis. Larger models converge much faster with a minimal increase in inference time, making them more efficient.
Image fusion aims to integrate complementary information from multiple source images to produce a more informative and visually consistent representation, benefiting both human perception and downstream vision tasks. Despite recent progress, most existing fusion methods are designed for specific tasks (i.e., multi-modal, multi-exposure, or multi-focus fusion) and struggle to effectively preserve source information during the fusion process. This limitation primarily arises from task-specific architectures and the degradation of source information caused by deep-layer propagation. To overcome these issues, we propose UniFusion, a unified image fusion framework designed to achieve cross-task generalization. First, leveraging DINOv3 for modality-consistent feature extraction, UniFusion establishes a shared semantic space for diverse inputs. Second, to preserve the understanding of each source image, we introduce a reconstruction-alignment loss to maintain consistency between fused outputs and inputs. Finally, we employ a bilevel optimization strategy to decouple and jointly optimize reconstruction and fusion objectives, effectively balancing their coupling relationship and ensuring smooth convergence. Extensive experiments across multiple fusion tasks demonstrate UniFusion's superior visual quality, generalization ability, and adaptability to real-world scenarios. Code is available at https://github.com/dusongcheng/UniFusion.
Xingyuan Li, Songcheng Du, Yang Zou +3
Zhejiang University · Northwest Polytechnical University · Dalian University of Technology +1
Complex degradations like noise, blur, and low resolution are typical challenges in real world image fusion tasks, limiting the performance and practicality of existing methods. End to end neural network based approaches are generally simple to design and highly efficient in inference, but their black-box nature leads to limited interpretability. Diffusion based methods alleviate this to some extent by providing powerful generative priors and a more structured inference process. However, they are trained to learn a single domain target distribution, whereas fusion lacks natural fused data and relies on modeling complementary information from multiple sources, making diffusion hard to apply directly in practice. To address these challenges, this paper proposes an efficient degradation aware diffusion framework for image fusion under arbitrary degradation scenarios. Specifically, instead of explicitly predicting noise as in conventional diffusion models, our method performs implicit denoising by directly regressing the fused image, enabling flexible adaptation to diverse fusion tasks under complex degradations with limited steps. Moreover, we design a joint observation model correction mechanism that simultaneously imposes degradation and fusion constraints during sampling to ensure high reconstruction accuracy. Experiments on diverse fusion tasks and degradation configurations demonstrate the superiority of the proposed method under complex degradation scenarios.
Yu Shi, Yu Liu, Zhong-Cheng Wu +3
Department of Biomedical Engineering, Hefei University of Technology · Faculty of Information Engineering and Automation, Kunming University of Science and Technology · School of Information Science and Technology, University of Science and Technology of China
The real-world adoption of portrait relighting is hindered by dataset domain gaps, camera sensitivity, and computational costs. We address these challenges with Hybrid Domain Knowledge Fusion, a paradigm that fuses the specialized strengths of synthetic, One-Light-at-A-Time (OLAT), and real-world datasets into a compact model. Our approach features specialized prior models hardened by domain-aware adaptation, followed by augmented knowledge distillation into a lightweight student model with multi-domain expertise. Our method demonstrates a 6x to 240x inference speedup while maintaining state-of-the-art (SOTA) visual quality in the experiments. Additionally, we construct a massive, high-fidelity synthetic dataset with diverse ground-truth intrinsics to support our training pipeline.