NatureFusion: Rethinking Infrared and Visible Image Fusion as Natural Complementary Enhancement
Authors: Jianfeng Ding, Linfeng Tang, Hao Zhang, Han Xu, Xunpeng Yi, Yibing Zhang, Meiqi Gong, Zhongyuan Wang, Jiayi Ma
Organizations: the National Engineering Research Center for Multimedia Software (NERCMS), School of Computer Science, Wuhan University, Wuhan 430072, China · the School of Robotics, Wuhan University, Wuhan 430072, China · the Electronic Information School, Wuhan University, Wuhan 430072, China · the School of Automation, Southeast University, Nanjing 210096, China
In the absence of ground truth, most existing infrared and visible image fusion methods pursue fidelity to the source images and directly superimpose two physically distinct modalities. As a result, salient infrared targets appear as bright patches of thermal radiation pasted onto the scene. These patches obscure the original color and texture of the targets, so the fused images look unnatural to human observers and deviate from the data distribution learned by vision models pretrained on visible images. Therefore, we redefine infrared and visible image fusion as natural complementary enhancement rather than source fidelity, and propose NatureFusion, which exploits cross-modal complementary cues to infer a clear and natural visible-style image from low-quality visible and infrared image pairs. Specifically, to address the lack of ground truth, we employ a generative foundation model as the teacher to directly generate visually natural fused images for supervision. Furthermore, to fully distill the teacher's generative capability while reducing computational cost, we fine-tune a pretrained generative model with conditional flow matching through a light-weight fusion adapter. Last but not least, to further improve downstream performance without sacrificing natural appearance, we freeze the fusion adapter and introduce a detector-supervised task adapter that strengthens the complementary contribution of infrared. Extensive comparisons with seven multi-modal fusion and seven single-modality enhancement methods demonstrate that NatureFusion generates visually natural fused images with the best perceptual quality. Detection and segmentation models pretrained on visible images achieve substantially higher accuracy on our fused images than on those of any competing method.
Figures & tables
Fig. 1: Comparison with multi-modal fusion and single-modality enhancement methods. Panels (a) and (b) show the visible and infrared inputs, (c) to (e) show the results of degradation-aware fusion, task-driven fusion, and single-modality enhancement methods, and (f) shows the result of NatureFusion. Red and blue boxes mark enlarged person and background regions, and gold boxes show person detections of a frozen RetinaNet with confidence scores of at least 0.25. (g) to (i) Comparison with seven multi-modal fusion methods in MUSIQ on M3FD, person AP50:95 on SMOD, and mIoU2 over Car and Person on MFNet. Detection and segmentation each use seven frozen models pretrained on visible images. Bars show means, and whiskers show ranges across images and models.
Fig. 2: Training and inference pipeline of NatureFusion. Flames and snowflakes denote trainable and frozen parameters, respectively.
Fig. 3: Qualitative comparison with multi-modal fusion methods.
Fig. 4: Qualitative comparison with single-modality enhancement methods.
Fig. 5: Person detections from the same frozen RetinaNet on an LLVIP scene. Blue boxes and numbers show predictions with confidence of at least 0.25. (a) Visible (upper right) and infrared (lower left) images are detected separately and then combined. Red circles enlarge the person region.
Fig. 6: Mean person AP versus IoU threshold. Fusion and Enhancement bands span the minimum to maximum AP of the multi-modal fusion and single-modality enhancement methods, respectively, and Margin shades where NatureFusion exceeds the best of them. Lower panels plot Δ AP, NatureFusion minus the best compared method.
Method
SMOD
LLVIP
M3FD
AP50:95↑
Rmiss↑
AP50:95↑
Rmiss↑
AP50:95↑
Rmiss↑
Retinexformer [ 12 ]
34.03
2.4
31.73
1.9
30.51
0.0
DiffLL [ 15 ]
29.82
0.5
29.72
2.6
33.64
0.3
QuadPrior [ 17 ]
34.87
1.3
35.31
2.7
35.15
0.0
DarkIR [ 14 ]
35.75
3.4
33.92
3.0
33.64
0.1
HVI-CIDNet [ 13 ]
34.45
2.4
30.70
1.9
33.93
0.2
TABLE IV: Detection comparison with single-modality enhancement methods.
Fig. 7: Frozen segmentation on MFNet by OCRNet-HR48. The left panel shows the visible and infrared inputs, with the red box marking the enlarged region, and each method’s crop is shown above its prediction.
Method
mIoU9↑
mIoU2↑
PSPNet
DLV3+
SegFormer
Mean
PSPNet
DLV3+
SegFormer
Mean
Visible
46.07
45.82
48.48
46.79
62.12
62.75
65.34
63.41
Infrared
42.18
39.74
43.98
41.97
70.00
70.83
71.48
70.77
Text-IF [ 2 ]
49.78
45.88
49.96
48.54
72.42
73.47
74.02
73.30
OmniFuse [ 4 ]
45.75
46.77
48.88
47.13
65.91
64.98
70.79
67.23
HCLFuse [ 29 ]
47.49
42.72
47.93
46.04
66.73
67.22
71.47
68.47
TABLE VII: Retrained segmentation on MFNet.
Stages
Setting
AP50:95↑
AP50↑
AP75↑
AP90↑
Rmiss↑
Test
NatureFusion, w/o IR
32.33
63.69
32.20
2.23
23.2
I
Teacher w/o IR
25.99
58.46
22.21
1.62
21.6
Teacher w/ IR
25.69
58.19
20.64
1.20
29.8
II
Fusion-only, VI+VI
32.12
63.62
31.89
2.42
24.1
Fusion-only, VI+IR
34.87
67.57
34.85
3.03
29.0
II+III
Fusion + Task, VI+VI
32.90
65.04
32.18
1.81
25.1
TABLE VIII: Role of infrared at each stage on LLVIP. w/o IR replaces infrared with an all-zero image at inference only, and VI+VI replaces infrared with the visible image in both training and inference. The best value in each column is marked in bold red.
Fig. 8: Rmiss under different infrared settings.
Variant
LLVIP
AP50:95↑
AP50↑
AP75↑
Rmiss↑
NIQE ↓
Zero-shot (VI+IR)
34.62
67.22
33.17
28.0
3.216
Fusion-only
34.44
67.30
33.87
29.2
2.774
w=1
34.85
67.36
34.65
29.2
2.763
w/o Ldet
34.82
67.75
34.16
31.4
2.747
w/o Lanc
35.02
68.00
34.42
31.3
2.747
TABLE IX: Component ablation on LLVIP. Zero-shot is the Distilled backbone without adapters, and other variants report the mean of two training runs.
w
AP55↑
AP70↑
AP90↑
AP95↑
Rmiss↑
1
77.89
65.59
11.51
1.70
43.1
1.5
78.00
66.21
11.90
1.63
43.4
2
78.69
68.23
12.01
1.76
45.0
3
78.50
68.08
11.81
1.63
44.9
TABLE X: Sensitivity to the amplification factor w on M3FD.
Fig. 10: Comparison under registration errors on LLVIP with (a) aligned inputs and (b) infrared shifted right by 64 pixels.
Fig. 12: Teacher and student outputs on an LLVIP scene.
Fig. 14: Mean generation time per image at native resolution on a single RTX 3090 (log scale).
Infrared and visible image fusion integrates complementary information from both modalities to provide a more comprehensive representation of the scene. We propose ProtoFusion, a protocol-guided generative framework for infrared and visible image fusion. Instead of relying on fixed fusion operators, hand-crafted objectives, or manually specified one-shot prompts, ProtoFusion formulates fusion as protocol-constrained generation. It represents fusion intent with a structured protocol, consisting of environment state cards, grounding principles, and a style profile, which explicitly specifies which evidence should be preserved or suppressed and which properties the fused result should satisfy. To optimize this protocol, we introduce protocol-guided prompt refinement, a closed-loop learning framework that iteratively performs generation, evaluation, and revision under multimodal evidence. Once learned, the optimized protocol is fixed at inference time and directly guides fusion generation. To address residual source inconsistencies, we further incorporate structured belief construction, which organizes modality reliability, scene evidence, and cross-modal conflicts into a structured representation for conservative correction after generation. Extensive experiments on standard benchmarks show that ProtoFusion improves quantitative performance, achieves favorable results in aesthetic evaluation, and produces visually coherent fusion outputs. Project Page: {https://erfect2020.github.io/ProtoFusion/}.
Pengwei Liang, Junjun Jiang, Junhui Hou
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China
Infrared-visible image fusion (IVIF) is pivotal for multimodal perception, yet reconciling the inherent information disparity between thermal and textural features remains a fundamental challenge. Existing prior-guided methods often rely on static constraints that induce optimization conflicts or utilize extrinsic semantic priors from large-scale foundation models (e.g., CLIP/DINO), which frequently fail to exploit the intrinsic modality characteristics essential for high-fidelity fusion. To address these issues, we propose P2Fusion, a prior-guided distillation-based framework that reformulates IVIF via dual intrinsic prompts. Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators. Specifically, a Teach-to-Fuse mechanism provides dual-granularity progressive guidance, coupled with a Gated Dynamic Expert Recalibration (GDER) module for decoupled feature refinement. This design enables the network to adaptively mediate modal competition through expert specialization. Extensive experiments demonstrate that P2Fusion achieves state-of-the-art performance across five mainstream datasets. Notably, our framework demonstrates consistent performance advantages in fusion quality, achieving state-of-the-art results in 14 out of 20 key evaluation metrics across 5 benchmarks. Furthermore, it effectively contributes to the robustness of downstream perception, such as +3.2% mAP on MSRS, +0.5% mAP on M3FD and +0.9% mAP on DroneVehicle for object detection. Our code will be available at https://github.com/YiShi99/P2Fusion
Yi Shi, Huichao Xie, Yuqing Wang +7
Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1
Infrared-visible image fusion aims to create an information-rich fused image by integrating the complementary thermal saliency from infrared sensing and fine textures from visible imaging. Such accurate fusion is essential for real-world perception applications in complex scenes, including nighttime autonomous driving, search and rescue, and surveillance, and can further benefit downstream tasks such as semantic segmentation. However, most existing fusion methods rely upon static trained weights that cannot adapt to scene-specific content at inference time, and often suffer from a granularity mismatch when coarse auxiliary semantics are injected, which makes it difficult to simultaneously highlight targets and preserve details. In this work, we propose EAPFusion to address these issues by using self-evolving intrinsic priors instead of relying on external auxiliary models. Concretely, EAPFusion maintains a compact set of intrinsic priors and progressively updates them across scales. These evolved priors are utilized to dynamically generate convolutional kernels, shifting the paradigm from fixed, pre-trained filters to instance-adaptive parameters via prior-conditioned dynamic convolution. Furthermore, we design a channel-level fusion module that shuffles and interleaves infrared and visible channels, applying local channel mixing to boost cross-modal complementarity. Experiments on different datasets, including cross-dataset evaluation and semantic segmentation, show that the proposed method achieves state-of-the-art quantitative and qualitative fusion results, and consistently boosts downstream performance. Code is coming soon.
Zhenyu Sun, Luobin Zhang, Axi Niu +2
Northwestern Polytechnical University, Xi’an, China