ProtoFusion: Protocol-Guided Prompt Refinement for Generative Infrared and Visible Image Fusion

Pengwei Liang; Junjun Jiang; Junhui Hou

cs.CVNov 2, 2026

ProtoFusion: Protocol-Guided Prompt Refinement for Generative Infrared and Visible Image Fusion

Authors: Pengwei Liang, Junjun Jiang, Junhui Hou

Organizations: Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China

Abstract

Infrared and visible image fusion integrates complementary information from both modalities to provide a more comprehensive representation of the scene. We propose ProtoFusion, a protocol-guided generative framework for infrared and visible image fusion. Instead of relying on fixed fusion operators, hand-crafted objectives, or manually specified one-shot prompts, ProtoFusion formulates fusion as protocol-constrained generation. It represents fusion intent with a structured protocol, consisting of environment state cards, grounding principles, and a style profile, which explicitly specifies which evidence should be preserved or suppressed and which properties the fused result should satisfy. To optimize this protocol, we introduce protocol-guided prompt refinement, a closed-loop learning framework that iteratively performs generation, evaluation, and revision under multimodal evidence. Once learned, the optimized protocol is fixed at inference time and directly guides fusion generation. To address residual source inconsistencies, we further incorporate structured belief construction, which organizes modality reliability, scene evidence, and cross-modal conflicts into a structured representation for conservative correction after generation. Extensive experiments on standard benchmarks show that ProtoFusion improves quantitative performance, achieves favorable results in aesthetic evaluation, and produces visually coherent fusion outputs. Project Page: {https://erfect2020.github.io/ProtoFusion/}.

Figures & tables

Figure 1: From implicit fusion biases to explicit protocol-constrained generation. (a) Fusion combines infrared thermal saliency with visible structure and texture. (b) Existing methods encode fusion intent implicitly through rules, networks, or conditioning. (c) ProtoFusion makes this intent explicit through an editable protocol specifying what to preserve and how to render it, with iterative refinement and optional structured belief correction.

Figure 2: Flowchart of protocol-guided prompt refinement (PGPR). PGPR is the main fusion loop. At each round, it composes a prompt from a structured protocol, generates a fused image, diagnoses the current proto-fusion result, predicts a structured edit action, and updates the protocol for the next round. The right panel expands protocol-guided prompt composition, where the protocol is organized into environment state cards, grounding principles, and a style profile. The bottom panel expands protocol-guided refinement, where diagnosis and protocol editing are performed in two stages.

Figure 3: Overview of structured belief construction (SBC). Given the source images and the selected PGPR output $(p^{\star},y^{\star})$ , SBC extracts finer-grained region-level evidence and organizes it into a belief state. This belief state captures modality reliability, cross-modal conflicts, and prompt attribute parsing. A conservative prompt correction is then predicted under a small, scoped edit constraint and used for an additional generation step.

Table 1: Quantitative comparison on FMB (top), MSRS (middle), and RoadScene (bottom). ProtoFusion uses PGPR+SBC on all three datasets. Best and second-best results are highlighted in bold and underlined, respectively.

		FMB				MSRS				RoadScene
Method	Venue	IAA	IQA	ISTA	Overall	IAA	IQA	ISTA	Overall	IAA	IQA	ISTA	Overall
CDDFuse	CVPR’23	21.0344	26.1464	39.4317	28.8708	27.6337	28.9799	41.5625	32.7254	18.5933	19.3558	26.8066	21.5852
DDFM	ICCV’23	22.0183	25.7826	38.4991	28.7667	26.6704	26.9214	38.7362	30.7760	23.2981	24.1244	35.1307	27.5177
TIMFusion	TPAMI’24	24.9254	27.8087	38.1953	30.3098	24.7261	26.4986	34.1963	28.4737	18.9853	19.1075	25.3255	21.1394
Text-IF	CVPR’24	22.8638	29.2772	41.5920	31.2443	26.6080	28.0942	41.1446	31.9489	18.9019	19.7031	27.0297	21.8782
C2RF	IJCV’25	19.1790	23.4826	36.4092	26.3569	25.1766	26.2832	37.9217	29.7939	18.4992	18.5967	22.2783	19.7914

Table 2: UniPercept evaluation on FMB, MSRS, and RoadScene. ProtoFusion uses PGPR+SBC on all three datasets. Higher is better; best and second-best results are bold and underlined, respectively.

Figure 4: Representative visual comparison on FMB, MSRS, and RoadScene. The examples illustrate target visibility and rendering under visible degradations.

Method	Venue	mIoU	mAcc	Road		Building		Sky		Car		Truck		Bus		Veg.		Pole
				IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc
CDDFuse	CVPR’23	0.510	0.632	0.560	0.811	0.575	0.700	0.679	0.692	0.631	0.884	0.450	0.499	0.622	0.837	0.465	0.474	0.100	0.160
DDFM	ICCV’23	0.472	0.588	0.536	0.724	0.585	0.709	0.692	0.704	0.555	0.786	0.269	0.343	0.565	0.797	0.485	0.497	0.089	0.144
TIMFusion	TPAMI’24	0.511	0.627	0.560	0.798	0.555	0.671	0.607	0.621	0.633	0.896	0.487	0.551	0.676	0.839	0.472	0.482	0.100	0.157
Text-IF	CVPR’24	0.513	0.637	0.565	0.807	0.579	0.706	0.675	0.687	0.641	0.888	0.439	0.518	0.643	0.864	0.471	0.481	0.095	0.147
C2RF	IJCV’25	0.482	0.605	0.534	0.781	0.573	0.703	0.607	0.615	0.607	0.865	0.410	0.485	0.607	0.820	0.443	0.450	0.080	0.124

Table 3: Semantic segmentation results on FMB. Best and second-best results are highlighted in bold and underlined, respectively.

Figure 5: Qualitative comparisons on degraded daytime scenes with haze and strong illumination.

Error category	Human Prompt	PGPR	PGPR+SBC
Content alteration	36.43	26.43	1 4 . 2 9
Geometric distortion	5.00	1.79	0 . 7 1
Information omission	4.64	3.93	2 . 8 6
Appearance artifacts	25.36	11.43	3 . 5 7
Any error	53.93	38.21	1 9 . 6 4

Table 4: Image-level error rates (%) on FMB; lower is better. Categories may overlap; Any error is their union. Only definite errors enter the numerator.

Figure 6: Ablation study on the influence of infrared input. Each group shows the original IR/VIS inputs, a shuffled IR input, and the results generated under VIS-only, shuffled IR+VIS, and IR+VIS settings.

Setting	IAA	IQA	ISTA	Overall
Human Prompt	29.1049	39.2397	46.0433	38.1293
Initial Protocol	31.6094	40.7125	45.5683	39.2967
Principles Only	30.6170	40.6623	45.9330	39.0708
w/o State Cards	34.4786	44.3286	48.4580	42.4217
w/o Style Profile	35.3647	45.3844	48.7804	43.1765
Full PGPR	3 9 . 8 0 4 0	5 2 . 3 0 0 4	5 1 . 2 2 8 1	4 7 . 7 7 7 5

Table 5: Protocol components on the FMB test pairs (UniPercept; higher is better; best in bold).

Budget	IAA	IQA	ISTA	Overall
$T=1$	31.6094	40.7125	45.5683	39.2967
$T=3$	32.6509	44.7830	48.3902	41.9414
$T=6$	39.8040	5 2 . 3 0 0 4	51.2281	4 7 . 7 7 7 5
$T=9$	4 0 . 0 7 0 1	51.5103	5 1 . 2 7 5 9	47.6188

Table 6: Refinement budget on all FMB test pairs (UniPercept; higher is better; best in bold).

Panel of Figure 7 — Figure 7: Visualization of the PGPR refinement trajectory. From left to right, we show the intermediate fused results across refinement rounds.

Method	IAA	IQA	ISTA	Overall
Human Prompt	29.1049	39.2397	46.0433	38.1293
VLM-optimized Prompt	37.5263	38.2645	43.3942	39.7283
PGPR	39.8040	52.3004	51.2281	47.7775
PGPR+SBC	4 2 . 0 1 4 3	5 3 . 7 6 9 6	5 1 . 7 6 8 3	4 9 . 1 8 4 1

Table 7: Prompt optimization and SBC comparison on the full FMB test set using UniPercept. Higher is better.

Panel of Figure 8 — Figure 8: Visual comparison of PGPR and PGPR+SBC on representative cases. SBC adjusts local target emphasis and appearance through a single prompt correction.

Panel of Figure 9 — Figure 9: Analysis of the learned protocol. (a) Intrinsic functional factors. (b) Prompt-level control allocation. Blue distributions show the final prompts; blue points and bars show the median and interquartile range; black diamonds denote the human-crafted prompt.

Generator configuration	Time (s/pair)
Qwen-Image-Edit	$\sim$ 188
Qwen-Image-2.1-Turbo	$\sim$ 9

Table 8: Inference latency per image pair, including prompt composition and one image-generation step (excluding optional SBC).

Figure 10: Comparison of image generators under a fixed learned protocol. The columns show the infrared and visible inputs, followed by ProtoFusion with Qwen-Image-Edit and GPT-Image-2. The protocol learned with Qwen-Image-Edit is reused without reoptimization.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12: Failure cases of fixed prompting for infrared and visible image fusion. Although the same handcrafted prompt is used for all image pairs, the generated results show diverse failure modes, including structure collapse, modality domination, loss of visible scene texture, hallucinated objects, unrealistic illumination changes, and severe style drift. These failures show that one-shot fixed prompting cannot reliably constrain the large synthesis space of a generative fusion model, motivating protocol-guided prompt refinement.

Panel of Figure 13 — Figure 13: Qualitative example of hallucination propagation in generative infrared and visible image fusion. A hallucinated “white umbrella” in the interpretation of the visible image is propagated into the fused result, producing a source-inconsistent semantic artifact.

Infrared-Visible Image Fusion

Infrared-visible image fusion combines thermal cues with visible-light detail to create images for human viewing and machine perception.

See all 41 papers

NatureFusion: Rethinking Infrared and Visible Image Fusion as Natural Complementary Enhancement
Nov 1, 2026Jianfeng Ding, Linfeng Tang, Hao Zhang +6Infrared-Visible Image Fusion Multi-Source Image Fusion
the National Engineering Research Center for Multimedia Software (NERCMS), School of Computer Science, Wuhan University, Wuhan 430072, China · the School of Robotics, Wuhan University, Wuhan 430072, China · the Electronic Information School, Wuhan University, Wuhan 430072, China +1
P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Aug 13, 2026Yi Shi, Huichao Xie, Yuqing Wang +7Multi-Source Image Fusion Infrared-Visible Image Fusion
Northwestern Polytechnical University, Xi’an, China · Hefei University of Technology, Hefei, China · Rocket Force University of Engineering, Xi’an, China +1
ConFusion: Continuous Fusion Space Learning for Fine-Grained Controllable Infrared and Visible Image Fusion
Jul 26, 2026Guo Yurong, He Yufei, Li Yonghao +3Multi-Source Image Fusion Infrared-Visible Image Fusion
North China Electric Power University Baoding, Hebei, China · Beijing University of Posts and Telecommunications Beijing, China

Nov 1, 2026J/K move · Enter open · S save

NatureFusion: Rethinking Infrared and Visible Image Fusion as Natural Complementary Enhancement

Jianfeng Ding, Linfeng Tang, Hao Zhang, Han Xu+5

the National Engineering Research Center for Multimedia Software (NERCMS), School of Computer Science, Wuhan University, Wuhan 430072, China · the School of Robotics, Wuhan University, Wuhan 430072, China · the Electronic Information School, Wuhan University, Wuhan 430072, China+1

Infrared-Visible Image Fusion Multi-Source Image Fusion

Open paper AskPDF arXiv

In the absence of ground truth, most existing infrared and visible image fusion methods pursue fidelity to the source images and directly superimpose two physically distinct modalities. As a result, salient infrared targets appear as bright patches of thermal radiation pasted onto the scene. These patches obscure the original color and texture of the targets, so the fused images look unnatural to human observers and deviate from the data distribution learned by vision models pretrained on visible images. Therefore, we redefine infrared and visible image fusion as natural complementary enhancement rather than source fidelity, and propose NatureFusion, which exploits cross-modal complementary cues to infer a clear and natural visible-style image from low-quality visible and infrared image pairs. Specifically, to address the lack of ground truth, we employ a generative foundation model as the teacher to directly generate visually natural fused images for supervision. Furthermore, to fully distill the teacher's generative capability while reducing computational cost, we fine-tune a pretrained generative model with conditional flow matching through a light-weight fusion adapter. Last but not least, to further improve downstream performance without sacrificing natural appearance, we freeze the fusion adapter and introduce a detector-supervised task adapter that strengthens the complementary contribution of infrared. Extensive comparisons with seven multi-modal fusion and seven single-modality enhancement methods demonstrate that NatureFusion generates visually natural fused images with the best perceptual quality. Detection and segmentation models pretrained on visible images achieve substantially higher accuracy on our fused images than on those of any competing method.

Abstract

Infrared-Visible Image Fusion

Explore similar work

NatureFusion: Rethinking Infrared and Visible Image Fusion as Natural Complementary Enhancement

P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation

ConFusion: Continuous Fusion Space Learning for Fine-Grained Controllable Infrared and Visible Image Fusion