FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Xilai Li; Chusheng Fang; Xiaosong Li

cs.CVApr 2, 2026

FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Authors: Xilai Li, Chusheng Fang, Xiaosong Li

Organizations: Foshan University, China

Abstract

Infrared and visible video fusion plays a critical role in intelligent surveillance and low-light monitoring. However, maintaining temporal stability while preserving spatial detail remains a fundamental challenge. Existing methods either focus on frame-wise enhancement with limited temporal modeling or rely on heavy spatio-temporal aggregation that often sacrifices high-frequency details. In this paper, we propose FTPFusion, a frequency-aware infrared and visible video fusion method based on temporal perturbation and sparse cross-modal interaction. Specifically, FTPFusion decomposes the feature representations into high-frequency and low-frequency components for collaborative modeling. The high-frequency branch performs sparse cross-modal spatio-temporal interaction to capture motion-related context and complementary details. The low-frequency branch introduces a temporal perturbation strategy to enhance robustness against complex video variations, such as flickering, jitter, and local misalignment. Furthermore, we design an offset-aware temporal consistency constraint to explicitly stabilize cross-frame representations under temporal disturbances. Extensive experiments on multiple public benchmarks demonstrate that FTPFusion consistently outperforms state-of-the-art methods across multiple metrics in both spatial fidelity and temporal consistency. The source code will be available at https://github.com/ixilai/FTPFusion.

Figures & tables

Figure 1. Comparison of the Limitations of Two Mainstream Video Fusion Methods.

Figure 2. The overall framework of the proposed algorithm.

Figure 3. Qualitative comparison of all methods on the M3SVD, HDO and VTMOT datasets.

Figure 4. Visual comparison of the temporal stability of different methods.

Figure 5. Comparison of segmentation results of different fusion methods in the video segmentation task.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Mode	Metric	Global Spearman $\uparrow$	Global Pearson $\uparrow$	Mean Seq. Spearman $\uparrow$	Monotonic Rate $\uparrow$	Pairwise Acc. $\uparrow$	Adjacent Sep. $\uparrow$	Avg. Rank $\downarrow$
Mixed Hard	BiSWE	0.8848	0.8798	0.8720	0.8889	0.9926	2.9125	1.625
	MMCI	0.9363	0.8626	0.8667	0.7778	0.9704	3.2218	1.625
	TCPE	0.4450	0.4148	0.5840	0.6111	0.8519	0.7433	2.750
	MS2R	-0.0261	-0.0753	0.2587	0.0000	0.5259	0.0081	4.000
Modality Drift	MMCI	0.9794	0.9588	0.9573	0.8889	0.9926	3.3475	1.625
	TCPE	0.9586	0.9560	0.9520	1.0000	1.0000	2.1464	1.625

Appendix

Table 3. Effectiveness validation of temporal metrics under the proposed stress benchmark. Best results in each column are in bold.

Perturbation	Strength	$E_{L}$	$E_{H}$	$R_{L}$	$\eta_{L}$	$\eta_{H}$
Flicker	0.0	0.000734	0.000371	0.589846	0.996621	0.940629
	0.5	0.000748	0.000371	0.598594	0.996591	0.940627
	1.0	0.000793	0.000371	0.616832	0.996481	0.940639
	1.5	0.000864	0.000371	0.659099	0.996323	0.940599
	2.0	0.000961	0.000371	0.691016	0.996096	0.940567
Jitter	0.00	0.000734	0.000371	0.589846	0.996621	0.940629

Appendix

Table 4. Temporal energy analysis under controlled perturbations.

E_{L}

and

E_{H}

denote the temporal variation energies of the low- and high-frequency branches, respectively;

R_{L}

is the low-frequency energy ratio;

\eta_{L}

and

\eta_{H}

are the temporal low-band energy ratios of the two branches.

Panel of Figure 6 — Figure 6. Qualitative comparison of all methods on two datasets.

Figure 7. Qualitative comparison of all methods on the M3SVD dataset.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 771 papers

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction
Apr 2, 2026Xilai Li, Weijun Jiang, Xiaosong Li +5Multimodal Fusion
Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China
DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion
May 25, 2026Xingyuan Li, Haoyuan Xu, Shulin Li +3Temporal Consistency Multimodal Fusion
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Software Technology & DUT-RU International School of ISE, Dalian University of Technology, Dalian, China · College of Information Science and Technology, Dalian Maritime University, Dalian, China
Uncertainty-aware Spatial-Frequency Registration and Fusion for Infrared and Visible Images
May 13, 2026Xingyuan Li, Haoyuan Xu, Xingyue Zhu +4Multi-Source Image Fusion Infrared-Visible Image Fusion
Dalian University of Technology · Northwestern Polytechnical University · Dalian Maritime University

Apr 2, 2026J/K move · Enter open · S save

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Xilai Li, Weijun Jiang, Xiaosong Li, Yang Liu+4

Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China

Multimodal Fusion

Open paper AskPDF arXiv

Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16 FPS at

640 \times 480

resolution. The source code will be available at https://github.com/ixilai/MAVFusion.

Abstract

Multimodal Fusion

Explore similar work

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion

Uncertainty-aware Spatial-Frequency Registration and Fusion for Infrared and Visible Images