Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

Jingxue Huang; Xiaosong Li; Yi Lin

cs.CVOct 26, 2026

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

Authors: Jingxue Huang, Xiaosong Li, Yi Lin

Organizations: College of Computer Science, Sichuan University, Chengdu 610000, China · Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University, Foshan 528225, China

Abstract

Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.

Figures & tables

Fig. 1: Comparison with an existing video fusion method [ 4 ] in terms of temporal interaction schemes, fusion quality and computational efficiency. All reported metrics are evaluated on the VTMOT dataset.

Fig. 2: Overview of the proposed multi-modal video fusion framework.

Fig. 3: Qualitative comparison of all methods on the M3SVD, VTUAV, HDO and VTMOT datasets.

Type	Pub.	Method	$\mathrm{Q}_{\mathrm{G}}\uparrow$	$\mathrm{Q}_{\mathrm{M}}\uparrow$	$\mathrm{VIF}\uparrow$	$\mathrm{FSIM}\uparrow$	$\mathrm{feaCD}\downarrow$	$\mathrm{flowD}\downarrow$	$\mathrm{FD3}\downarrow$	$\mathrm{Q}_{\mathrm{G}}\uparrow$	$\mathrm{Q}_{\mathrm{M}}\uparrow$	$\mathrm{VIF}\uparrow$	$\mathrm{FSIM}\uparrow$	$\mathrm{feaCD}\downarrow$	$\mathrm{flowD}\downarrow$	$\mathrm{FD3}\downarrow$
Type	Pub.	Method	M3SVD Dataset							VTUAV Dataset
Image	TIP 26	DSPFusion	0.61	0.58	0.31	0.782	1.76	8.72	1.15	0.57	0.93	0.34	0.793	1.73	12.51	1.09
	CVPR 26	ISFusion	0.51	0.35	0.25	0.777	1.69	6.96	1.12	0.48	0.74	0.33	0.808	1.77	9.35	1.05
	CVPR 25	GIFNet	0.47	0.25	0.24	0.776	1.78	6.32	1.14	0.39	0.55	0.27	0.804	1.81	10.23	1.09
	IJCV 24	CoCoNet	0.35	0.19	0.18	0.697	1.76	11.15	1.30	0.33	0.41	0.21	0.739	1.83	17.47	1.18
Video	CVPR 26	VideoFusion	0.36	0.30	0.30	0.802	1.70	5.81	1.33	0.29	0.49	0.30	0.811	1.94	7.95	1.36

TABLE I: Quantitative comparisons across the M3SVD, VTUAV, HDO, and VTMOT datasets. The best, second-best, and third-best results are denoted by bestcolor , secondcolor , and thirdcolor , respectively.

Fig. 4: Qualitative comparisons of the temporal consistency for source and fused videos in Scene ’0111_1803’ from the M3SVD dataset. We visualize the pixels of the selected column (the red line).

Fig. 5: Temporal curves of different metrics over consecutive frames of the sequence from Scene ’0111_1803’ of the M3SVD dataset. Mean denotes average value and SD represents standard deviation.

Fig. 6: Comparison of segmentation results produced by different video fusion methods on scene ‘0118_1913’ of the M3SVD dataset. The bottom row of each image presents cropped views of the blue-boxed region at two-frame intervals from $T$ to $T+8$ . The segmentation prompt used for this scene is ’the person on the right side wearing a hoodie.’

Fig. 7: Bubble chart comparing the computational efficiency and fusion performance of different methods. The bubble size is determined by the average rank of each method over all datasets and evaluation metrics, where a larger bubble denotes better overall performance.

Fig. 8: Visualization of variation responses in the pixel space and different pretrained feature spaces for Scene ’002’ from the HDO dataset. (a) Adjacent-frame variation. (b) Variation from the temporal mean.

	CLIP	DINO	MAE
$\rho=0.10$	0.21	0.40	0.58
$\rho=0.20$	0.30	0.46	0.60
$\rho=0.25$	0.33	0.48	0.60
$\rho=0.30$	0.36	0.49	0.60
$\rho=0.40$	0.42	0.52	0.62

TABLE II: IoU of the Top-

K

high-response regions selected from the pixel-level variation map and different pretrained feature maps. The best result in each row is highlighted in bold.

	QG $\uparrow$	QM $\uparrow$	VIF $\uparrow$	FSIM $\uparrow$	feaCD $\downarrow$	flowD $\downarrow$	FD3 $\downarrow$
$\rho=0.10$	0.553	0.646	0.310	0.808	1.715	10.795	1.041
$\rho=0.25$	0.554	0.649	0.308	0.809	1.712	10.738	1.034
$\rho=0.50$	0.549	0.640	0.307	0.809	1.713	10.650	1.037
$\rho=1.00$	0.544	0.615	0.303	0.807	1.721	10.751	1.038

TABLE III: Quantitative comparison with different

\rho

on HDO dataset. The best and second-best results are denoted by bestcolor and secondcolor , respectively.

Perturbation	Setting	feaCD $\downarrow$	flowD $\downarrow$	FD3 $\downarrow$
Clean	Baseline	1.71	10.74	1.03
Misalign	t=1	2.04 (+19.0%)	10.85 (+1.1%)	1.36 (+31.1%)
	t=2	2.02 (+18.0%)	11.19 (+4.2%)	1.50 (+44.9%)
	t=3	1.99 (+16.0%)	11.39 (+6.0%)	1.59 (+53.6%)
Local Lag	area=0.6	1.97 (+14.9%)	11.22 (+4.5%)	1.38 (+34.0%)
	area=0.5	1.96 (+14.5%)	11.16 (+3.9%)	1.33 (+29.1%)

TABLE IV: Sensitivity comparison of temporal metrics under different degradation types. Each degradation type is evaluated under progressively more challenging settings. Values in parentheses denote relative changes with respect to the clean baseline. The baseline values are obtained from the proposed method on the HDO dataset without degradation. The most desirable response in each row, jointly considering sensitivity, correct response direction, and monotonicity, is highlighted in bold.

	Baseline	area=0.60	area=0.50	area=0.40
CLIP	1.72	1.84 (+7%)	1.83 (+6%)	1.81 (+6%)
DINO	1.35	1.60 (+19%)	1.57 (+17%)	1.54 (+14%)
MAE	1.03	1.38 (+34%)	1.33 (+29%)	1.28 (+24%)

TABLE V: Sensitivity comparison of FD3 using CLIP [ 55 ] , DINO [ 56 ] , and MAE as pretrained feature extractors under local content lag perturbations. Percentages in parentheses denote relative changes from the clean baseline. The baseline values are obtained by evaluating each metric on the unperturbed fusion results of the proposed method on the HDO dataset. The largest relative increase from the corresponding clean baseline is highlighted in bold.

	$\mathrm{Q}_{\mathrm{G}}$	$\mathrm{Q}_{\mathrm{M}}$	$\mathrm{VIF}$	$\mathrm{FSIM}$	$\mathrm{feaCD}$	$\mathrm{flowD}$	$\mathrm{FD3}$
w ViT	0.551	0.635	0.306	0.808	1.713	10.771	1.037
w/o STIM	0.543	0.633	0.305	0.809	1.717	10.537	1.052
w CLIP	0.522	0.621	0.309	0.809	1.728	10.427	1.052
w DINO	0.539	0.643	0.321	0.812	1.713	10.476	1.051
w/o $\mathcal{L}_{\mathrm{rev}}$	0.553	0.641	0.308	0.809	1.720	10.764	1.035
w/o $\mathcal{L}_{\mathrm{temp}}$ & $\mathcal{L}_{\mathrm{rev}}$	0.536	0.628	0.302	0.806	1.722	10.984	1.055

TABLE VI: Ablation study of architectural variants, pretrained feature spaces, and loss terms on the HDO dataset. The best results are highlighted in bold.

Fig. 9: Computational efficiency of ViT and Ours with respect to the number of input frames at two spatial resolutions. For each line chart, the left and right groups correspond to resolutions of

640\times 480

and

1280\times 1024

, respectively.

Fig. 10: Visual comparison of temporal consistency between two ablation variants: (II) w/o STIM and (VI) w/o $\mathcal{L}_{\mathrm{temp}}$ & $\mathcal{L}_{\mathrm{rev}}$ .

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 777 papers

MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion
Oct 29, 2026Meiqi Gong, Hao Zhang, Jiayi MaMultimodal Fusion Temporal Consistency
Electronic Information School, Wuhan University, Wuhan, China · School of Robotics, Wuhan University, Wuhan, China
MambaVF: State Space Model for Efficient Video Fusion
Feb 5, 2026Zixiang Zhao, Yukun Cui, Lilun Deng +4Multimodal Fusion
ETH Zürich · Xi’an Jiaotong University · Nanyang Technological University +1
MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction
Apr 2, 2026Xilai Li, Weijun Jiang, Xiaosong Li +5Multimodal Fusion
Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China

Oct 29, 2026J/K move · Enter open · S save

MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion

Meiqi Gong, Hao Zhang, Jiayi Ma

Electronic Information School, Wuhan University, Wuhan, China · School of Robotics, Wuhan University, Wuhan, China

Multimodal Fusion Temporal Consistency

Open paper AskPDF arXiv

Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.

Abstract

Multimodal Fusion

Explore similar work

MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion

MambaVF: State Space Model for Efficient Video Fusion

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction