Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction
Authors: Jingxue Huang, Xiaosong Li, Yi Lin
Organizations: College of Computer Science, Sichuan University, Chengdu 610000, China · Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University, Foshan 528225, China
Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.
Figures & tables
Fig. 1: Comparison with an existing video fusion method [ 4 ] in terms of temporal interaction schemes, fusion quality and computational efficiency. All reported metrics are evaluated on the VTMOT dataset.
Fig. 2: Overview of the proposed multi-modal video fusion framework.
Fig. 3: Qualitative comparison of all methods on the M3SVD, VTUAV, HDO and VTMOT datasets.
Type
Pub.
Method
QG↑
QM↑
VIF↑
FSIM↑
feaCD↓
flowD↓
FD3↓
QG↑
QM↑
VIF↑
FSIM↑
feaCD↓
flowD↓
FD3↓
M3SVD Dataset
VTUAV Dataset
Image
TIP 26
DSPFusion
0.61
0.58
0.31
0.782
1.76
8.72
1.15
0.57
0.93
0.34
0.793
1.73
12.51
1.09
CVPR 26
ISFusion
0.51
0.35
0.25
0.777
1.69
6.96
1.12
0.48
0.74
0.33
0.808
1.77
9.35
1.05
CVPR 25
GIFNet
0.47
0.25
0.24
0.776
1.78
6.32
1.14
0.39
0.55
0.27
0.804
1.81
10.23
1.09
IJCV 24
CoCoNet
0.35
0.19
0.18
0.697
1.76
11.15
1.30
0.33
0.41
0.21
0.739
1.83
17.47
1.18
Video
CVPR 26
VideoFusion
0.36
0.30
0.30
0.802
1.70
5.81
1.33
0.29
0.49
0.30
0.811
1.94
7.95
1.36
TABLE I: Quantitative comparisons across the M3SVD, VTUAV, HDO, and VTMOT datasets. The best, second-best, and third-best results are denoted by bestcolor , secondcolor , and thirdcolor , respectively.
Fig. 4: Qualitative comparisons of the temporal consistency for source and fused videos in Scene ’0111_1803’ from the M3SVD dataset. We visualize the pixels of the selected column (the red line).
Fig. 5: Temporal curves of different metrics over consecutive frames of the sequence from Scene ’0111_1803’ of the M3SVD dataset. Mean denotes average value and SD represents standard deviation.
Fig. 6: Comparison of segmentation results produced by different video fusion methods on scene ‘0118_1913’ of the M3SVD dataset. The bottom row of each image presents cropped views of the blue-boxed region at two-frame intervals from T to T+8 . The segmentation prompt used for this scene is ’the person on the right side wearing a hoodie.’
Fig. 7: Bubble chart comparing the computational efficiency and fusion performance of different methods. The bubble size is determined by the average rank of each method over all datasets and evaluation metrics, where a larger bubble denotes better overall performance.
Fig. 8: Visualization of variation responses in the pixel space and different pretrained feature spaces for Scene ’002’ from the HDO dataset. (a) Adjacent-frame variation. (b) Variation from the temporal mean.
CLIP
DINO
MAE
ρ=0.10
0.21
0.40
0.58
ρ=0.20
0.30
0.46
0.60
ρ=0.25
0.33
0.48
0.60
ρ=0.30
0.36
0.49
0.60
ρ=0.40
0.42
0.52
0.62
TABLE II: IoU of the Top- K high-response regions selected from the pixel-level variation map and different pretrained feature maps. The best result in each row is highlighted in bold.
QG ↑
QM ↑
VIF ↑
FSIM ↑
feaCD ↓
flowD ↓
FD3 ↓
ρ=0.10
0.553
0.646
0.310
0.808
1.715
10.795
1.041
ρ=0.25
0.554
0.649
0.308
0.809
1.712
10.738
1.034
ρ=0.50
0.549
0.640
0.307
0.809
1.713
10.650
1.037
ρ=1.00
0.544
0.615
0.303
0.807
1.721
10.751
1.038
TABLE III: Quantitative comparison with different ρ on HDO dataset. The best and second-best results are denoted by bestcolor and secondcolor , respectively.
Perturbation
Setting
feaCD ↓
flowD ↓
FD3 ↓
Clean
Baseline
1.71
10.74
1.03
Misalign
t=1
2.04 (+19.0%)
10.85 (+1.1%)
1.36 (+31.1%)
t=2
2.02 (+18.0%)
11.19 (+4.2%)
1.50 (+44.9%)
t=3
1.99 (+16.0%)
11.39 (+6.0%)
1.59 (+53.6%)
Local Lag
area=0.6
1.97 (+14.9%)
11.22 (+4.5%)
1.38 (+34.0%)
area=0.5
1.96 (+14.5%)
11.16 (+3.9%)
1.33 (+29.1%)
TABLE IV: Sensitivity comparison of temporal metrics under different degradation types. Each degradation type is evaluated under progressively more challenging settings. Values in parentheses denote relative changes with respect to the clean baseline. The baseline values are obtained from the proposed method on the HDO dataset without degradation. The most desirable response in each row, jointly considering sensitivity, correct response direction, and monotonicity, is highlighted in bold.
Baseline
area=0.60
area=0.50
area=0.40
CLIP
1.72
1.84 (+7%)
1.83 (+6%)
1.81 (+6%)
DINO
1.35
1.60 (+19%)
1.57 (+17%)
1.54 (+14%)
MAE
1.03
1.38 (+34%)
1.33 (+29%)
1.28 (+24%)
TABLE V: Sensitivity comparison of FD3 using CLIP [ 55 ] , DINO [ 56 ] , and MAE as pretrained feature extractors under local content lag perturbations. Percentages in parentheses denote relative changes from the clean baseline. The baseline values are obtained by evaluating each metric on the unperturbed fusion results of the proposed method on the HDO dataset. The largest relative increase from the corresponding clean baseline is highlighted in bold.
QG
QM
VIF
FSIM
feaCD
flowD
FD3
w ViT
0.551
0.635
0.306
0.808
1.713
10.771
1.037
w/o STIM
0.543
0.633
0.305
0.809
1.717
10.537
1.052
w CLIP
0.522
0.621
0.309
0.809
1.728
10.427
1.052
w DINO
0.539
0.643
0.321
0.812
1.713
10.476
1.051
w/o Lrev
0.553
0.641
0.308
0.809
1.720
10.764
1.035
w/o Ltemp & Lrev
0.536
0.628
0.302
0.806
1.722
10.984
1.055
TABLE VI: Ablation study of architectural variants, pretrained feature spaces, and loss terms on the HDO dataset. The best results are highlighted in bold.
Fig. 9: Computational efficiency of ViT and Ours with respect to the number of input frames at two spatial resolutions. For each line chart, the left and right groups correspond to resolutions of 640×480 and 1280×1024 , respectively.
Fig. 10: Visual comparison of temporal consistency between two ablation variants: (II) w/o STIM and (VI) w/o Ltemp & Lrev .
Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.
Meiqi Gong, Hao Zhang, Jiayi Ma
Electronic Information School, Wuhan University, Wuhan, China · School of Robotics, Wuhan University, Wuhan, China
Video fusion aims to integrate complementary information from multiple source videos while preserving temporal consistency. Effective modeling of temporal dynamics is essential to this goal, yet existing methods incur substantial computational overhead from optical flow estimation and feature warping. In this paper, we present MambaVF, an efficient video fusion framework that uses state space model (SSM) to achieve temporal modeling without explicit motion estimation. First, by formulating video fusion as a sequential state update process, MambaVF captures long-range temporal dependencies with linear complexity, significantly reducing computation and memory costs. Second, the lightweight SSM-based fusion module eliminates conventional flow-guided alignment. Instead, it introduces a mutual state fusion module and a spatio-temporal bidirectional scanning mechanism to enable information aggregation across video streams. Experiments on multiple benchmarks confirm that MambaVF reaches state-of-the-art performance in different video fusion applications (multi-exposure, multi-focus, infrared-visible, medical), while reducing parameters by >90% and FLOPs by >80%, resulting in >50% shorter runtime. Project page: https://mambavf.github.io
Zixiang Zhao, Yukun Cui, Lilun Deng +4
ETH Zürich · Xi’an Jiaotong University · Nanyang Technological University +1
Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16 FPS at 640×480 resolution. The source code will be available at https://github.com/ixilai/MAVFusion.
Xilai Li, Weijun Jiang, Xiaosong Li +5
Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China