MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion

Meiqi Gong; Hao Zhang; Jiayi Ma

cs.CVOct 29, 2026

MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion

Authors: Meiqi Gong, Hao Zhang, Jiayi Ma

Organizations: Electronic Information School, Wuhan University, Wuhan, China · School of Robotics, Wuhan University, Wuhan, China

Abstract

Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.

Figures & tables

Figure 1: Comparison of existing video fusion methods and our proposed MemoryFusion. (a) and (b) illustrate the differences in how temporal information is exploited. (c) shows the corresponding visual comparison results, where the top-right region displays the results of stacked 40 consecutive frames in the same row for temporal consistency evaluation. Higher VIF means better fusion quality and lower flowD means better temporal consistency.

Figure 2: The overall framework of the proposed MemoryFusion.

Figure 3: Implementation of memory banks.

Figure 4: Qualitative results on M3SVD dataset under degradation.

Method	M3SVD						HDO
Method	MI $\uparrow$	CC $\uparrow$	SCD $\uparrow$	VIF $\uparrow$	flowD $\downarrow$	feaCD $\downarrow$	MI $\uparrow$	CC $\uparrow$	SCD $\uparrow$	VIF $\uparrow$	flowD $\downarrow$	feaCD $\downarrow$
LRRNet	1.7889	0.4584	1.4884	0.3906	5.5065	1.9309	1.0616	0.6725	0.7250	0.2416	7.4598	1.9523
Diff-IF	2.0692	0.4912	1.6068	0.4847	4.0890	1.8210	1.1387	0.6851	1.1483	0.3039	8.0450	1.9587
MMIF-EMMA	2.1175	0.4758	1.6443	0.4758	4.6592	1.8412	1.1668	0.6945	1.2582	0.3071	7.9598	1.9565
Text-DiFuse	1.6253	0.4576	1.4989	0.3514	8.6578	1.9744	1.1682	0.7024	1.4096	0.2940	9.7262	1.9732
TC-MoA	2.0178	0.4660	1.5276	0.4054	4.1954	1.8118	1.0942	0.7023	1.1639	0.3181	7.8125	1.9570

Table 1: Quantitative results on the M3SVD and HDO datasets. Bold indicates the best result.

Figure 5: Qualitative results on HDO dataset under degradation.

Figure 6: Temporal consistency comparison.

M3SVD	MI $\uparrow$	CC $\uparrow$	SCD $\uparrow$	VIF $\uparrow$	flowD $\downarrow$	feaCD $\downarrow$
w/o TSM	2.0173	0.4821	1.5734	0.4943	4.2172	1.8038
w/o STMB	2.0239	0.4767	1.5538	0.4817	3.6586	1.8061
random LTMB	2.0199	0.4755	1.5580	0.4825	3.7010	1.8036
w/o LTMB	2.0031	0.4646	1.5509	0.4770	3.7768	1.8078
w/o stage2 tloss	2.1987	0.4884	1.5961	0.4932	3.2065	1.7985
w/o stage2 Conv3D	2.1007	0.4942	1.5994	0.4938	3.5290	1.7956

Table 2: Quantitative results of ablation experiments.

Figure 7: Qualitative results of ablation experiments.

Method	MI $\uparrow$	CC $\uparrow$	SCD $\uparrow$	VIF $\uparrow$	flowD $\downarrow$	feaCD $\downarrow$
MMIF-EMMA	2.1175	0.4758	1.6443	0.4758	4.6592	1.8412
MMIF-EMMA+TRM	2.0641	0.4932	1.6624	0.4762	2.9549	1.8125
UniVF	2.1316	0.4848	1.6122	0.4677	4.0271	1.8138
UniVF+TRM	2.0337	0.4921	1.6325	0.4673	2.8025	1.8122
Ours-stage1	2.1631	0.4872	1.5975	0.4877	3.6135	1.8088
MemoryFusion	2.2350	0.4952	1.6152	0.4926	2.6759	1.7640

Table 3: Quantitative comparison of applying TRM to different fusion methods. The best results are highlighted in bold.

	LRRNet	Diff-IF	MMIF-EMMA	Text-DiFuse	TC-MoA	TemCoCo	VideoFusion	UniVF	Stage1	Stage2
FLOPs (G)	14.2	86.9	41.5	1268	4833	505.6	301.1	740.7	213.6	1.65
Param (M)	0.047	22.36	1.52	119.4	340	19.18	6.74	9.15	17.2	0.033
Time (s)	0.162	4.23	0.056	24.2	0.185	0.109	0.105	0.245	0.099	0.001

Table 4: Comparison of computational efficiency.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure s1 : Results of medical sequence fusion on Harvard dataset.

Figure s2 : Qualitative results of non-degraded scene.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 777 papers

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction
Oct 26, 2026Jingxue Huang, Xiaosong Li, Yi LinMultimodal Fusion
College of Computer Science, Sichuan University, Chengdu 610000, China · Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University, Foshan 528225, China
MambaVF: State Space Model for Efficient Video Fusion
Feb 5, 2026Zixiang Zhao, Yukun Cui, Lilun Deng +4Multimodal Fusion
ETH Zürich · Xi’an Jiaotong University · Nanyang Technological University +1
DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion
May 25, 2026Xingyuan Li, Haoyuan Xu, Shulin Li +3Temporal Consistency Multimodal Fusion
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Software Technology & DUT-RU International School of ISE, Dalian University of Technology, Dalian, China · College of Information Science and Technology, Dalian Maritime University, Dalian, China

Oct 26, 2026J/K move · Enter open · S save

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

Jingxue Huang, Xiaosong Li, Yi Lin

College of Computer Science, Sichuan University, Chengdu 610000, China · Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University, Foshan 528225, China

Multimodal Fusion

Open paper AskPDF arXiv

Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.

Abstract

Multimodal Fusion

Explore similar work

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

MambaVF: State Space Model for Efficient Video Fusion

DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion