STAR-VF: Spatio-Temporal Aware Representation Condensation for Data-Efficient Video Fusion
Authors: Yuchen Xian, Yunqiu Xu, Guosheng Lin, Yang He
Organizations: Zhejiang University, Hangzhou, Zhejiang, China · Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore
Video fusion aims to integrate complementary cues from heterogeneous sources, producing visually informative videos with coherent spatio-temporal structures. Jointly learning cross-modal complementarity and temporal dependencies typically relies on large-scale, high-quality video datasets, whose storage and repeated use during optimization incur substantial costs, making data-efficient learning highly desirable. To address this challenge, we rethink video fusion from the perspective of dataset condensation and propose STAR-VF, the first framework that condenses spatio-temporal representations for data-efficient video fusion. STAR-VF condenses the original training set into a compact set of learnable spatio-temporal representations, which are decoded into synthetic video clips and optimized using feature-level supervision provided by a frozen proxy model. The resulting condensed data preserves fusion-relevant spatial and temporal cues without requiring ground-truth fused videos as supervision. To adapt fusion learning to condensed representations, we equip the final fusion model with flow-free inter-frame temporal modeling and query-driven intra-frame aggregation, enabling direct temporal interaction and adaptive selection of fusion-relevant information in representation space. Extensive experiments on infrared-visible, multi-exposure, multi-focus, and medical video fusion benchmarks demonstrate that STAR-VF, trained with only approximately 1/16 of the original temporal training set, achieves fusion quality comparable to methods trained on the full data. Further analyses validate the contributions of learned condensation and flow-free temporal modeling, and demonstrate the robustness of the condensed supervision across different proxy encoders and downstream fusion models. Project page: https://zju-xyc.github.io/STAR-VF-Project-Page/.
Figures & tables
Figure 1. Comparison of temporal alignment paradigms under video fusion dataset condensation. Conventional flow-based alignment is not naturally suited to condensed synthetic clips , whereas our method models order-aware temporal dependency in compact representation space.
Figure 2. Overview of the proposed STAR-VF framework. In Stage I, learnable compact spatio-temporal canvases and the canvas decoder U are jointly optimized under feature-level supervision from a frozen proxy feature extractor, yielding the learned condensed set S∗ and decoder U∗ . In Stage II, the learned condensed set is decoded into synthetic temporal samples to train the final fusion model Fθ with flow-free temporal alignment and query-driven intra-frame aggregation. During inference, only the trained fusion model is used, without the condensed set or proxy model.
Method
Train Ratio
IVF (VTMOT ( Zhu et al., 2025 ) )
MVF (Harvard ( Summers, 2003 ) )
MEF (YouTube-HDR ( Wang et al., 2024 ) )
MFF (DAVIS ( Pont-Tuset et al., 2017 ) )
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF ( Zhao et al., 2025 )
1
0.65
6.81
0.65
3.95
0.35
0.78
4.27
0.75
29.92
1.23
0.96
7.25
0.65
6.44
0.34
0.98
7.41
0.90
6.04
1.11
VideoFusion ( Tang et al., 2026 )
1
0.62
6.59
0.62
3.14
0.59
0.78
4.91
0.69
19.71
1.31
0.91
6.82
0.58
9.98
1.82
0.96
7.34
0.82
9.07
6.13
TemCoCo ( Gong et al., 2025 )
1
0.63
6.68
0.60
6.26
0.59
0.77
5.40
0.24
28.41
1.57
0.90
7.34
0.53
11.62
1.70
0.97
7.45
0.84
12.34
4.08
MambaVF ( Zhao et al., 2026b )
1
-
-
0.65
3.91
0.36
-
-
0.76
29.61
1.27
-
-
-
6.96
0.16
-
-
-
8.29
-
STAR-VF (ours)
∼ 1/16
0.64
6.73
0.65
3.91
0.35
0.80
4.37
0.76
28.36
1.26
0.91
7.28
0.55
10.23
1.28
0.96
7.42
0.91
5.53
1.18
Table 1. Quantitative comparison on IVF (VTMOT), MVF (Harvard), MEF (YouTube-HDR), and MFF (DAVIS). The training-data ratio rtrain denotes the fraction of the original temporal training units used to train each method. STAR-VF uses only approximately 1/16 of the original training source, yet achieves fusion quality comparable to the full-data baselines. The best and second-best results are highlighted in red and blue , respectively.
Figure 3. Qualitative comparison on four representative benchmarks. For each example, we show the input pair, our method, the full-size variant trained on the original data, and competing video fusion baselines ( VideoFusion , UniVF , and TemCoCo ). The cyan boxes indicate enlarged local regions for detailed comparison.
Figure 4. Frame-by-frame qualitative comparison on a representative multi-source video fusion sequence.
Dataset
Task
Train seqs
Nreal
Ns
Ncond
rcond
VTMOT
IVF
75
22,500
88
1,408
1/16.0
Harvard
MVF
49
1,308
6
96
1/13.6
YouTube-HDR
MEF
450
66,595
261
4,176
1/15.9
DAVIS
MFF
120
8,195
32
512
1/16.0
Table 2. Training-source budgets used by STAR-VF.
Model
Resolution
Params
GFLOPs
Latency
Memory
UniVF ( Zhao et al., 2025 )
256 × 256
9.16M
934.7
531.4ms
696.8MB
Condensed (ours)
256 × 256
0.32M
184.0
110.5ms
696.8MB
UniVF ( Zhao et al., 2025 )
480 × 854
9.16M
6139.1
820.7ms
4110.5MB
Condensed (ours)
480 × 854
0.32M
1066.6
468.5ms
4110.5MB
Table 3. Computational efficiency comparison.
Figure 5. Visual comparison between the full dataset and the learned condensed dataset.
Figure 6. Alignment consistency comparison across video fusion methods, evaluated by frame-wise aligned ROI luminance trajectories after min-max normalization
Method
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF ( Zhao et al., 2025 )
0.65
6.81
0.65
3.95
0.35
w/o condensation
0.64
6.73
0.64
3.91
0.36
Ours (w/o RAFT)
0.64
6.73
0.65
3.91
0.35
Table 4. Ablation of flow-free temporal modeling on IVF.
Grid
#Frames
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
1 × 1 (full)
1
0.64
6.73
0.64
3.91
0.36
2 × 2
4
0.90
7.25
0.87
5.07
0.36
3 × 3
9
0.89
7.82
0.88
5.10
0.36
4 × 4 (ours)
16
0.64
6.73
0.65
3.91
0.35
5 × 5
25
0.90
6.62
0.88
5.17
0.36
Table 5. Ablation on condensation grid size on IVF.
Downstream
Source/Proxy
EN ↑
CC ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF
Full
6.814
0.654
0.652
3.950
0.353
UniVF
Default
6.370
0.625
0.626
5.505
0.379
UniVF
ResNet-18
6.320
0.601
0.626
5.101
0.382
UniVF
VGG-16
6.510
0.660
0.628
5.205
0.375
STAR-VF
Default
6.731
0.641
0.648
3.920
0.356
STAR-VF
ResNet-18
6.720
0.591
0.635
3.586
0.355
Table 6. Proxy robustness and downstream transferability on VTMOT. “Full” denotes UniVF trained on the complete real set; all other rows use a 1/16 condensed source.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Illustration of the capacity–compactness trade-off controlled by the grid size ρ . Increasing ρ enlarges both the latent canvas size and the decoded synthetic clip length Tcond=ρ2 , thereby increasing the effective supervision budget available to Stage II. At the same time, a larger ρ weakens the compression effect by making the condensed source more flexible. The default setting 4×4 in STAR-VF is selected as a balanced operating point under this trade-off.
Figure 8. Distribution of random initialization across five metrics, with the condensation result overlaid as a reference point. For comparability across metrics with different scales and optimization directions, all values are transformed into direction-aligned normalized scores. A wider spread of the random runs indicates stronger seed sensitivity and lower stability.
Model
EN ↑
CC ↑
SSIM ↑
BiSWE ↓
MS2R ↓
Random_1
5.27
0.38
0.45
2.65
0.28
Random_2
4.76
0.43
0.46
2.83
0.29
Random_3
6.80
0.67
0.65
4.03
0.35
Random_4
5.32
0.41
0.47
2.94
0.31
Random_5
6.04
0.51
0.52
3.05
0.33
Random Mean
5.64
0.48
0.51
3.10
0.31
Appendix
Table 7. Comparison between random initialization and condensation on VTMOT. Higher is better for EN, CC, and SSIM, while lower is better for BiSWE and MS2R.
Figure 9. Mean-and-variance summary of random initialization across different metrics. Error bars denote the standard deviation over random runs, while the star marks the condensation result. The comparison shows that random initialization exhibits larger performance variability, whereas condensation provides a more stable and consistently strong outcome.
Method
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
GIFNet
0.67
6.41
0.56
5.52
0.38
CDDFuse
0.66
6.77
0.65
3.98
0.35
TC-MoA
0.64
6.74
0.65
4.69
0.38
SwinFusion
0.65
6.81
0.66
3.93
0.37
UniVF ( Zhao et al., 2025 )
0.65
6.81
0.65
3.95
0.35
VideoFusion ( Tang et al., 2026 )
0.62
6.59
0.62
3.14
0.59
Appendix
Table 8. Quantitative comparison on IVF (VTMOT ( Zhu et al., 2025 ) ). Red and Blue indicate the best and second-best results. Notably, STAR-VF is trained using only 1/16 of the original training data, yet still achieves competitive performance on VTMOT.
Video fusion aims to integrate complementary information from multiple source videos while preserving temporal consistency. Effective modeling of temporal dynamics is essential to this goal, yet existing methods incur substantial computational overhead from optical flow estimation and feature warping. In this paper, we present MambaVF, an efficient video fusion framework that uses state space model (SSM) to achieve temporal modeling without explicit motion estimation. First, by formulating video fusion as a sequential state update process, MambaVF captures long-range temporal dependencies with linear complexity, significantly reducing computation and memory costs. Second, the lightweight SSM-based fusion module eliminates conventional flow-guided alignment. Instead, it introduces a mutual state fusion module and a spatio-temporal bidirectional scanning mechanism to enable information aggregation across video streams. Experiments on multiple benchmarks confirm that MambaVF reaches state-of-the-art performance in different video fusion applications (multi-exposure, multi-focus, infrared-visible, medical), while reducing parameters by >90% and FLOPs by >80%, resulting in >50% shorter runtime. Project page: https://mambavf.github.io
Zixiang Zhao, Yukun Cui, Lilun Deng +4
ETH Zürich · Xi’an Jiaotong University · Nanyang Technological University +1
Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.
Meiqi Gong, Hao Zhang, Jiayi Ma
Electronic Information School, Wuhan University, Wuhan, China · School of Robotics, Wuhan University, Wuhan, China
Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16 FPS at 640×480 resolution. The source code will be available at https://github.com/ixilai/MAVFusion.
Xilai Li, Weijun Jiang, Xiaosong Li +5
Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China