STAR-VF: Spatio-Temporal Aware Representation Condensation for Data-Efficient Video Fusion
Authors: Yuchen Xian, Yunqiu Xu, Guosheng Lin, Yang He
Organizations: Zhejiang University, Hangzhou, Zhejiang, China · Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore
Video fusion aims to integrate complementary cues from heterogeneous sources, producing visually informative videos with coherent spatio-temporal structures. Jointly learning cross-modal complementarity and temporal dependencies typically relies on large-scale, high-quality video datasets, whose storage and repeated use during optimization incur substantial costs, making data-efficient learning highly desirable. To address this challenge, we rethink video fusion from the perspective of dataset condensation and propose STAR-VF, the first framework that condenses spatio-temporal representations for data-efficient video fusion. STAR-VF condenses the original training set into a compact set of learnable spatio-temporal representations, which are decoded into synthetic video clips and optimized using feature-level supervision provided by a frozen proxy model. The resulting condensed data preserves fusion-relevant spatial and temporal cues without requiring ground-truth fused videos as supervision. To adapt fusion learning to condensed representations, we equip the final fusion model with flow-free inter-frame temporal modeling and query-driven intra-frame aggregation, enabling direct temporal interaction and adaptive selection of fusion-relevant information in representation space. Extensive experiments on infrared-visible, multi-exposure, multi-focus, and medical video fusion benchmarks demonstrate that STAR-VF, trained with only approximately 1/16 of the original temporal training set, achieves fusion quality comparable to methods trained on the full data. Further analyses validate the contributions of learned condensation and flow-free temporal modeling, and demonstrate the robustness of the condensed supervision across different proxy encoders and downstream fusion models. Project page: https://zju-xyc.github.io/STAR-VF-Project-Page/.
Figures & tables
Figure 1. Comparison of temporal alignment paradigms under video fusion dataset condensation. Conventional flow-based alignment is not naturally suited to condensed synthetic clips , whereas our method models order-aware temporal dependency in compact representation space.
Figure 2. Overview of the proposed STAR-VF framework. In Stage I, learnable compact spatio-temporal canvases and the canvas decoder U are jointly optimized under feature-level supervision from a frozen proxy feature extractor, yielding the learned condensed set S∗ and decoder U∗ . In Stage II, the learned condensed set is decoded into synthetic temporal samples to train the final fusion model Fθ with flow-free temporal alignment and query-driven intra-frame aggregation. During inference, only the trained fusion model is used, without the condensed set or proxy model.
Method
Train Ratio
IVF (VTMOT ( Zhu et al., 2025 ) )
MVF (Harvard ( Summers, 2003 ) )
MEF (YouTube-HDR ( Wang et al., 2024 ) )
MFF (DAVIS ( Pont-Tuset et al., 2017 ) )
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF ( Zhao et al., 2025 )
1
0.65
6.81
0.65
3.95
0.35
0.78
4.27
0.75
29.92
1.23
0.96
7.25
0.65
6.44
0.34
0.98
7.41
0.90
6.04
1.11
VideoFusion ( Tang et al., 2026 )
1
0.62
6.59
0.62
3.14
0.59
0.78
4.91
0.69
19.71
1.31
0.91
6.82
0.58
9.98
1.82
0.96
7.34
0.82
9.07
6.13
TemCoCo ( Gong et al., 2025 )
1
0.63
6.68
0.60
6.26
0.59
0.77
5.40
0.24
28.41
1.57
0.90
7.34
0.53
11.62
1.70
0.97
7.45
0.84
12.34
4.08
MambaVF ( Zhao et al., 2026b )
1
-
-
0.65
3.91
0.36
-
-
0.76
29.61
1.27
-
-
-
6.96
0.16
-
-
-
8.29
-
STAR-VF (ours)
∼ 1/16
0.64
6.73
0.65
3.91
0.35
0.80
4.37
0.76
28.36
1.26
0.91
7.28
0.55
10.23
1.28
0.96
7.42
0.91
5.53
1.18
Table 1. Quantitative comparison on IVF (VTMOT), MVF (Harvard), MEF (YouTube-HDR), and MFF (DAVIS). The training-data ratio rtrain denotes the fraction of the original temporal training units used to train each method. STAR-VF uses only approximately 1/16 of the original training source, yet achieves fusion quality comparable to the full-data baselines. The best and second-best results are highlighted in red and blue , respectively.
Figure 3. Qualitative comparison on four representative benchmarks. For each example, we show the input pair, our method, the full-size variant trained on the original data, and competing video fusion baselines ( VideoFusion , UniVF , and TemCoCo ). The cyan boxes indicate enlarged local regions for detailed comparison.
Figure 4. Frame-by-frame qualitative comparison on a representative multi-source video fusion sequence.
Dataset
Task
Train seqs
Nreal
Ns
Ncond
rcond
VTMOT
IVF
75
22,500
88
1,408
1/16.0
Harvard
MVF
49
1,308
6
96
1/13.6
YouTube-HDR
MEF
450
66,595
261
4,176
1/15.9
DAVIS
MFF
120
8,195
32
512
1/16.0
Table 2. Training-source budgets used by STAR-VF.
Model
Resolution
Params
GFLOPs
Latency
Memory
UniVF ( Zhao et al., 2025 )
256 × 256
9.16M
934.7
531.4ms
696.8MB
Condensed (ours)
256 × 256
0.32M
184.0
110.5ms
696.8MB
UniVF ( Zhao et al., 2025 )
480 × 854
9.16M
6139.1
820.7ms
4110.5MB
Condensed (ours)
480 × 854
0.32M
1066.6
468.5ms
4110.5MB
Table 3. Computational efficiency comparison.
Figure 5. Visual comparison between the full dataset and the learned condensed dataset.
Figure 6. Alignment consistency comparison across video fusion methods, evaluated by frame-wise aligned ROI luminance trajectories after min-max normalization
Method
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF ( Zhao et al., 2025 )
0.65
6.81
0.65
3.95
0.35
w/o condensation
0.64
6.73
0.64
3.91
0.36
Ours (w/o RAFT)
0.64
6.73
0.65
3.91
0.35
Table 4. Ablation of flow-free temporal modeling on IVF.
Grid
#Frames
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
1 × 1 (full)
1
0.64
6.73
0.64
3.91
0.36
2 × 2
4
0.90
7.25
0.87
5.07
0.36
3 × 3
9
0.89
7.82
0.88
5.10
0.36
4 × 4 (ours)
16
0.64
6.73
0.65
3.91
0.35
5 × 5
25
0.90
6.62
0.88
5.17
0.36
Table 5. Ablation on condensation grid size on IVF.
Downstream
Source/Proxy
EN ↑
CC ↑
SSIM ↑
BiSWE ↓
MS2R ↓
UniVF
Full
6.814
0.654
0.652
3.950
0.353
UniVF
Default
6.370
0.625
0.626
5.505
0.379
UniVF
ResNet-18
6.320
0.601
0.626
5.101
0.382
UniVF
VGG-16
6.510
0.660
0.628
5.205
0.375
STAR-VF
Default
6.731
0.641
0.648
3.920
0.356
STAR-VF
ResNet-18
6.720
0.591
0.635
3.586
0.355
Table 6. Proxy robustness and downstream transferability on VTMOT. “Full” denotes UniVF trained on the complete real set; all other rows use a 1/16 condensed source.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Illustration of the capacity–compactness trade-off controlled by the grid size ρ . Increasing ρ enlarges both the latent canvas size and the decoded synthetic clip length Tcond=ρ2 , thereby increasing the effective supervision budget available to Stage II. At the same time, a larger ρ weakens the compression effect by making the condensed source more flexible. The default setting 4×4 in STAR-VF is selected as a balanced operating point under this trade-off.
Figure 8. Distribution of random initialization across five metrics, with the condensation result overlaid as a reference point. For comparability across metrics with different scales and optimization directions, all values are transformed into direction-aligned normalized scores. A wider spread of the random runs indicates stronger seed sensitivity and lower stability.
Model
EN ↑
CC ↑
SSIM ↑
BiSWE ↓
MS2R ↓
Random_1
5.27
0.38
0.45
2.65
0.28
Random_2
4.76
0.43
0.46
2.83
0.29
Random_3
6.80
0.67
0.65
4.03
0.35
Random_4
5.32
0.41
0.47
2.94
0.31
Random_5
6.04
0.51
0.52
3.05
0.33
Random Mean
5.64
0.48
0.51
3.10
0.31
Appendix
Table 7. Comparison between random initialization and condensation on VTMOT. Higher is better for EN, CC, and SSIM, while lower is better for BiSWE and MS2R.
Figure 9. Mean-and-variance summary of random initialization across different metrics. Error bars denote the standard deviation over random runs, while the star marks the condensation result. The comparison shows that random initialization exhibits larger performance variability, whereas condensation provides a more stable and consistently strong outcome.
Method
CC ↑
EN ↑
SSIM ↑
BiSWE ↓
MS2R ↓
GIFNet
0.67
6.41
0.56
5.52
0.38
CDDFuse
0.66
6.77
0.65
3.98
0.35
TC-MoA
0.64
6.74
0.65
4.69
0.38
SwinFusion
0.65
6.81
0.66
3.93
0.37
UniVF ( Zhao et al., 2025 )
0.65
6.81
0.65
3.95
0.35
VideoFusion ( Tang et al., 2026 )
0.62
6.59
0.62
3.14
0.59
Appendix
Table 8. Quantitative comparison on IVF (VTMOT ( Zhu et al., 2025 ) ). Red and Blue indicate the best and second-best results. Notably, STAR-VF is trained using only 1/16 of the original training data, yet still achieves competitive performance on VTMOT.
Foshan University, Foshan, China · Kunming University of Science and Technology, Kunming, China · China University of Mining and Technology, Beijing, China