Large-scale video-language pretraining enables strong generalization across multimodal tasks but often incurs prohibitive computational costs. Although recent advances in masked visual modeling help mitigate this issue, they still suffer from two fundamental limitations: severe visual information loss under high masking ratios and temporal information leakage caused by inter-frame correlations. To address these challenges, we propose ClusterSTM, a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining. ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters, then conducts cluster-wise masking by retaining the token with the highest temporal density within each cluster. Our masking strategy ensure that the retained tokens capture holistic video content while exhibit strong temporal correlation. Additionally, we introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction. Extensive experiments across multiple benchmarks demonstrate that ClusterSTM achieves superior performance on video-text retrieval, video question answering, and video captioning tasks, establishing a new state-of-the-art among efficient video-language models.
Figures & tables
Figure 1: Comparison of different masking strategies. By selecting the token with the highest temporal density within each cluster, the cluster-wise spatio-temporal masking ensures strong temporal correlation among the retained tokens, thereby effectively mitigating the issue of temporal information leakage.
Figure 2: The token with the highest temporal density in a frame typically maintains the highest temporal density in subsequent frames, regardless of its spatial displacement, thereby ensuring that the retained tokens exhibit strong temporal correlation.
Figure 3: The ClusterSTM pipeline consists of two main components: the Cluster-wise Spatio-Temporal Masking strategy and the Video-Text Relevance Generation process. The Cluster-wise Spatio-Temporal Masking strategy first performs intra-frame clustering, followed by Temporal-Density-based Cluster-wise Masking. In this way, the retained tokens not only comprehensively capture the holistic content of each frame but also exhibit strong temporal semantic consistency. The Video-Text Relevance Generation process then produces fine-grained video-text relevance matrices, which serve as reconstruction targets for MRM loss computation.
Figure 4: A schematic illustration of the Video-Text Relevance Generation process. The module first aggregates each target token with its neighboring tokens through a pooling operator to obtain an enhanced token. This enhanced token is then multiplied with the text feature to produce a high-quality video-text relevance matrix.
Method
#Pairs
#F
MSRVTT
DiDeMo
ANet
MSVD
Frozen [ 3 ]
5M
4
18.7
20.2
-
-
VIOLETv2 [ 14 ]
5M
5
21.9
24.4
-
-
Singularity [ 22 ]
5M
12
28.4
36.9
30.8
-
UMT [ 27 ]
5M
4
29.6
33.4
28.3
36.2
STM [ 52 ]
5M
4
29.8
34.4
30.6
38.7
ClusterSTM
5M
4
31.2
36.5
31.4
40.3
Table 1: Zero-shot text-to-video retrieval Recall@1 results on MSRVTT, DiDeMo, ActivityNet (ANet) and MSVD. #Pairs denotes the number of pretraining pairs, and #F indicates the number of video frames sampled during evaluation. The best results are indicated in bold . Models pretrained with massive datasets are noted in gray .
Method
#Pairs
MSRVTT
DiDeMo
ActivityNet
R1
R5
R10
R1
R5
R10
R1
R5
R10
ClipBERT [ 23 ]
0.2M
22.0
46.8
59.9
20.4
48.0
60.8
21.3
49.0
63.5
Frozen [ 3 ]
5M
31.0
59.5
70.5
31.0
59.8
72.4
-
-
-
ALPRO [ 24 ]
5M
33.9
60.7
73.2
35.9
67.5
78.8
-
-
-
BridgeFormer [ 15 ]
5M
37.6
64.8
75.1
37.0
62.2
73.9
-
-
-
Singularity [ 22 ]
5M
36.8
65.9
75.5
47.4
75.2
84.0
43.0
70.6
81.3
Table 2: Text-to-video retrieval on MSRVTT, DiDeMo and AcitivityNet. #Pairs denotes the number of pretraining pairs. The best results are indicated in bold . Models pretrained with massive datasets are noted in gray .
Method
#Pairs
TGIF
MSRVTT
ANet-QA
MSVD-QA
Act.
Trans
Frame
MC
QA
ClipBERT [ 23 ]
0.2M
82.8
87.8
60.3
88.2
37.4
-
-
ALPRO [ 24 ]
5M
-
-
-
-
42.1
-
45.9
Singularity [ 22 ]
5M
-
-
-
92.0
42.7
41.8
-
VIOLETv2 [ 14 ]
5M
94.8
99.0
72.8
97.6
44.5
-
-
LAVENDER [ 28 ]
5M
96.6
99.1
72.2
96.6
44.2
-
-
Table 3: Video QA results on TGIF, MSRVTT and ActivityNet. The best results are indicated in bold . Models pretrained with massive datasets are noted in gray .
Method
#Pairs
MSRVTT
MSVD
SwinBERT [ 30 ]
5M
53.8
120.6
VIOLETv2 [ 14 ]
5M
58.0
139.2
LAVENDER [ 28 ]
5M
58.0
139.2
HiTeA [ 59 ]
5M
62.5
145.1
STM [ 52 ]
5M
63.9
145.4
ClusterSTM
5M
64.4
145.6
Table 4: Video captioning results under CIDEr. The best results are indicated in bold . Models pretrained with massive datasets are noted in gray .
Masking Strategy
Retrieval
QA
Captioning
frame-wise
33.9
34.6
50.8
random
37.5
38.5
52.2
tube-wise
38.1
39.7
53.3
cluster-wise-S
39.8
41.4
55.4
cluster-wise-ST
41.3
42.7
56.2
Table 5: Experimental results of different masking strategies on MSRVTT dataset across three downstream tasks. The best results are indicated in bold .
Image(%)
Video(%)
Retrieval
QA
Captioning
50
80
40.1
41.8
55.6
50
90
41.1
42.4
56.1
75
80
40.9
42.1
56.0
75
90
41.3
42.7
56.2
90
95
39.5
41.2
55.3
Table 6: Experimental results of different masking ratios for image and video tokens on MSRVTT dataset across three downstream tasks. The best results are indicated in bold .
Reconstrction Target
Retrieval
QA
Captioning
pixels
36.3
37.4
52.1
HOG
38.5
39.5
53.7
features
40.2
41.5
55.3
video-text relevance
41.3
42.7
56.2
Table 7: Experimental results of different reconstruction targets on MSRVTT dataset across three downstream tasks. The best results are indicated in bold .
VTC
VTM
MLM
MRM
Retrieval
QA
Captioning
✓
-
-
-
36.8
37.8
51.2
✓
✓
-
-
38.2
38.9
52.7
✓
✓
✓
-
38.9
39.6
54.0
✓
✓
✓
✓
41.3
42.7
56.2
Table 8: Effect of different pretraining objectives on MSRVTT dataset across three downstream tasks. The best results are indicated in bold .
Method
MSRVTT
DiDeMo
ANet
MSVD
UMT [ 27 ]
29.6
33.4
28.3
36.2
+ ClusterSTM
30.4
34.7
30.2
38.5
Table 9: Effectiveness of cluster-wise spatio-temporal masking strategy. The best results are indicated in bold .