Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for text-vision tasks, typically by fine-tuning them to produce universal representations. However, their performance on video-text retrieval often remains below that of dedicated Video Foundation Models. In this paper, we revisit the video-text retrieval capabilities of MLLMs. We first analyze the zero-shot retrieval capabilities of pretrained MLLMs and show that they already encode substantial retrieval-relevant information: combining intermediate-layer embeddings with a calibrated MLLM head yields strong performance without any training. To further exploit these capabilities, we introduce a lightweight text-based optimization strategy that requires no paired multimodal data. Our method uses dense video captions as rich textual surrogates for video semantics and short summaries as compact, query-like views, showing that a carefully designed text-alignment objective can improve multimodal alignment through the shared MLLM backbone. We backup these findings through in-depth analysis. Remarkably, our lightweight text-only optimization outperforms its counterpart trained with paired video-text supervision, which is substantially more computationally expensive. Furthermore, our method outperforms existing approaches, often by a substantial margin, and achieves state-of-the-art performance across common video-text retrieval benchmarks. More broadly, our findings highlight the untapped retrieval capabilities of pretrained MLLMs and establish text-only optimization as a more efficient and effective substitute for large-scale visual supervision.
Figures & tables
Figure 1: An overview of VidVec. (a) Zero-shot retrieval: extract video and text embeddings from an intermediate MLLM layer for initial ranking. (b) Zero-shot reranking: leverage the calibrated MLLM head for pairwise scoring to rerank top- K candidates. (c) In-context optimization: lightweight model alignment using only ∼ 60K text-only pairs for embedding extraction via a text-to-text mapping from dense video captions to short summaries, designed to mirror the video-text inference setup - dense caption serves as a surrogate for the full video content ( yellow ) and a short summary acts as a compact anchor closer to retrieval queries ( blue ).
Table 2: Text-to-Video (T2V) retrieval - VidVec-O ( Optimized Embedding ) vs. state-of-the-art MLLM embedders MLLM embedders (7B size) on four benchmarks. VidVec-O uses only 60K text-only in-context pairs, whereas prior methods are trained on × 10- × 100 larger scale of vision-text data.
Method
V-T Pairs
MSR-VTT
MSVD
VATEX
DiDeMo
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
Clip4Clip [ 39 ]
n/a
32.0
-
38.5
-
-
-
-
-
ViCLIP [ 62 ]
n/a
42.4
41.3
49.1
75.1
-
-
18.4
27.9
InternVideo-L [ 58 ]
12M
40.7
39.6
43.4
67.6
49.5
69.5
31.5
33.5
UMT-L [ 37 ]
10M
42.6
38.6
49.9
75.4
-
-
48.6
49.9
LanguageBind [ 73 ]
10M
44.8
40.9
53.9
72.0
-
-
39.9
-
Table 4: Video-Text retrieval performance (Recall@1): VidVec vs. state-of-the-art Video Foundation Models (VFMs). Dashed entries indicate not reported results in the corresponding paper.
Supervision
Training
# Train
Training
MSR-VTT
DiDeMo
Regime
Examples
Time
R@1
R@5
R@10
R@1
R@5
R@10
Video–text
Compute-matched
∼ 3.5K
<0.5 h
43.3
66.6
75.5
35.8
60.9
71.8
Video–text
Full data
∼ 60K
∼ 10 h
51.8
73.6
81.9
51.4
74.8
81.4
Text only
ours
∼ 60K
<0.5 h
52.5
76.3
83.8
53.7
79.4
85.0
Table 5: Text-only optimization vs. direct video–text fine-tuning.
Figure 6Figure 7
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
#Videos
#Captions
MSRVTT
1,000
1,000
MSVD
670
27,763
VATEX
1,500
15,000
DiDeMo
1,004
1,004
ActivityNet
4,917
4,917
Appendix
Table 6: Benchmark datasets statistics, with standard test subsets.
MSR-VTT
MSVD
VATEX
DiDeMo
Backbone
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Qwen3-VL
50.1
73.2
82.6
52.6
79.2
86.3
60.4
89.6
94.8
46.9
72.0
79.1
Keye-VL-1.5
51.7
75.7
84.4
59.8
84.3
89.8
68.0
91.3
95.6
51.2
77.5
83.2
VideoLLaMA3
52.5
76.3
83.8
60.8
84.9
90.1
68.2
93.6
96.8
53.7
79.4
85.0
Appendix
Table 7: Text-to-video retrieval performance across different video MLLM backbones. All models are evaluated after applying our text-only optimization procedure. Results are reported as Recall@K.
MSR-VTT
MSVD
VATEX
DiDeMo
Backbone
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Qwen3-VL
51.5
75.8
83.0
77.5
89.6
93.1
84.3
96.8
98.3
48.0
71.4
79.1
Keye-VL-1.5
54.9
77.2
85.2
84.9
95.5
98.1
88.9
97.1
98.9
51.9
75.6
81.8
VideoLLaMA3
54.9
77.5
84.1
85.7
95.5
97.8
89.6
98.5
99.3
56.5
79.7
86.0
Appendix
Table 8: Video-to-text retrieval performance across different video MLLM backbones. All models are evaluated after applying our text-only optimization procedure. Results are reported as Recall@K.
MSR-VTT
DiDeMo
Model
R@1
R@5
R@10
R@1
R@5
R@10
LamRA
25.0
47.4
57.6
22.8
44.7
56.9
VLM2Vec
34.5
58.5
67.9
29.3
53.5
62.2
VLM2Vec-v2
28.3
50.4
59.4
30.4
53.9
63.1
UniME-v2
27.6
44.9
52.9
31.5
53.0
61.9
VidVec-O
48.9
71.4
81.1
43.0
69.0
76.3
Appendix
Table 9: Comparison to MMEB-v2 results on Text-to-Video retrieval (Recall@K).
Figure 9: Layer-wise t-SNE visualizations. In the early to intermediate layers, the embeddings of the two modalities occupy relatively distinct regions, indicating the presence of a modality gap. As the layers progress, however, the model eventually maps both modalities into a more closely aligned semantic space.
Phase
Duration
Time per Sample
Video encoding
1 min 6 sec
∼ 0.134 sec
Text encoding
10 sec
∼ 0.021 sec
Appendix
Table 13: Encoding runtime for layer selection on the MSR-VTT validation set using the selected backbone. Hidden states from all intermediate layers are extracted in a single forward pass.
Setting
R@1
R@5
R@10
First stage
52.5
76.3
83.8
K=5
56.4
76.3
83.8
K=10
56.2
77.4
83.8
K=20
55.3
76.9
85.0
Appendix
Table 14: Effect of reranking depth K on MSR-VTT text-to-video retrieval.
Method
R@1
First stage
52.5
RankVideo
51.7
Qwen3-VL-Reranker
57.1
Ours
56.4
Appendix
Table 15: Comparison of reranking methods on MSR-VTT text-to-video retrieval. All methods rerank the same top- 5 candidates produced by the first-stage retriever.
Method
Model Name
LamRA
LamRA-Ret
VLM2Vec
VLM2Vec-Qwen2VL-7B
MMRet-v1.5
BGE-VL-v1.5-mmeb
B3
B3_Qwen2_7B
VLM2Vec-V2
VLM2Vec-V2.0
UNITE
Unite-Base-Qwen2-VL-7B
Appendix
Table 16: MLLM embedder baselines used in our evaluation.
Figure 10: Pairwise similarity distributions among video embeddings, dense captions, and short summaries in the zero-shot setting and after in-context optimization. The high video–dense-caption similarity in the zero-shot setting indicates that dense captions can serve as a proxy for the visual video representation, while the post-optimization distribution shifts suggest improved alignment in a shared video–text semantic space. Results are computed on 100 triplets from the VideoUFO dataset.