Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for text-vision tasks, typically by fine-tuning them to produce universal representations. However, their performance on video-text retrieval often remains below that of dedicated Video Foundation Models. In this paper, we revisit the video-text retrieval capabilities of MLLMs. We first analyze the zero-shot retrieval capabilities of pretrained MLLMs and show that they already encode substantial retrieval-relevant information: combining intermediate-layer embeddings with a calibrated MLLM head yields strong performance without any training. To further exploit these capabilities, we introduce a lightweight text-based optimization strategy that requires no paired multimodal data. Our method uses dense video captions as rich textual surrogates for video semantics and short summaries as compact, query-like views, showing that a carefully designed text-alignment objective can improve multimodal alignment through the shared MLLM backbone. We backup these findings through in-depth analysis. Remarkably, our lightweight text-only optimization outperforms its counterpart trained with paired video-text supervision, which is substantially more computationally expensive. Furthermore, our method outperforms existing approaches, often by a substantial margin, and achieves state-of-the-art performance across common video-text retrieval benchmarks. More broadly, our findings highlight the untapped retrieval capabilities of pretrained MLLMs and establish text-only optimization as a more efficient and effective substitute for large-scale visual supervision.
Figures & tables
Figure 1: An overview of VidVec. (a) Zero-shot retrieval: extract video and text embeddings from an intermediate MLLM layer for initial ranking. (b) Zero-shot reranking: leverage the calibrated MLLM head for pairwise scoring to rerank top- K candidates. (c) In-context optimization: lightweight model alignment using only ∼ 60K text-only pairs for embedding extraction via a text-to-text mapping from dense video captions to short summaries, designed to mirror the video-text inference setup - dense caption serves as a surrogate for the full video content ( yellow ) and a short summary acts as a compact anchor closer to retrieval queries ( blue ).
Table 2: Text-to-Video (T2V) retrieval - VidVec-O ( Optimized Embedding ) vs. state-of-the-art MLLM embedders MLLM embedders (7B size) on four benchmarks. VidVec-O uses only 60K text-only in-context pairs, whereas prior methods are trained on × 10- × 100 larger scale of vision-text data.
Method
V-T Pairs
MSR-VTT
MSVD
VATEX
DiDeMo
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
Clip4Clip [ 39 ]
n/a
32.0
-
38.5
-
-
-
-
-
ViCLIP [ 62 ]
n/a
42.4
41.3
49.1
75.1
-
-
18.4
27.9
InternVideo-L [ 58 ]
12M
40.7
39.6
43.4
67.6
49.5
69.5
31.5
33.5
UMT-L [ 37 ]
10M
42.6
38.6
49.9
75.4
-
-
48.6
49.9
LanguageBind [ 73 ]
10M
44.8
40.9
53.9
72.0
-
-
39.9
-
Table 4: Video-Text retrieval performance (Recall@1): VidVec vs. state-of-the-art Video Foundation Models (VFMs). Dashed entries indicate not reported results in the corresponding paper.
Supervision
Training
# Train
Training
MSR-VTT
DiDeMo
Regime
Examples
Time
R@1
R@5
R@10
R@1
R@5
R@10
Video–text
Compute-matched
∼ 3.5K
<0.5 h
43.3
66.6
75.5
35.8
60.9
71.8
Video–text
Full data
∼ 60K
∼ 10 h
51.8
73.6
81.9
51.4
74.8
81.4
Text only
ours
∼ 60K
<0.5 h
52.5
76.3
83.8
53.7
79.4
85.0
Table 5: Text-only optimization vs. direct video–text fine-tuning.
Figure 6Figure 7
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
#Videos
#Captions
MSRVTT
1,000
1,000
MSVD
670
27,763
VATEX
1,500
15,000
DiDeMo
1,004
1,004
ActivityNet
4,917
4,917
Appendix
Table 6: Benchmark datasets statistics, with standard test subsets.
MSR-VTT
MSVD
VATEX
DiDeMo
Backbone
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Qwen3-VL
50.1
73.2
82.6
52.6
79.2
86.3
60.4
89.6
94.8
46.9
72.0
79.1
Keye-VL-1.5
51.7
75.7
84.4
59.8
84.3
89.8
68.0
91.3
95.6
51.2
77.5
83.2
VideoLLaMA3
52.5
76.3
83.8
60.8
84.9
90.1
68.2
93.6
96.8
53.7
79.4
85.0
Appendix
Table 7: Text-to-video retrieval performance across different video MLLM backbones. All models are evaluated after applying our text-only optimization procedure. Results are reported as Recall@K.
MSR-VTT
MSVD
VATEX
DiDeMo
Backbone
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Qwen3-VL
51.5
75.8
83.0
77.5
89.6
93.1
84.3
96.8
98.3
48.0
71.4
79.1
Keye-VL-1.5
54.9
77.2
85.2
84.9
95.5
98.1
88.9
97.1
98.9
51.9
75.6
81.8
VideoLLaMA3
54.9
77.5
84.1
85.7
95.5
97.8
89.6
98.5
99.3
56.5
79.7
86.0
Appendix
Table 8: Video-to-text retrieval performance across different video MLLM backbones. All models are evaluated after applying our text-only optimization procedure. Results are reported as Recall@K.
MSR-VTT
DiDeMo
Model
R@1
R@5
R@10
R@1
R@5
R@10
LamRA
25.0
47.4
57.6
22.8
44.7
56.9
VLM2Vec
34.5
58.5
67.9
29.3
53.5
62.2
VLM2Vec-v2
28.3
50.4
59.4
30.4
53.9
63.1
UniME-v2
27.6
44.9
52.9
31.5
53.0
61.9
VidVec-O
48.9
71.4
81.1
43.0
69.0
76.3
Appendix
Table 9: Comparison to MMEB-v2 results on Text-to-Video retrieval (Recall@K).
Figure 9: Layer-wise t-SNE visualizations. In the early to intermediate layers, the embeddings of the two modalities occupy relatively distinct regions, indicating the presence of a modality gap. As the layers progress, however, the model eventually maps both modalities into a more closely aligned semantic space.
Phase
Duration
Time per Sample
Video encoding
1 min 6 sec
∼ 0.134 sec
Text encoding
10 sec
∼ 0.021 sec
Appendix
Table 13: Encoding runtime for layer selection on the MSR-VTT validation set using the selected backbone. Hidden states from all intermediate layers are extracted in a single forward pass.
Setting
R@1
R@5
R@10
First stage
52.5
76.3
83.8
K=5
56.4
76.3
83.8
K=10
56.2
77.4
83.8
K=20
55.3
76.9
85.0
Appendix
Table 14: Effect of reranking depth K on MSR-VTT text-to-video retrieval.
Method
R@1
First stage
52.5
RankVideo
51.7
Qwen3-VL-Reranker
57.1
Ours
56.4
Appendix
Table 15: Comparison of reranking methods on MSR-VTT text-to-video retrieval. All methods rerank the same top- 5 candidates produced by the first-stage retriever.
Method
Model Name
LamRA
LamRA-Ret
VLM2Vec
VLM2Vec-Qwen2VL-7B
MMRet-v1.5
BGE-VL-v1.5-mmeb
B3
B3_Qwen2_7B
VLM2Vec-V2
VLM2Vec-V2.0
UNITE
Unite-Base-Qwen2-VL-7B
Appendix
Table 16: MLLM embedder baselines used in our evaluation.
Figure 10: Pairwise similarity distributions among video embeddings, dense captions, and short summaries in the zero-shot setting and after in-context optimization. The high video–dense-caption similarity in the zero-shot setting indicates that dense captions can serve as a proxy for the visual video representation, while the post-optimization distribution shifts suggest improved alignment in a shared video–text semantic space. Results are computed on 100 triplets from the VideoUFO dataset.
Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-toVideo Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).
Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei +3
Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/
Fida M. Thoker, Renaud Vandeghen, Karen Sanchez +2
King Abdullah University of Science and Technology (KAUST) · University of Li`ege
Recent advancements in chain-of-thought (CoT) reasoning have shown promise in enhancing video understanding and reasoning capabilities of multimodal large language models (MLLMs). However, existing CoT-based MLLMs require labor-intensive CoT annotations and incur substantial training and inference overhead. While visual latent reasoning has emerged as a more efficient alternative, existing methods primarily focus on image tasks and heavily rely on additional supervision signals for visual latent generation (e.g., CoT traces, auxiliary images, or fine-grained annotations), limiting their scalability and transferability to video tasks. To bridge this gap, we introduce VideoLatent, a novel MLLM equipped with a latent injection module tailored for video understanding and reasoning. Specifically, VideoLatent learns to perform visual latent reasoning using a new latent self-forcing training paradigm, which comprises latent alignment and latent diversity objectives, and relies solely on standard video-question-answer triplets. Extensive experiments across 14 benchmarks demonstrate that our model consistently outperforms existing standard and latent MLLMs on general video understanding and complex video reasoning. Compared with Video-R1, our VideoLatent achieves superior computational efficiency, reducing training/inference overhead by ∼6×/∼68×. Moreover, experiments demonstrate that our method has strong generalizability to different MLLM backbones and different model scales.