ViLL-E: Video LLM Embeddings for Retrieval

Rohit Gupta; Jayakrishnan Unnikrishnan; Fan Fei; Sheng Liu; Son Tran; Mubarak Shah

cs.CVApr 13, 2026

ViLL-E: Video LLM Embeddings for Retrieval

Authors: Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei, Sheng Liu, Son Tran, Mubarak Shah

Abstract

Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-toVideo Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).

Figures & tables

Panel of Figure 1 — Figure 1: Left & Bottom Right : VideoLLMs lag expert models on some retrieval - based tasks, e.g. Temporal Localization (in green [ 17 , 10 , 15 ] ), and are incapable of others, e.g. Text - to - Video Retrieval (in red [ 49 , 1 , 44 ] ). A key difference between existing VideoLLMs and state of the art expert models in these tasks is the use of embeddings( Top Right ). Our approach, ViLL - E (VideoLLM - Embed, pronounced willy ) equips VideoLLMs with embedding generation. ViLL-E matches SotA on retrieval tasks while remaining competitive on generation tasks such as VideoQA (in orange [ 47 , 22 ] ). ◆ indicates limited capabilities.

Figure 2: Our model ViLL-E is a multi-modal LLM which has been equipped with additional embedding generation capabilities. Individual video frames are first encoded by a pretrained vision encoder, with extracted features projected into embedding vectors. These visual embeddings are combined with textual embeddings of an input prompt and jointly processed autoregressively through a pre-trained Multi-Modal LLM’s transformer layers. After the model autoregressively generates an end-of-sequence ( |EOS| ) token, all the generated tokens are collected and passed to our embedding head. The embedding head employs an attention-based learnable pooling mechanism to extract and aggregate relevant information across the tokens; The generated tokens are further passed through projection layers and mean pooled to yield a compact, informative video embedding. The ability to dynamically generate different number of tokens to feed the embedding head allows the model to handle videos of different complexity levels and is key for its success at varied retrieval tasks.

Figure 3: Three stages of training. Stage 1 ( § 3.4 ) , the large scale joint contrastive and generative pre-training, which utilizes (a) Video-Text retrieval task and (b) Video captioning on the large scale Shutterstock video dataset. Stage 2 ( § 3.5 ) , continues a similar joint pre-training approach (retrieval + captioning tasks as shown in (a) and (b)) on a smaller high quality dataset created using Claude-3-Sonnet . Stage 3 ( § 3.6 ) , our multi-task fine-tuning stage, in addition to the prior tasks (retrieval + captioning) integrates (b) Video QA, (c) Video-Text matching and (d) Temporal Localization to improve all-around retrieval capabilities.

Figure 4: Comparing original caption against Claude-3-Sonnet generated high-quality caption.

	Text to Moment Retrieval			Text to Video Retrieval
Method	ActivityNet	Charades	QVHighlights	MSR-VTT	DiDeMo	VATEX
	R@1,IoU=0.5	R@1,IoU=0.5	mAP	R@1	R@1	R@1
Expert Models
M-DETR NeurIPS 21	-	53.6 FT	35.7 FT	✗	✗	✗
QD-DETR CVPR 23	33.2 FT	57.3 FT	38.9 FT	✗	✗	✗
TR-DETR AAAI 24	-	57.6 FT	39.9 FT	✗	✗	✗

Table 1: Results on retrieval tasks . In moment retrieval, we outperform specialized Video LLMs by over 8%, and approach the performance of SotA dataset-specific expert models. In video retrieval we beat finetuned SotA models. FT models fine-tuned on train splits. ✗ - model is incapable of task

Dataset	#	Cap	QA	Loc
MSR-VTT	10K	✓	✓	✗
ActivityNet	30K	✓	✓	✓
DiDeMo	10K	✓	✗	✓
Shutterstock	50K	✓	✗	✗

Table 2: Multi-Task Dataset.

Method	MSR	VCG	MSVD	MVBench	VidMME
General Video LLMs
Qwen-2.5VL	-	-	-	67.0	61.5
LLaVA-Video	-	3.5	-	58.6	63.3
Temporally-Specialized Video LLMs
ST-LLM	63.2	3.15	74.6	54.9	37.9
LLaVA-ST	59.0	3.3	75.9	64.2

Table 3: ViLL-E is competitive with SotA VideoLLMs on QA tasks in both simple VideoQA (MSR - VTT QA, VideoChatGPT Benchmark and MSVD - QA) and Comprehensive Benchmarks (MVBench and VideoMME).

MSR-VTT	T $\rightarrow$ V		V $\rightarrow$ T
Method	R@1	R@5	R@1	R@5
VidLA	58.0	81.1	56.1	80.5
Ours (1 Stage)	62.5	78.1	55.3	74.8
Ours (2 Stage)	62.8	80.1	57.3	83.5

Table 4: Two-stage retrieval + matching.

Method	R@1	R@5	R@10
COVR-BLIP AAAI 24	45.46	70.46	79.54
Thawakar et al. CVPR 24	47.52	72.18	82.37
Ours	53.13	74.80	85.95

Table 5: Zero-Shot Composed Video Retrieval.

Method	Short Caption				Long Caption
	Avg. = 27 words				Avg. = 500 words
	T $\rightarrow$ V		V $\rightarrow$ T		T $\rightarrow$ V		V $\rightarrow$ T
	R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
SigLIP ICCV23	62.5	83.2	65.4	85.1	51.7	72.7	60.4	81.0
VidLA CVPR24	61.2	85.7	65.9	85.0	45.3	65.2	55.9	72.7
LongCLIP ECCV24	63.0	85.4	61.5	83.6	73.5	91.7	74.5	92.8

Table 6: Zero-Shot Video Detailed Caption Retrieval.

Table 7: Ablation Experiments.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

#	Pre.	VCG	MSR	VTX	QVH
Ours (256)	×	3.1	49.3	45.3	32.6
32	×	2.7	34.8	35.5	25.7
64	×	3.0	42.5	39.2	29.5
128	×	3.1	48.6	44.9	32.0
512	×	3.1	49.5	45.0	32.4

Appendix

Table 8: Number of Pooling Tokens

#	Pre.	MSR	VCG	MSR	DDM	VTX	ANet	QVH	Ch.
Ours	×	55.9	3.1	49.3	45.5	45.3	32.3	32.6	49.3
1	×	50.2	2.6	45.8	37.6	38.9	23.7	24.1	36.8
5	×	54.1	2.8	47.9	40.3	41.2	27.3	27.8	42.3

Appendix

Table 9: Number of Embedding Tokens

Tokens	Latency (ms)
Fixed – 1	238
Fixed – 5	257
Fixed – 10	326
Adaptive (Ours)	262

Appendix

Table 10: Inference latency vs. number of embedding tokens for 12-frames (batch size 16) on NVIDIA L40S.

Method	0–60s	60–120s	120–180s	180s+	Overall
VidLA Baseline	65.7	66.5	65.0	63.5	65.2
Fixed Token = 1	66.2	64.7	63.5	62.2	64.1
Fixed Tokens = 5	66.6	67.4	66.4	65.4	66.5
Adaptive (Ours)	67.9	68.5	67.8	67.1	67.9

Appendix

Table 11: ActivityNet Captions Text-to-Video Retrieval (R@1) broken down by video duration.

Hyperparameter	Stage 1/2	Stage 3
Number of GPUs	32	8
Batch Size (per GPU)	12	8
Gradient Accumulation	2	4
Optimizer	Lion	AdamW
Base Learning Rate	1e-5	5e-6
Warmup Steps	2000	1000

Appendix

Table 12: Training hyperparameters across different stages. ∗ - only applied to new embedding head.

Window	Stride	IoU	R@1, IoU=0.5
5s	5s	0.3	39.0
5s	5s	0.4	39.4
5s	5s	0.5	35.7
10s	10s	0.4	39.2
10s	5s	0.4	41.7
15s	10s	0.4	39.9

Appendix

Table 13: Ablation of localization inference protocol choices. Score on ActivityNet Dataset.

Figure 5: Two Step Retrieval Inference.

Figure 6: Composed Video Retrieval Inference.

Figure 7: We ensure balance of concepts in our dataset by sub-sampling videos from original dataset by undersampling common concepts. The y-axis is log-scale.

Figure 8: Our captions significantly increases the level of detail. Blue - original captions, Red - Generated.

Figure 9: Stage 1 Training . Joint Generative and Contrastive Pre-Training. This stage requires 2 forward passes of the model per step. In the first step, the videos along with prompt are passed to the model and the video caption and video embedding are generated simultaneously. In the second forward pass, only the captions are passed to generate caption embeddings. The ground-truth caption for the video serves as the positive match, whereas the captions for the other videos in the batch are negatives for the contrastive loss.

Figure 10: Stage 3: Temporal Localization Task Training . Contextualized Text Embeddings ( Left Half ): Text embedding are generated based on text input such as “Playing with glow slime in the dark.” A selection of sparse frames from the long video is sampled and added to the prompt to provide context. Sliding Window Video Clip Embeddings ( Right Half ): The long video is divided into clips via a sliding window mechanism, which extracts several short clips. Video embeddings for the clips are used to create the positive and hard negative matches for the text, based on the annotations from the dataset. Training is done using the contrastive loss.

Figure 11: Qualitative Video Captioning results on MSR-VTT

Figure 12: Illustrating our model’s retrieval embeddings and their alignment in various sub-domains of videos

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 167 papers

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning
Oct 4, 2026Haozhe Chi, Song Jin, Yang Jin +1Multimodal Embedding Vision-Language Retrieval
Peking University · Renmin University of China
QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding
Jul 6, 2026Wei Ao, Lan Wang, Vishnu Naresh BoddetiTemporal Video Grounding Streaming Video Understanding
Michigan State University, East Lansing MI 48824, USA
Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
Jan 14, 2025Yifang Xu, Yunzhuo Sun, Benxiang Zhai +4Video Moment Retrieval Temporal Video Grounding
Nanjing University · Dalian University of Technology · Nanjing University of Information Science and Technology

Oct 4, 2026J/K move · Enter open · S save

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Haozhe Chi, Song Jin, Yang Jin, Yadong Mu

Peking University · Renmin University of China

Multimodal Embedding Vision-Language Retrieval Video Retrieval Streaming Video Understanding Query-Aware Video Frame Selection

Open paper AskPDF arXiv

Long-video embedding requires capturing sparse query-relevant evidence under a limited visual-token budget. Uniform sampling can miss brief events in videos spanning minutes or hours, whereas encoding more frames in a single context increases memory and computation. We introduce Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed. QASLR selects a bounded set of frames, restores their temporal order, and processes them clip by clip with a vision-language backbone. A compact set of persistent think tokens cross-attends to each clip's features, while an embed token reads out a normalized representation after every update. This design integrates evidence across multiple backbone calls without requiring all selected frames to share a single context. Training combines final contrastive learning, step-wise contrastive supervision, and final-embedding self-distillation. Intermediate supervision trains partial-video readouts for retrieval, while self-distillation regularizes them toward the final representation. Query-aware selection produces query-conditioned representations for candidate-set scoring and reranking, whereas query-independent selection enables reusable corpus indexing. Under the full training recipe, HourVideo retrieval Hit@1 increases from 54.2 to 70.7 and from 57.4 to 72.8 for 2B and 8B Qwen3-VL-Embedding backbones, respectively. Gains extend to the evaluated moment-retrieval and video-QA tasks, and the streaming head transfers to a second Qwen-family embedding backbone. These results support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Abstract

Video-Language Models

Explore similar work

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models