Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis

Maria-Eirini Pegia; Dimitrios Stefanopoulos; Björn Þór Jónsson; Anastasia Moumtzidou; Ilias Gialampoukidis; Stefanos Vrochidis; Ioannis Kompatsiaris

cs.IRMar 7, 2026

Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis

Authors: Maria-Eirini Pegia, Dimitrios Stefanopoulos, Björn Þór Jónsson, Anastasia Moumtzidou, Ilias Gialampoukidis, Stefanos Vrochidis, Ioannis Kompatsiaris

Organizations: Information Technologies Institute, CERTH-ITI, Thessaloniki, Greece. · School of Computer Science, Reykjavík University, Reykjavík, Iceland.

Abstract

Text-to-video retrieval enables users to find relevant video content using natural language queries, a task that has grown increasingly important with the rapid expansion of online video. Over the past six years, research has produced numerous methods, such as dual encoders, attention-driven models, and multimodal fusion approaches; however, fundamental questions remain about model behavior, dataset influence, and query difficulty. In this work, we evaluate 14 state-of-the-art retrieval methods across 3 widely used datasets under a unified preprocessing and evaluation framework. We analyze caption characteristics, including length, clarity, semantic category, and Action vs. Scene balance, and link these to model performance. Our results show that short, clear, and simple captions, such as those describing single actions or color attributes, achieve higher recall, while complex events, multi-step activities, or fine-grained scene descriptions remain challenging for all existing models. Attention-driven architectures better handle temporally dependent or multi-step queries, whereas dual-encoder and multimodal fusion models perform well primarily on simpler or single-category captions. Cross-dataset generalization improves with larger, more diverse caption sets, but generative captions do not consistently enhance retrieval accuracy. Overall, our findings highlight key dataset factors, benchmark challenges, and the interplay between query content and model architecture, providing guidance for developing more effective text-to-video retrieval systems.

Figures & tables

Figure 1: Recall@10 (left $y$ -axis) and model complexity in FLOPs (right $y$ -axis, black line) from 2021 to 2025 for various methods and datasets. Methods are sorted chronologically by year and alphabetically within the same year ( $x$ -axis). Datasets are shown as LSMDC (orange), MSRVTT (blue), MSVD (green). Year blocks are highlighted with soft background colors: 2021 (light gray), 2022 (blue), 2023 (yellow), 2024 (soft green), 2025 (red/pink).

Panel of Figure 2 — Figure 2: Distribution of papers and their relevance: (a) Number of papers published across different conferences and journals, highlighting the contribution per venue; (b) Breakdown of papers by relevance, distinguishing between relevant, non-relevant, and those associated with GitHub repositories.

Panel of Figure 3 — Figure 3: Trends of video retrieval papers per year.

Aspect	Prior Surveys	This Study
	(2020–2025)
Focus	Reviewed architectures, benchmarks, datasets Rossetto et al. (2020) ; Zhu et al. (2023) ; Perez-Martin et al. (2022) ; Liu et al. (2023) ; Lei et al. (2024) ; Lan et al. (2023) ; Madan et al. (2024) ; Xing et al. (2024) ; Tang et al. (2025b) ; Wan et al. (2025) ; Schiappa et al. (2023)	14 methods, 3 datasets evaluated (Section 4.1.1 )
Query Analysis	Rarely examined Rossetto et al. (2020) ; Zhu et al. (2023) ; Perez-Martin et al. (2022) ; Liu et al. (2023)	Difficulty and category analyzed (Section 4.1.3 - 4.1.4 )
Dataset Influence	Broad overview Zhu et al. (2023) ; Perez-Martin et al. (2022) ; Lei et al. (2024) ; Lan et al. (2023)	Composition, size, bias quantified (Section 4.1 )
Captions	Mostly ignored Rossetto et al. (2020) ; Zhu et al. (2023) ; Schiappa et al. (2023)	Clarity, simplicity, length examined (Section 4.1.4 )
Performance	Overall SOTA reported Rossetto et al. (2020) ; Zhu et al. (2023) ; Perez-Martin et al. (2022) ; Liu et al. (2023)	Strengths/weaknesses highlighted (Section 3.3 )

Table 2: Comparison of prior surveys and this study in text-video retrieval (2020–2025).

Method	Input	Temporal	CLIP?	Year	GitHub
	(V / F)	(✓/ ✗)	(✓/ ✗)		Link
SSVR Wray et al. (2021)	V	✗	✗	2021	https://mwray.github.io/SSVR/
CenterCLIP Zhao et al. (2022)	F	✓	✓	2022	https://github.com/mzhaoshuai/CenterCLIP
CLIP4CLIP Luo et al. (2022b)	V	✓	✓	2022	https://github.com/ArrowLuo/CLIP4Clip
EMCL Jin et al. (2022)	F	✓	✓	2022	https://github.com/jpthu17/EMCL
PRVR Dong et al. (2022a)	V	✓	✗	2022	https://github.com/HuiGuanLab/ms-sl

Table 3: State-of-the-art methods from the past 6 years with available GitHub and key features used in our evaluation.

Dataset	Videos	Frames/Vid	Dur (s)	Caps/Vid	Train	Test
LSMDC Rohrbach et al. (2017)	118,081	14	2-30	1	6,209	1,000
MSRVTT Xu et al. (2016)	10,000	46	10-32	20	7,000	1,000
MSVD Chen and Dolan (2011)	1,970	30	1-62	40	1,200	670

Table 4: Benchmark Datasets.

Dataset	Pure queries											Multi	UnRec
	Act	Scn	Cog	Spch	Perc	Mot	StChg	Temp	Col	Scn2	Total
LSMDC Rohrbach et al. (2017)	82	34	6	7	30	32	-	17	11	8	227	665	108
MSRVTT Xu et al. (2016)	29	48	-	12	4	6	1	1	12	3	113	841	46
MSVD Chen and Dolan (2011)	17	61	-	3	-	4	1	-	4	-	89	553	28

Table 5: Categories of all queries per dataset, including pure, multimodal, and unrecognized queries.

Panel of Figure 4 — Figure 4: Recall@10 across datasets: (Top) literature results, (Middle) our results, and (Bottom) average recall difference comparison (Ours vs. Literature).

Panel of Figure 5 — Figure 5: Query difficulty (average GT index) for 14 methods on LSMDC, MSRVTT and MSVD.

Method	LSMDC Rohrbach et al. (2017)			MSRVTT Xu et al. (2016)			MSVD Chen and Dolan (2011)
	Easy	Med	Hard	Easy	Med	Hard	Easy	Med	Hard
SSVR Wray et al. (2021)	32.4	44.0	23.6	40.7	20.7	38.6	15.1	30.1	54.8
CenterCLIP Zhao et al. (2022)	40.8	21.3	27.9	36.1	21.1	42.8	18.1	45.1	36.9
CLIP4CLIP Luo et al. (2022b)	32.4	32.1	55.5	30.6	30.8	38.6	17.0	33.0	50.0
EMCL Jin et al. (2022)	50.2	31.0	18.8	54.6	20.3	25.1	30.1	54.8	15.1
PRVR Dong et al. (2022a)	32.4	34.0	33.6	34.5	30.3	55.2	14.9	25.4	59.7

Table 6: Average Recall@k (%) for k = {1, 10, 50} per difficulty level for 14 methods across LSMDC, MSRVTT, and MSVD datasets.

Figure 6: Effect of sentence length on retrieval performance across LSMDC (orange), MSRVTT (blue) and MSVD (green).

Panel of Figure 7 — Figure 7: Recall@k (k = 1, 10, 50) across task categories on LSMDC dataset.

Panel of Figure 8 — Figure 8: Recall@k (k = 1, 10, 50) across task categories on MSRVTT dataset.

Panel of Figure 9 — Figure 9: Recall@k (k = 1, 10, 50) across task categories on MSVD dataset.

Panel of Figure 10 — Figure 10: Recall@k (k=1, 10, 50) across difficulty levels on LSMDC (1st row), MSRVTT (2nd row) and MSVD (3rd row).

Category	FRE $\uparrow$	FKG $\downarrow$	Words $\uparrow$	Unique $\uparrow$	Avg Len $\uparrow$	Profanity	PPL $\downarrow$
LSMDC
Easy	78.36	4.90	10.36	9.68	4.73	0	411.16
Medium	81.46	3.84	7.82	7.50	4.83	1	449.23
Hard	78.54	4.77	9.91	9.44	4.67	1	298.48
MSRVTT
Easy	72.54	5.38	9.00	8.43	4.17	0	551.09

Table 8: Linguistic Metrics per dataset captions

Category	LSMDC				MSRVTT				MSVD
	n	Easy	Med	Hard	n	Easy	Med	Hard	n	Easy	Med	Hard
Act	53	45	35	20	29	50	30	20	17	40	40	20
Mot	32	40	35	25	6	45	30	25	4	38	37	25
Col	8	50	20	30	12	45	25	30	4	60	20	20
Scn	32	20	50	30	30	25	45	30	61	20	50	30
Scn2	0	-	-	-	3	25	45	30	0	-	-	-

Table 9: Distribution of word categories across difficulty levels in LSMDC, MSRVTT, and MSVD datasets (

N

= 200 words per dataset). Values show total count

n

per category (Freq =

n

N

) and percentage of words classified as Easy, Medium (Med), or Hard (%). Frequencies are based on pure queries per dataset.

Dataset	Classification of pure queries
	Act	Scn	Cog	Spch	Perc	Mot	StChg	Temp	Col	Scn2
LSMDC Rohrbach et al. (2017)	46.15	11.54	0.96	1.92	17.31	9.62	-	6.73	0.96	4.81
MSRVTT Xu et al. (2016)	25.23	36.04	-	11.71	4.50	4.50	3.60	1.80	10.81	1.81
MSVD Chen and Dolan (2011)	15.20	56.80	-	4.80	4.80	5.60	4.80	-	8.00	-

Table 10: Chosen pure queries for statistical evaluation (in

\%

Panel of Figure 11 — Figure 11: k-Means Clustering: Per Dataset (1-3) & All (4).

Figure 12: Training times (in seconds) for all methods, ordered by their average Recall@1 across all datasets.

Dataset / Method	Recall			Recall
	@1	@10	@50	@1	@10	@50
LSMDC	Original			Multiple Captions
EMCL Jin et al. (2022)	16.40	31.40	39.90	18.80	34.80	43.30
TempMe Shen et al. (2024)	17.78	35.50	44.70	20.70	36.30	46.40
X-CLIP Ma et al. (2022)	16.90	33.60	41.30	19.00	34.30	41.90
MSRVTT	Original			Single caption

Table 11: Retrieval performance per dataset and evaluation setting. Recall@K (%) is reported for each method on both the original captions and multiple/single caption scenarios, allowing comparison across datasets LSMDC, MSRVTT, and MSVD.

Panel of Figure 13 — Figure 13: Cross-dataset learning across MSRVTT, MSVD, and LSMDC datasets: (1) Start with the original datasets; (2) Reduce the number of captions per video to one for MSRVTT and MSVD; (3) Increase the number of captions for LSMDC.

Level	Frame Rate (in fps)	Resolution (in ppi)	CRF
Low/No-compression	original	original	18
Medium	10	original	18
High	3	original	18

Table 12: Frame Rate.

Panel of Figure 14 — Figure 14: LSMDC: Recall & Training Time vs Frame Rate.

Level	Frame Rate (in fps)	Resolution (in ppi)	CRF
Low/No-compression	original	original	18
Medium	10	480	28
High	3	224	35

Table 13: Compression Rate.

Panel of Figure 15 — Figure 15: LSMDC: Recall & Training Time vs Compression Rate.

Video Retrieval

Video retrieval matches natural-language, visual, or multimodal queries to relevant videos, clips, and moments.

See all 61 papers

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos
Mar 15, 2026Minghan Li, Tongna Chen, Tianrui Lv +3Video Retrieval Temporal Video Understanding
Soochow University China
R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking
May 31, 2026Zixu Li, Yupeng Hu, Zhiheng Fu +3Cross-Modal Retrieval Composed Video Retrieval
1Shandong University · 2Harbin Institute of Technology (Shenzhen)
ViLL-E: Video LLM Embeddings for Retrieval
Apr 13, 2026Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei +3Video-Language Models Vision-Language Models

Mar 15, 2026J/K move · Enter open · S save

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

Minghan Li, Tongna Chen, Tianrui Lv, Yishuai Zhang+2

Soochow University China

Video Retrieval Temporal Video Understanding

Open paper AskPDF arXiv

Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture state changes and final outcomes. We introduce GenState-AI, a benchmark comprising 956 AI-generated videos and 195 verified state-change queries across Toy, Cartoon, Home, and Manipulation domains. Its construction uses prompt-controlled variants that target either conflicting temporal/state outcomes or semantic substitutions. Following human verification, all valid videos contribute to subset-level retrieval pools, while complete query-specific triplets support diagnostic hard-negative ordering. We evaluate VCF-Lik, Qwen3-VL, and LanguageBind using both standard Recall@K and triplet-based consistency metrics. Qwen3-VL (8B) achieves the best macro-average R@1 and triplet consistency, while LanguageBind obtains the highest semantic discrimination accuracy. However, stronger retrieval does not consistently yield correct hard-negative ordering, and Qwen3-VL and LanguageBind remain weak on Manipulation triplets, where VCF-Lik performs substantially better. GenState-AI therefore provides a controlled diagnostic testbed for distinguishing global retrieval quality from fine-grained temporal and semantic grounding in AI-generated videos.

Abstract

Video Retrieval

Explore similar work

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking

ViLL-E: Video LLM Embeddings for Retrieval