Advancing Video-Text Pretraining with Multi-View Captions

Fida M. Thoker; Renaud Vandeghen; Karen Sanchez; Marc Van Droogenbroeck; Bernard Ghanem

cs.CVSep 28, 2026

Advancing Video-Text Pretraining with Multi-View Captions

Authors: Fida M. Thoker, Renaud Vandeghen, Karen Sanchez, Marc Van Droogenbroeck, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology (KAUST) · University of Li`ege

Abstract

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/

Figures & tables

Figure 1: Captioning pipeline. Overview of our framework for improving supervision quality in video-text pretraining. Given an input video, we construct multiple complementary textual views, refine them through visual reasoning to improve grounding, and generate semantically consistent positives to enrich supervision. Red highlights the potentially incorrect segments in the detailed caption, while green represents the corrected segments after refinement. The resulting supervision corpus is used to learn richer video-text representations.

Method	#Samples	Source	MSR-VTT	DiDeMo	ActivityNet	LSMDC	MSVD
Method	#Samples	Source	R@1	R@1	R@1	R@1	R@1
Frozen ( Bain et al., 2021 )	5M	WebVid-2M+Img3M	18.7	20.2	–	–	–
Singularity ( Lei et al., 2023 )	5M	WebVid-2M+Img3M	28.4	36.9	30.8	–	–
UMT-B ( Li et al., 2023b )	5M	WebVid-2M+Img3M	29.6	33.4	28.3	16.8	36.2
STM-B ( Wu et al., 2025 )	5M	WebVid-2M+Img3M	29.8	34.4	30.6	–	38.7
ClusterSTM-B ( Zhuang et al., 2026 )	5M	WebVid-2M+Img3M	31.2	36.5	31.4	–	40.3

Table 1: Zero-shot text-to-video retrieval. We evaluate our method on MSR-VTT, DiDeMo, ActivityNet, LSMDC, and MSVD using Recall@1. Img3M = CC3M, Img15M = CC12M + SBU + COCO + VG. Within each pretraining-scale group, bold and underlined numbers represent the best and second-best models, respectively.

Method	#Samples	Source	MSR-VTT	DiDeMo	ActivityNet	LSMDC	MSVD
Method	#Samples	Source	R@1	R@1	R@1	R@1	R@1
ClipBERT ( Lei et al., 2021 )	5.4M	COCO+VG	22.0	20.4	21.3	–	–
Frozen ( Bain et al., 2021 )	5M	WebVid-2M+Img3M	31.0	34.6	–	15.0	33.7
UMT-B ( Li et al., 2023b )	5M	WebVid-2M+Img3M	46.3	54.8	52.1	30.3	47.4
STM-B ( Wu et al., 2025 )	5M	WebVid-2M+Img3M	48.5	56.9	53.6	–	–
ClusterSTM-B ( Zhuang et al., 2026 )	5M	WebVid-2M+Img3M	49.7	58.5	54.9	–	–

Table 2: Fine-Tuned Text-to-video retrieval. We evaluate our method on MSR-VTT, DiDeMo, ActivityNet, LSMDC, and MSVD. Img3M = CC3M Img15M = CC12M + SBU + COCO + VG CC = Conceptual Captions, VG = Visual Genome. Within each pretraining-scale group, bold and underlined numbers represent the best and second-best models, respectively.

Method	#Samples	Source	CARE-S	CARE-T	DREAM-D	DREAM-E	S2S-W	S2S-S
			R@1	R@1	R@1	R@1	R@1	R@1
CLIP-B/16 ( Radford et al., 2021 )	400M	CLIP-400M	45.6	30.3	32.6	13.6	10.7	22.7
CLIP-L/14 ( Radford et al., 2021 )	400M	CLIP-400M	49.1	33.5	44.3	14.6	65.8	45.4
ViCLIP-B ( Wang et al., 2024b )	400M+ 10M	CLIP-400M + IV-10M-FLT	56.0	31.3	54.9	10.4	52.2	43.9
ViCLIP-L ( Wang et al., 2024b )	400M + 10M	CLIP-400M + IV-10M-FLT	55.7	34.5	55.5	23.3	55.1	43.6
Long-CLIP-L/14 ( Zhang et al., 2024 )	400M	CLIP-400M + SG4V-1M	65.6	33.3	58.3	24.3	74.7	47.7

Table 3: Fine-Grained and Detailed Zero-Shot Text-to-Video Retrieval. We evaluate on CaReBench Spatial (CARE-S), CaReBench Temporal (CARE-T), DREAM-1K-Detailed (DREAM-D), DREAM-1K-Events (DREAM-E), and Shot2Story whole-clip (S2S-W) and single-shot (S2S-S) splits. Source labels use IV for InternVid and SG4V for ShareGPT4V. For prior methods, we use officially public checkpoints. Bold and underlined numbers mark the best and second-best results.

Supervision	MSVD	ANet	DiDeMo	LSMDC	CARE-T	Shot2Story-S
Original only ( $O$ )	35.4	25.9	29.3	11.8	27.1	42.2
Ours: summary only ( $S$ )	37.0	46.2	46.0	15.1	48.0	63.2
Ours: full ( $S+D^{*}+P$ )	38.5	50.1	52.4	16.7	54.1	67.4
Original + ours ( $O+S+D^{*}+P$ )	38.8	50.1	53.0	18.2	54.1	67.9

Table 4: Comparison of supervision recipes. We compare the original captions (

O

), our summary captions (

S

), our full generated supervision (

S+D^{*}+P

), and its combination with the original captions (

O+S+D^{*}+P

). We report text-to-video R@1 across six downstream benchmarks.

Supervision	MSVD	ANet	DiDeMo	LSMDC	CARE-T	Shot2Story-S
Summary only ( $S$ )	37.0	46.2	46.0	15.1	48.0	63.2
Detailed only ( $D$ )	26.4	42.7	38.4	12.4	41.1	62.3
Summary+detailed ( $S+D$ )	36.6	48.5	48.6	15.9	50.1	65.9
Summary+refined detailed ( $S+D^{*}$ )	37.5	49.6	49.7	16.1	51.8	66.7
Summary+refined+positive ( $S+D^{*}+P$ )	38.5	50.1	52.4	16.7	54.1	67.4

Table 5: Impact of caption complementarity, refinement, and semantic positive captions. We compare summary captions (

S

), detailed captions (

D

), their combination, refined detailed captions (

D^{*}

), and semantic positive captions (

P

). Original captions (

O

) are excluded throughout. We report text-to-video R@1.

Text Representation	MSVD	ANet	DiDeMo	LSMDC	CARE-T	Shot2Story-S
Single CLS	38.8	50.1	53.0	18.2	54.1	67.9
Dual CLS (Ours)	39.4	51.3	54.3	18.7	56.4	68.5

Table 6: Impact of granularity-aware text representations. We compare a standard single- CLS representation with our dual- CLS design that separately models summary and detailed views. The detailed view includes refined detailed and semantic positive captions. Both use the same supervision (

O+S+D^{*}+P

) and training setup. We report text-to-video R@1.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	#Samples	ANet-QA	MSR-VTT-QA	MSVD-QA
ClipBERT ( Lei et al., 2021 )	0.2M	–	37.4	–
ALPRO ( Li et al., 2022a )	5M	–	42.1	45.9
JustAsk ( Yang et al., 2021 )	69M	38.9	41.5	47.5
VideoCLIP ( Xu et al., 2021 )	136M	–	–	–
All-in-one ( Wang et al., 2023 )	138M	–	44.3	47.9
MERLOT ( Zellers et al., 2021 )	180M	41.4	43.1	–

Appendix

Table 7: Video Question-Answering Results. We evaluate transfer to ActivityNet-QA, MSR-VTT-QA, and MSVD-QA. Our models show particularly strong performance on ActivityNet-QA while remaining competitive across the three benchmarks.

Figure 2: Video frames. Example of sub-sampled frames fed to the MLLMs for caption generation.

Caption type	Avg. words	Std. words
Original caption	9.4	2.6
Qwen summary caption	13.8	2.6
Tarsier summary caption	16.2	4.5
Qwen detailed caption	96.0	27.1
Tarsier detailed caption	80.1	25.0
Qwen refined detailed caption	60.6	19.5

Appendix

Table 8: Caption statistics. We report the average and standard deviation of caption length in words for each intermediate and supervision view.

Figure 3: Video frames. Example of sub-sampled frames fed to the MLLMs for caption generation. This example shows the effect of refining the detailed captions to remove incorrect elements.

Masking ratio	MSVD	ANet	DiDeMo	LSMDC	CARE-T	Shot2Story-S
50%	38.8	50.1	53.0	18.2	54.1	67.9
25%	39.5	51.9	53.0	15.8	53.3	68.5
10%	39.7	52.5	54.3	15.7	55.9	68.3

Appendix

Table 9: Effect of Masking Ratio. We compare different video masking ratios while retaining the same caption supervision (

O+S+D^{*}+P

). We report text-to-video R@1; bold indicates the best result in each column.

Epochs	MSVD	ANet	DiDeMo	LSMDC	CARE-T	Shot2Story-S
5	31.6	40.8	42.6	12.9	42.9	59.7
10	36.2	46.5	48.7	15.8	50.3	64.9
15	37.9	48.9	51.5	17.3	52.6	66.7
20	38.8	50.1	53.0	18.2	54.1	67.9

Appendix

Table 10: Effect of training duration. We evaluate different pretraining durations while keeping the model, supervision, and training configuration fixed. We report text-to-video R@1 across six downstream benchmarks.

Task	Dataset	LR	Epochs	DropPath
Retrieval	MSR-VTT	2e-5 (B), 1e-5 (L)	10(B),7(L)	0.2(B),0.3(L)
	DiDeMo	2e-5(B), 2e-5(L)	12(B),5(L)	0.1(B),0.3(L)
	ActivityNet	4e-5(B), 1e-5 (L)	20(B/L)	0.1(B),0.2(L)
	LSMDC	2e-5 (B), 1e-5 (L)	10(B),8(L)	0.1(B),0.2(L)
	MSVD	2e-5 (B), 1e-5 (L)	10(B/L)	0.2(B),0.3(L)
VQA	ActivityNet-QA	4e-5(B),1e-5(L)	12(B),10(L)	0.2(B),0.3(L)

Appendix

Table 11: Fine-Tuning Hyperparameters. Settings used for retrieval and VQA experiments across datasets.

Explore similar work

Building a Precise Video Language with Human-AI Oversight
Apr 22, 2026Zhiqiu Lin, Chancharik Mitra, Siyuan Cen +13Video-Language Models Cinematic Ideal
1Carnegie Mellon University · 2Harvard University
MoVA: Learning Asymmetric Dual Projections for Modular Long Video-Text Alignment
Jul 1, 2026Peiyuan Zhu, Shaoan Xie, Zijian Li +5Image-Text Alignment Entanglement
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Carnegie Mellon University, Pittsburgh, PA, USA
Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning
Aug 11, 2026Liangyu Fu, Junbo Wang, Yuke Li +3Text-To-Video Generation Model Cross-Modal
School of Software, Northwestern Polytechnical University, Xi’an 710129, China · School of Information Science and Technology, Beijing University of Technology, Beijing 100124, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China +1

Apr 22, 2026, cs.CVJ/K move · Enter open · S save

Building a Precise Video Language with Human-AI Oversight

Zhiqiu Lin, Chancharik Mitra, Siyuan Cen, Isaac Li+12

1Carnegie Mellon University · 2Harvard University

Video-Language Models Cinematic Ideal

Open paper AskPDF arXiv

Video-language models (VLMs) learn to reason about the dynamic visual world through natural language. We introduce a suite of open datasets, benchmarks, and recipes for scalable oversight that enable precise video captioning. First, we define a structured specification for describing subjects, scenes, motion, spatial, and camera dynamics, grounded by hundreds of carefully defined visual primitives developed with professional video creators such as filmmakers. Next, to curate high-quality captions, we introduce CHAI (Critique-based Human-AI Oversight), a framework where trained experts critique and revise model-generated pre-captions into improved post-captions. This division of labor improves annotation accuracy and efficiency by offloading text generation to models, allowing humans to better focus on verification. Additionally, these critiques and preferences between pre- and post-captions provide rich supervision for improving open-source models (Qwen3-VL) on caption generation, reward modeling, and critique generation through SFT, DPO, and inference-time scaling. Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance. With modest expert supervision, the resulting model outperforms closed-source models such as Gemini-3.1-Pro. Finally, we apply our approach to re-caption large-scale professional videos (e.g., films, commercials, games) and fine-tune video generation models such as Wan to better follow detailed prompts of up to 400 words, achieving finer control over cinematography including camera motion, angle, lens, focus, point of view, and framing. Our results show that precise specification and human-AI oversight are key to professional-level video understanding and generation. Data and code are available on our project page: https://linzhiqiu.github.io/papers/chai/

Abstract

Explore similar work

Building a Precise Video Language with Human-AI Oversight

MoVA: Learning Asymmetric Dual Projections for Modular Long Video-Text Alignment

Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning