Towards Cost-Efficient Video Dataset Construction with Large Language Models

Mohammed Shabot; Jonathan Adams; Jan Van den Stock; Joost Vennekens

cs.CVOct 12, 2026

Towards Cost-Efficient Video Dataset Construction with Large Language Models

Authors: Mohammed Shabot, Jonathan Adams, Jan Van den Stock, Joost Vennekens

Organizations: AI Lab, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium · Neuropsychiatry, KU Leuven, ON5/B Herestraat 49 - box 1029, 3000 Leuven, Belgium

Abstract

Research on video classification models has witnessed significant growth recently, accompanied by a significant increase in the demand for datasets to train such models. Existing video dataset curation approaches share a common bottleneck: candidate videos must be downloaded before any relevance judgment is made, which results in storage, time and annotation costs. In this work, we present a filtering step, using Large Language Models (LLMs) to pre-screen videos based on their metadata, before actually downloading these videos, thereby saving on these costs. We evaluate the LLMs against a strict human majority vote derived from four independent annotators and validate the reliability of the annotations through inter-annotator agreement analysis (Gwet's AC1). We compare the LLM pre-filter against two no-LLM baselines, namely two keyword heuristics. Via a Leave-One-Rater-Out (LOO) mechanism and an Add-One-Rater-In (AORI), we evaluate the performance of the LLMs as virtual annotators. Finally, we report cost comparisons relative to existing approaches. To ensure reproducability, we made our implementation available on github.

Figures & tables

Annotator pair	A – D	A – C	A – B	C – D	B – D	B – C
AC1	0.694	0.652	0.611	0.550	0.529	0.507

Table 1: Pairwise human AC1, metadata panel (

n=1047

Model	Prec.	95% CI	Recall	95% CI	F1	95% CI
QWEN3.6-36B	0.354	[0.271, 0.434]	0.565	[0.480, 0.645]	0.435	[0.357, 0.504]
Gemma4:31B	0.261	[0.193, 0.331]	0.857	[0.791, 0.917]	0.401	[0.314, 0.478]
GPT-5.2	0.285	[0.198, 0.373]	0.526	[0.421, 0.621]	0.370	[0.276, 0.450]
Claude Sonnet 4.6	0.286	[0.211, 0.366]	0.604	[0.506, 0.695]	0.388	[0.307, 0.464]
Regex B (stoplist+keyword)	0.269	[0.183, 0.353]	0.390	[0.286, 0.492]	0.318	[0.231, 0.401]
Regex A (stoplist)	0.181	[0.127, 0.233]	0.708	[0.624, 0.793]	0.288	[0.215, 0.353]

Table 2: Metadata-level classification performance (KEEP as positive class),

n=1047

, 95% cluster-bootstrap CIs. Lower block: no-LLM baselines.

Model	Rater	$\boldsymbol{\Delta_{\mathrm{LOO}}}$	90% CI
GPT-5.2	A	$-0.2973$	$[-0.3390,\,-0.2552]$
	B	$-0.2723$	$[-0.3142,\,-0.2290]$
	C	$-$ 0.2311	$[-0.2814,\,-0.1823]$
	D	$-0.2742$	$[-0.3177,\,-0.2314]$
Claude Sonnet 4.6	A	$-0.3304$	$[-0.3745,\,-0.2867]$
	B	$-0.3036$	$[-0.3498,\,-0.2626]$

Table 3: Leave-one-rater-out, metadata level (

n=1047

): change in panel AC1 (

\Delta_{\mathrm{LOO}}

, 90% CI) when each human annotator is replaced by an LLM. Reference (4-human) panel AC1

=0.59

. Every substitution is a significant degradation (90% CI entirely below zero).

Figure 1: Confusion matrix between metadata-level and video-level human majority labels (2-of-3 majority of annotators A, B and C, $n=298$ )

Model	5-rater panel AC1	$\Delta_{\mathrm{AORI}}$	90% CI
GPT-5.2	0.3755	$-0.2150$	[ $-0.2503$ , $-0.1802$ ]
Claude Sonnet 4.6	0.3482	$-0.2423$	[ $-0.2793$ , $-0.2068$ ]
QWEN3.6-36B	0.5379	$-$ 0.0525	[ $-0.0709$ , $-0.0350$ ]
Gemma4:31B	0.4246	$-0.1659$	[ $-0.1968$ , $-0.1354$ ]

Table 4: Add-one-rater-in (AORI), metadata level (

n=1047

): change in panel AC1 (

\Delta_{\mathrm{AORI}}

, 90% CI) when an LLM is added as a fifth annotator alongside a four-human panel. Reference panel AC1

=0.59

Method	Total	Saving	Cost/TP (meta)	Cost/TP (video)
Manual (no filter)	$119.20	—	$1.19	$1.37
GPT-5.2	$47.47	2.5 $\times$	$0.75	$0.97
Gemma4:31B	$46.00	2.6 $\times$	$0.55	$0.73
QWEN3.6-36B	$48.00	2.5 $\times$	$0.59	$0.74
Claude Sonnet 4.6	$54.28	2.2 $\times$	$0.66	$0.87
Regex A (stoplist)	$71.20	1.7 $\times$	$1.02	$1.13

Table 5: Full pipeline cost for

n=298

candidates ($8/hr, 3 min/video). Local models and baselines incur no-LLM cost.

Comparison (A $-$ B)	Metric	$\Delta$	95% CI	Sig.
QWEN3.6-36B – Regex A (stoplist)	F1	$+0.147$	[ $+0.085$ , $+0.207$ ]	yes
QWEN3.6-36B – Regex B (stoplist+keyword)	F1	$+0.117$	[ $+0.036$ , $+0.198$ ]	yes
Gemma4:31B – Regex A (stoplist)	F1	$+0.112$	[ $+0.065$ , $+0.162$ ]	yes
Claude Sonnet 4.6 – Regex A (stoplist)	F1	$+0.100$	[ $+0.037$ , $+0.156$ ]	yes
GPT-5.2 – Regex A (stoplist)	F1	$+0.082$	[ $+0.012$ , $+0.139$ ]	yes
Gemma4:31B – Regex B (stoplist+keyword)	F1	$+0.082$	[ $+0.013$ , $+0.154$ ]	yes

Table 9

Feature	Mean	SD	Median [IQR]
Title length (characters)	58.1	20.5	57.0 [43.0–73.0]
Title length (words)	9.9	3.6	10.0 [7.0–12.0]
Description length (characters)	1,196	1,084	901.0 [390.5–1,642]
Description length (words)	166.5	159.2	121.0 [49.2–225.0]
URLs in description	4.7	5.9	3.0 [0.0–7.0]
Hashtags in description	2.4	5.4	0.0 [0.0–3.0]

Table 7: Descriptive statistics of the YouTube metadata

Model	ROC-AUC	95% CI	PR-AUC	95% CI
QWEN3.6-36B	0.819	[0.780, 0.854]	0.370	[0.283, 0.462]
Gemma4:31B	0.730	[0.690, 0.776]	0.261	[0.200, 0.333]
GPT-5.2	0.728	[0.678, 0.775]	0.292	[0.216, 0.389]
Claude Sonnet 4.6	0.760	[0.710, 0.808]	0.314	[0.237, 0.406]

Table 8: Threshold-free classification performance at metadata level (

n=1047

Model	Metric	$t=0.5$	Tuned $t$	$\Delta$	95% CI	Sig.
QWEN3.6-36B	Precision	0.298	0.354	$+0.055$	[ $+0.019$ , $+0.096$ ]	yes
	Recall	0.851	0.565	$-0.286$	[ $-0.358$ , $-0.208$ ]	yes
	F1	0.442	0.435	$-0.007$	[ $-0.055$ , $+0.041$ ]	no
Gemma4:31B	Precision	0.232	0.261	$+0.030$	[ $+0.013$ , $+0.048$ ]	yes
	Recall	0.961	0.857	$-0.104$	[ $-0.168$ , $-0.045$ ]	yes
	F1	0.373	0.401	$+0.027$	[ $+0.003$ , $+0.051$ ]	yes

Table 9: Tuned versus fixed threshold (

t=0.5

), metadata level (

n=1047

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 156 papers

Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Sep 30, 2026Zhixi Zhu, Kristina GligoricVLM Evaluation Video-Language Model Evaluation
Johns Hopkins University
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Sep 9, 2026Killian Steunou, Yannis Tevissen, Mounîm A. El YacoubiAudio-Visual Understanding Efficient VLM Inference
SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Sep 15, 2026Haoyu Guo, Yuan Feng, Junlin Lv +3Efficient VLM Inference Long-Video Understanding
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1

Sep 30, 2026J/K move · Enter open · S save

Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics

Zhixi Zhu, Kristina Gligoric

Johns Hopkins University

VLM Evaluation Video-Language Model Evaluation LLM-Assisted Annotation Computational Social Science

Open paper AskPDF arXiv

Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single

2\times8

image grid built via simple shot-transition detection approaches full-video understanding (

κ

within~.05), at

\sim 15\%

of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.

Abstract

Training Data Curation

Explore similar work

Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs