A Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos
Authors: David Miranda Paredes, Jose M. Saavedra, Marcelo Pizarro
Organizations: Facultad de Ingenier´ıa y Ciencias Aplicadas, Universidad de los Andes, Chile, Av. Monse˜nor ´Alvaro del Portillo 12455, Santiago, 100190, RM, Chile.
News videos are among the most prevalent content types produced by television stations and online streaming platforms, yet generating textual descriptions to facilitate indexing and retrieval largely remains a manual process. Video Large Language Models (VidLLMs) offer significant potential to automate this task, but a comprehensive evaluation in the news domain is still lacking. This work presents a comparative study of eight state-of-the-art open-source VidLLMs for automatic news video captioning, evaluated on two complementary benchmark datasets: a Chilean TV news corpus (approximately 1,345 clips) and a BBC News corpus (9,838 clips). We employ lexical metrics (METEOR, ROUGE-L), semantic metrics (BERTScore, CLIPScore, Text Similarity, Mean Reciprocal Rank), and two novel fidelity metrics proposed in this work: the Thematic Fidelity Score (TFS) and Entity Fidelity Score (EFS). Our analysis reveals that standard metrics exhibit limited discriminative power for news video captioning due to surface-form dependence, static-frame insensitivity, and function-word inflation. TFS and EFS address these gaps by directly assessing thematic structure preservation and named-entity coverage in the generated captions. Results show that Gemma~3 achieves the highest overall performance across both datasets and most evaluation dimensions, with Qwen-VL as a consistent runner-up.
Figures & tables
Figure 1: General pipeline of modern video captioning systems based on vision-language models (VLMs). The process begins with the raw video input, which undergoes temporal frame sampling (uniform, adaptive, or keyframe-based). The selected frames are processed by a vision encoder (typically ViT, SigLIP, EVA-CLIP, or similar architectures) to extract visual features. These features are then mapped into the language model’s embedding space via a cross-modal projector (usually an MLP adapter). Finally, the multimodal LLM (such as Qwen2-VL, LLaVA-series, InternVL, among others) integrates the visual tokens with the textual prompt and autoregressively generates the natural language caption.
Author
Year
Gemma 3.0
Google DeepMind
2025
Vript
Yang et al., Shanghai Jiao Tong Univ.
2024
InternVL 3.5
OpenGVLab, Shanghai AI Lab
2025
VideoLLaMA 2
Alibaba DAMO Academy
2024
VideoLLaMA 3
Alibaba DAMO Academy
2025
LLaVA-NeXT-Video
Li et al., ByteDance / NTU
2024
Table 1: Author and release year of each video description model evaluated in this study.
Model
Architecture
Visual Encoder
LLM Backbone
Size (Parameters)
Gemma 3.0
SigLIP encoder + MLP projector
SigLIP
Gemma
Gemma-3-12b-it
Vript
Video-script alignment + GPT-4V captioning
EVA ViT-g
ST-LLM (Vicuna)
Vriptor-STLLM-7B
InternVL 3.5
InternViT + MLP projector
InternViT-6B
Qwen2.5
InternVL3-8B
VideoLLaMA 2
STC + BEATs Audio encoder + ViT + Mistral
CLIP ViT-L/14
Mistral-Instruct, Qwen2
VideoLLaMA2.1-7B
VideoLLaMA 3
STC + Differential frame pruner + SigLIP
SigLIP
Qwen2.5-7B
VideoLLaMA3-7B
LLaVA-NeXT-Video
AnyRes + temporal pooling + MLP projector
CLIP ViT-L/14
Qwen2
LLaVA-NeXT-Video-7B
Table 2: Architectural details and parameter counts of each video description model evaluated in this study.
#
Thematic Label
1
Politics and Elections
2
International Affairs and Conflicts
3
Economy, Business and Finance
4
Society and Social Issues
5
Health and Medicine
6
Crime and Justice
Table 3: The 15 candidate thematic labels used for zero-shot classification.
Figure 2: A sample of video content from the ChTv dataset. ChTv was downloaded from https://www.13.cl/ .
Figure 3: Distribution of Video Clip durations and their description lengths (number of words) within the ChTv Dataset.
Figure 4: Distribution of ’Thematic Descriptors’ in the ChTv Dataset.
Figure 5: A sample of video content from the BBC dataset. BBC was downloaded from https://www.bbc.com/ .
Figure 6: Distribution of Video Clip durations and their description lengths (number of words) within the BBC News Dataset.
Figure 7: Distribution of ’Thematic Descriptors’ in the BBC News Dataset
Model
Text Sim.
CLIPScore
METEOR
ROUGE-L
BERTScore F1
Gemma 3
0.3112
0.3385
0.1281
0.0721
0.7809
Vript
0.2227
0.3175
0.1063
0.0604
0.7726
InternVL 3.5
0.2291
0.3178
0.1040
0.0583
0.7699
Video-LLaMA 2
0.2007
0.3134
0.1135
0.0768
0.7819
Video-LLaMA 3
0.1794
0.3065
0.0984
0.0766
0.7867
LLaVA-NeXT-Video
0.1999
0.3121
0.1050
0.0552
0.7703
Table 4: Summary of model performance metrics on the ChTV dataset . Bold values indicate the best score per metric.
Model
Text Sim.
CLIPScore
METEOR
ROUGE-L
BERTScore F1
Gemma 3
0.4387
0.3642
0.2017
0.1014
0.8330
Vript
0.2371
0.3228
0.1194
0.0624
0.8046
InternVL 3.5
0.2787
0.3358
0.1310
0.0669
0.8067
Video-LLaMA 2
0.2901
0.3237
0.1507
0.0928
0.8219
Video-LLaMA 3
0.3172
0.3426
0.1591
0.0874
0.8183
LLaVA-NeXT-Video
0.2294
0.3207
0.1202
0.0622
0.8048
Table 5: Summary of model performance metrics on the BBC dataset . Bold values indicate the best score per metric.
Figure 8: Shuffled-pairs validation of Text Similarity for the best-performing model (Gemma 3) and the lowest-performing model, illustrating the discriminative power of the metric.
Model
ChTV Dataset
BBC Dataset
Gemma 3
0.619
0.681
Vript
0.319
0.218
InternVL 3.5
0.326
0.263
Video-LLaMA 2
0.269
0.305
Video-LLaMA 3
0.235
0.328
LLaVA-NeXT-Video
0.252
0.211
Table 6: Mean Reciprocal Rank (MRR) across models and datasets. Gemma 3 achieves the highest MRR on both datasets, confirming that it consistently ranks first or near-first among all models for any given video.
Figure 9: Mean caption length across models on the BBC dataset.
Figure 10: Mean caption length across models on the ChTV dataset.
ChTV
BBC
Model
TFS
EFS
TFS
EFS
Gemma 3
0.861
0.059
0.912
0.314
Qwen-VL
0.865
0.051
0.904
0.198
Video-LLaMA 2
0.852
0.007
0.895
0.053
LLaVA-OneVision
0.860
0.007
0.894
0.059
Video-LLaMA 3
0.849
0.002
0.893
0.138
Table 7: Results of terms of TFS and EFS values both datasets. Models sorted by BBC TFS. Bold values indicate the best score per column.
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
Many decision-making tasks, where both accuracy and efficiency matter, still require human supervision. For example, tasks like traffic officers reviewing hour-long dashcam footage or researchers screening conference videos can benefit from concise summaries that reduce cognitive load and save time. Yet current vision-language models (VLMs) often produce verbose, redundant outputs that hinder task performance. Existing video caption evaluation depends on costly human annotations and overlooks the summaries' utility in downstream tasks. We address these gaps with Video-to-text Information Bottleneck Evaluation (VIBE), an annotation-free method that scores VLM outputs using two metrics: grounding (how well the summary aligns with visual content) and utility (how informative it is for the task). VIBE selects from randomly sampled VLM outputs by ranking them according to the two scores to support effective human decision-making. Human studies on LearningPaper24, SUTD-TrafficQA, and LongVideoBench show that summaries selected by VIBE consistently improve performance-boosting task accuracy by up to 61.23% and reducing response time by 75.77% compared to naive VLM summaries or raw video.
While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understanding remains challenging. Existing benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a benchmark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs frequently generate hallucinated descriptions in dense video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bottlenecks in current Video-LLMs.
Changbeen Kim, Junwon Chang, Kipyo Kim +3
Korea University · Sungkyunkwan University · The University of Tokyo +1