Meta-Video: Efficient Standardized Evaluation for Video Understanding
Authors: Soumya Shamarao Jahagirdar, Anna Kukleva, Vishaal Udandarao, Samuel Thomas, Brian Kingsbury, Rogerio Feris, James R. Glass. Ameya Prabhu, Hilde Kuehne
Organizations: University of Tübingen · Max Planck Institute for Informatics, SIC · IBM Research · MIT-IBM Watson AI Lab · MIT · Tübingen AI Center
Video understanding benchmarks are the primary yardstick for measuring progress in video-language modeling, and a growing body of work relies on them to claim improvements. The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully. We show these unexamined choices are a major source of variance. Even for the same model checkpoint, evaluation configuration alone can shift accuracy by up to 19.5pp. Hence, it is unclear whether reported gains reflect genuine improvements in capability or differences in evaluation setup. Moreover, many benchmark samples offer little evidence of visual capability, since they can be answered without the video, and the full suites are expensive to evaluate. We address these issues in two parts. First, we curate existing suites for discriminability and efficiency, producing Meta-Video, a balanced 1,000-question benchmark over 791 videos spanning diverse tasks and durations, which largely preserves model rankings, at 5 times inference speedup for a full-frame count sweep experiment (4-128 frames). Second, we establish a standardized evaluation protocol by sweeping the configuration space to isolate choices that measurably affect results, distill them into best practices and reporting standards, and release reproducible reference scores across five models and seven benchmarks. We hope that the proposed evaluation framework and the Meta-Video benchmark will support video understanding research by providing a reproducible and efficient foundation to measure progress in a reliable way.
Figures & tables
General Video Understanding
Video Reasoning
Model
VideoMME
TempCompass
MVBench
LongVideoBench
LVBench
VideoMMMU
MMVU mcq
MMVU val
VSIBench
SciVideoBench
VideoMathQA
Minerva
Base model: Qwen2.5-VL-7B
Qwen2.5-VL-7B [ 5 ]
65.1
71.7
69.6
56.0
45.3
47.4
–
50.1
–
–
–
–
VideoRFT [ 59 ]
52.8 -12.3
69.2 -2.5
59.4 -10.2
56.0
45.3
47.4
61.3
–
31.8
–
–
–
Videochat-R1.5 [ 67 ]
64.4 -0.7
–
67.4 -2.2
–
–
46.7 -0.7
–
–
–
–
–
–
Open-o3-video [ 45 ]
62.4 -2.7
–
–
–
–
51.2 +3.8
–
–
–
–
–
–
Table 1: Performance reported for the Qwen2.5-VL-7B and Qwen3-VL-8B backbone. across general video understanding and video reasoning benchmarks. Max Δ denotes the range of scores for each benchmark, differing by up to 19.5 pp despite identical model weights. Ours (reproduced) are reported for 128 frames setting.
Figure 1: Meta-Video statistics. The benchmark contains 1,000 QA pairs over 791 distinct videos.
Figure 2: Evaluation framework. Parameters in the evaluation pipeline that affect reported accuracy for video understanding benchmarks.
bck
bs
Video-MME
TempComp
MVBench
LongVideoB
Avg.
Meta-V
HF
1
50.0
64.4
59.2
50.3
56.0
24.8
4
50.0
64.7
59.6
50.5
56.2
24.4
8
50.0
65.0
59.3
50.7
56.2
24.1
16
50.1
64.8
59.2
50.4
56.1
24.6
32
50.0
64.9
59.3
50.4
56.1
24.6
Runtime
157
46.6
166
77.2
447
133
Table 2: Sensitivity of evaluation settings and parameters on Qwen2.5-VL-7B. Each sub-table varies one setting; all other settings are fixed as listed. All runs use the official lmms_eval prompt. All values are reported as accuracy (%), except Runtime , which gives the total runtime in minutes of all runs in the block above them. Avg. is the mean accuracy over the four benchmarks; in the Runtime rows, Avg. gives the summed runtime over all benchmarks. ± denotes the standard deviation over repeated runs. Column labels: nf (number of frames), bck (backend), bs (batch size), rdr (video reader), px (max pixels), samp (sampling; ≤n means 1 fps capped at n frames), T (temperature).
Evaluation Card (Table ref )
Scope
Models
Model-A-7B, Model-B-13B
Datasets
Dataset-1, Dataset-2
Runtime
Inference backend
backend-name
Batch size
{ 1, 8, 32 }
Input
Input modality
.mp4
No. of frames
{ 8, 16, 32 }
Table 3: Evaluation card. Reporting the settings that affects a score makes results reproducible and comparable across papers. Options in bold are the ones used; a set {⋅} denotes a parameter sweep.
Model
#Frames
Video-
Long-
Temp-
MMVU
LV-
MLVU
VSI-
Avg.
Meta-V
MME
VideoBench
Compass
(Val)
Bench
Bench
Short
Medium
Long
Avg.
InternVL3.5-4B
4
54.5
52.1
69.1
46.4
38.0
59.5
46.3
52.3
35.4
24.2
32.3
30.6
8
57.7
54.5
68.9
46.7
38.9
60.6
50.6
54.0
45.0
32.7
35.1
37.6
16
60.7
56.8
71.1
48.6
41.7
63.4
55.1
56.8
50.4
36.2
37.9
41.5
32
63.1
58.3
71.3
50.8
42.6
67.1
56.8
58.6
53.9
41.2
41.5
45.5
64
64.8
59.1
69.8
49.9
43.1
67.7
57.3
58.8
52.7
41.1
40.1
44.6
Table 4: Re-evaluated vs. reported scores. For each model, the upper rows give our own evaluation under a single fixed protocol as a function of the frame budget, with all other settings held fixed; the shaded row at the bottom of each block reproduces the numbers as reported in the literature, placed next to our largest frame budget for direct comparison. Avg. is the mean over the seven benchmarks; Meta-V is highlighted in blue and reported separately for short, medium and long videos, together with their unweighted mean. Bold marks the best frame budget per model in each column. – denotes an entry that is not reported or not evaluated. Evaluation card is provided in 11 .
Prompt
Input modality
Layout
Number of frames
Max new token
Meta-V
.mp4
.jpg
Small
Medium
Long
Avg.
P0 (default)
✓
✗
–
4
default
26.7
22.7
25.1
24.9 -2.1
P1 (think step-by-step)
✓
✗
–
4
1024
17.8
27.7
30.7
24.8 -2.26
P∗ (single letter only)
✓
✗
–
4
default
26.0
27.4
27.9
27.0
P0 (default)
✗
✓
VT
4
default
30.7
26.5
27.9
28.5 -1.23
P1 (think step-by-step)
✗
✓
VT
4
1024
21.4
31.6
26.8
26.4 -3.36
Table 5: Meta-Video : Prompt and input modality ablation (.mp4 video vs. .jpg frames) with 4 frames. Average is the weighted mean of short, medium and long splits. Evaluation card is provided in Table 12 .
Prompt
Layout
Number of frames
Max new token
Meta-V
Small
Medium
Long
Avg.
P0 (default)
VT
32
default
39.8
36.4
34.2
37.1 -1.3
P1 (think step-by-step)
VT
32
1024
28.9
34.9
27.5
30.5 -7.9
P∗ (single letter only)
VT
32
default
37.9
42.3
34.2
38.4
P0 (default)
TVT
32
default
41.8
37.9
30.7
37.4 -0.7
P1 (think step-by-step)
TVT
32
1024
28.2
32.8
31.0
30.6 -7.5
Table 6: Meta-Video : Prompt and layout ablation across 32, 64 and 128 frames. All runs use .jpg frames as input; V and T denote the position of the visual input and the text prompt, respectively. Average is the weighted mean of short, medium and long splits; arrows show the difference to P∗ within each block. Evaluation card is provided in Table 13 .
Figure 3: Reported vs. reproduced MMVU accuracy. Left: paper-reported scores (dark) vs. our reproductions at 32/64/128 frames (light). Right: prompt sensitivity of Qwen2.5-VL-7B across six prompts. The shaded band is the range of reproduced RL results, which falls within the base model’s prompt variance.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Batch size
HF
vLLM original
vLLM fixed
Δ
1
24.81
31.00
26.25
−4.75
4
24.34
30.52
26.16
−4.35
8
24.01
30.62
26.86
−3.77
16
24.61
30.40
26.66
−3.74
32
24.62
30.47
27.22
−3.26
Appendix
Table 7: In this table, we provide the scores of Meta-V after re-evaluating with the fixed vLLM, and compare it with the existing vLLM scores.
Figure 4: Blind baseline performance vs. four uniformly sampled frames, for Qwen2.5-VL-7b
Model
Video-
Long-
Temp-
MMVU
LV-
MV-
VideoMMMU
SciVideo-
MME
VideoBench
Compass
(Val)
Bench
Bench
Perc.
Comp.
Adapt.
Bench
#Q
2700
1337
1580
1000
1549
4000
300
300
300
1000
Qwen2.5-VL-7B
42.7
41.8
44.9
35.0
24.9
40.3
37.3
28.7
22.3
17.3
Qwen3-VL-8B
39.0
41.6
37.9
39.7
15.2
31.2
26.7
29.7
26.3
23.1
InternVL3.5-8B
43.6
45.9
41.7
39.4
26.7
35.3
35.0
31.0
30.3
23.9
Appendix
Table 8: Blind baseline. Scores when every input frame is replaced by a black frame, so the only usable signal is the question and answer text. All runs use 4 frames, HuggingFace inference, batch size 1, temperature 0; max_pixels =602,112 for the Qwen models and max_patches =12 for InternVL3.5. #Q is the number of questions evaluated.
#Frames
Model
4
8
16
32
64
128
Qwen2.5-VL-7B
50.1
53.4
58.1
61.4
63.5
65.6
Video-R1-7B
48.5 (-1.6)
52.0 (-1.4)
56.3 (-1.8)
60.0 (-1.4)
61.5 (-2.0)
62.5 (-3.1)
VideoChat-R1.5-7B
51.7 (+1.6)
54.1 (+0.7)
58.8 (+0.7)
61.4 (0.0)
63.4 (-0.1)
63.7 (-1.9)
Video-R2-7B
50.1 (0.0)
53.4 (0.0)
57.6 (-0.5)
61.3 (-0.1)
63.4 (-0.1)
64.9 (-0.7)
Appendix
Table 9: VideoMME (2,700 questions) accuracy (%) vs. number of uniformly sampled frames. All reasoning models are fine-tuned from Qwen2.5-VL-7B-Instruct; parentheses give the difference to the base model.
MMVU
SciVideoBench
Model
Prompt
32
64
128
32
64
128
Qwen2.5-VL-7B
default
63.36
65.92
64.64
26.1
26.9
27.9
CoT
66.24
65.28
68.96
24.4
23.6
25.6
video r1
64.00
64.96
63.04
23.4
26.1
26.7
video r2
62.08
61.28
63.36
23.6
25.3
25.5
video chat r15 think
59.20
59.52
60.64
24.0
27.3
26.7
Appendix
Table 10: Accuracy (%) on MMVU (validation split, MCQ, 625 items) and SciVideoBench (test split, 1000 items) with 32, 64, and 128 input frames. Best result per column in bold.
Figure 5: Video duration of VideoMME [ 21 ] , LongVideoBench [ 65 ] , LVBench [ 61 ] .
Table 11: Evaluation card for Table 4 . Options in bold are the ones used; a set {⋅} denotes a parameter sweep (one run per value).
Evaluation Card (Table 5 )
Scope
Models
Qwen2.5-VL-7B
Datasets
Meta-V (short / medium / long)
Runtime
Inference backend
HF
Batch size
1 (.mp4; .jpg with vt) ; 16 (.jpg with tvt, vtv)
Input
Input modality
{ .mp4, .jpg }
No. of frames
4
Appendix
Table 12: Evaluation card for Table 5 . Options in bold are the ones used; a set {⋅} denotes a parameter sweep (one run per value).
Evaluation Card (Table 6 )
Scope
Models
Qwen2.5-VL-7B
Datasets
Meta-V (short / medium / long)
Runtime
Inference backend
HF
Batch size
16
Input
Input modality
.jpg (frames passed as images)
No. of frames
{ 32, 64, 128 }
Appendix
Table 13: Evaluation card for Table 6 . Options in bold are the ones used; a set {⋅} denotes a parameter sweep (one run per value).
4 frames
8 frames
16 frames
32 frames
64 frames
128 frames
Model
Dataset
1
2
4
8
12
1
2
4
8
12
1
2
4
8
1
2
1
1
InternVL3.5-4B
Video-MME
54.6
53.4
53.4
53.2
53.2
57.7
58.1
58.2
56.5
56.5
60.7
60.8
60.8
58.7
63.1
63.0
64.8
64.1
MVBench
67.0
67.9
67.8
66.5
66.7
70.0
70.6
70.5
68.2
68.0
71.4
71.7
71.8
67.1
71.9
71.5
71.3
69.4
TempCompass (MC)
69.2
68.2
68.2
68.2
68.2
68.9
68.8
68.8
68.8
68.8
71.1
70.2
70.2
70.2
71.4
69.4
69.9
68.9
MLVU (dev)
59.6
59.7
59.5
58.2
57.0
60.6
61.1
61.1
60.8
60.1
63.4
63.3
62.8
62.6
67.1
67.2
67.7
68.5
LongVideoBench (val)
52.1
52.7
52.6
52.3
52.3
54.5
54.4
54.4
54.3
53.9
56.8
57.6
57.6
55.9
58.3
57.8
59.2
58.9
Appendix
Table 14: InternVL3.5 accuracy (%) versus number of input frames and maximum number of tiles per frame ( max_num_patches ; second header row). Input: raw .mp4 decoded to exactly N frames, tiled video path ( hf_tiled ). HF backend, fixed seed. Bold: best setting per row.
4 frames
8 frames
16 frames
32 frames
64 frames
128 frames
Model
Dataset
1
2
4
8
12
1
2
4
8
12
1
2
4
8
1
2
1
1
InternVL3.5-4B
TempCompass (MC)
64.6
65.4
65.4
65.4
65.4
66.8
67.0
67.0
67.0
67.0
68.4
66.6
66.6
66.6
68.4
66.3
68.4
66.1
MMVU (val)
46.6
46.4
46.4
47.6
44.0
46.1
47.1
47.1
47.1
45.1
47.1
47.9
47.9
48.2
48.2
47.9
47.8
47.5
InternVL3.5-8B
TempCompass (MC)
65.0
65.3
65.3
65.3
65.3
66.8
66.8
66.8
66.8
66.8
68.2
67.6
67.6
67.6
69.1
67.7
67.8
66.8
MMVU (val)
54.1
57.4
57.4
55.1
54.4
55.6
56.3
56.3
56.3
55.7
56.4
55.8
55.8
55.6
55.4
56.1
54.6
55.1
Appendix
Table 15: InternVL3.5 accuracy (%) versus number of input frames and maximum number of tiles per frame ( max_num_patches ; second header row). Input: N pre-extracted .jpg frames passed as images, stock HF image tiler ( hf ). HF backend, fixed seed. Bold: best setting per row.
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
Geuntaek Lim, Sungjune Park, Jaeyun Lee +5
Sejong University, South Korea · NAVER Cloud, South Korea
A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index
Enxin Song, Yinuo Xu, Shusheng Yang +2
University of Pennsylvania · New York University · Princeton University