Meta-Video: Efficient Standardized Evaluation for Video Understanding

Soumya Shamarao Jahagirdar; Anna Kukleva; Vishaal Udandarao; Samuel Thomas; Brian Kingsbury; Rogerio Feris; James R. Glass. Ameya Prabhu; Hilde Kuehne

cs.CVOct 13, 2026

Meta-Video: Efficient Standardized Evaluation for Video Understanding

Authors: Soumya Shamarao Jahagirdar, Anna Kukleva, Vishaal Udandarao, Samuel Thomas, Brian Kingsbury, Rogerio Feris, James R. Glass. Ameya Prabhu, Hilde Kuehne

Organizations: University of Tübingen · Max Planck Institute for Informatics, SIC · IBM Research · MIT-IBM Watson AI Lab ·

Abstract

Video understanding benchmarks are the primary yardstick for measuring progress in video-language modeling, and a growing body of work relies on them to claim improvements. The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully. We show these unexamined choices are a major source of variance. Even for the same model checkpoint, evaluation configuration alone can shift accuracy by up to 19.5pp. Hence, it is unclear whether reported gains reflect genuine improvements in capability or differences in evaluation setup. Moreover, many benchmark samples offer little evidence of visual capability, since they can be answered without the video, and the full suites are expensive to evaluate. We address these issues in two parts. First, we curate existing suites for discriminability and efficiency, producing Meta-Video, a balanced 1,000-question benchmark over 791 videos spanning diverse tasks and durations, which largely preserves model rankings, at 5 times inference speedup for a full-frame count sweep experiment (4-128 frames). Second, we establish a standardized evaluation protocol by sweeping the configuration space to isolate choices that measurably affect results, distill them into best practices and reporting standards, and release reproducible reference scores across five models and seven benchmarks. We hope that the proposed evaluation framework and the Meta-Video benchmark will support video understanding research by providing a reproducible and efficient foundation to measure progress in a reliable way.

Figures & tables

Base model: Qwen2.5-VL-7B
	General Video Understanding					Video Reasoning
Model	VideoMME	TempCompass	MVBench	LongVideoBench	LVBench	VideoMMMU	MMVU mcq	MMVU val	VSIBench	SciVideoBench	VideoMathQA	Minerva
Qwen2.5-VL-7B [ 5 ]	65.1	71.7	69.6	56.0	45.3	47.4	–	50.1	–	–	–	–
VideoRFT [ 59 ]	52.8 -12.3	69.2 -2.5	59.4 -10.2	56.0	45.3	47.4	61.3	–	31.8	–	–	–
Videochat-R1.5 [ 67 ]	64.4 -0.7	–	67.4 -2.2	–	–	46.7 -0.7	–	–	–	–	–	–
Open-o3-video [ 45 ]	62.4 -2.7	–	–	–	–	51.2 +3.8	–	–	–	–	–	–

Table 1: Performance reported for the Qwen2.5-VL-7B and Qwen3-VL-8B backbone. across general video understanding and video reasoning benchmarks. Max

\Delta

denotes the range of scores for each benchmark, differing by up to

19.5

pp despite identical model weights. Ours (reproduced) are reported for 128 frames setting.

Panel of Figure 1 — Figure 1: Meta-Video statistics. The benchmark contains 1,000 QA pairs over 791 distinct videos.

Figure 2: Evaluation framework. Parameters in the evaluation pipeline that affect reported accuracy for video understanding benchmarks.

bck	bs	Video-MME	TempComp	MVBench	LongVideoB	Avg.	Meta-V
HF	1	50.0	64.4	59.2	50.3	56.0	24.8
	4	50.0	64.7	59.6	50.5	56.2	24.4
	8	50.0	65.0	59.3	50.7	56.2	24.1
	16	50.1	64.8	59.2	50.4	56.1	24.6
	32	50.0	64.9	59.3	50.4	56.1	24.6
Runtime		157	46.6	166	77.2	447	133

Table 2: Sensitivity of evaluation settings and parameters on Qwen2.5-VL-7B. Each sub-table varies one setting; all other settings are fixed as listed. All runs use the official lmms_eval prompt. All values are reported as accuracy (%), except Runtime , which gives the total runtime in minutes of all runs in the block above them. Avg. is the mean accuracy over the four benchmarks; in the Runtime rows, Avg. gives the summed runtime over all benchmarks.

\pm

denotes the standard deviation over repeated runs. Column labels: nf (number of frames), bck (backend), bs (batch size), rdr (video reader), px (max pixels), samp (sampling;

\leq n

means 1 fps capped at

n

frames),

T

(temperature).

Evaluation Card (Table ref )
Scope	Models	Model-A-7B, Model-B-13B
	Datasets	Dataset-1, Dataset-2
Runtime	Inference backend	backend-name
	Batch size	$\{$ 1, 8, 32 $\}$
Input	Input modality	.mp4
	No. of frames	$\{$ 8, 16, 32 $\}$

Table 3: Evaluation card. Reporting the settings that affects a score makes results reproducible and comparable across papers. Options in bold are the ones used; a set

\{\cdot\}

denotes a parameter sweep.

Model	#Frames	Video-	Long-	Temp-	MMVU	LV-	MLVU	VSI-	Avg.	Meta-V
Model	#Frames	MME	VideoBench	Compass	(Val)	Bench	MLVU	Bench	Avg.	Short	Medium	Long	Avg.
InternVL3.5-4B	4	54.5	52.1	69.1	46.4	38.0	59.5	46.3	52.3	35.4	24.2	32.3	30.6
	8	57.7	54.5	68.9	46.7	38.9	60.6	50.6	54.0	45.0	32.7	35.1	37.6
	16	60.7	56.8	71.1	48.6	41.7	63.4	55.1	56.8	50.4	36.2	37.9	41.5
	32	63.1	58.3	71.3	50.8	42.6	67.1	56.8	58.6	53.9	41.2	41.5	45.5
	64	64.8	59.1	69.8	49.9	43.1	67.7	57.3	58.8	52.7	41.1	40.1	44.6

Table 4: Re-evaluated vs. reported scores. For each model, the upper rows give our own evaluation under a single fixed protocol as a function of the frame budget, with all other settings held fixed; the shaded row at the bottom of each block reproduces the numbers as reported in the literature, placed next to our largest frame budget for direct comparison. Avg. is the mean over the seven benchmarks; Meta-V is highlighted in blue and reported separately for short, medium and long videos, together with their unweighted mean. Bold marks the best frame budget per model in each column. – denotes an entry that is not reported or not evaluated. Evaluation card is provided in 11 .

Prompt	Input modality		Layout	Number of frames	Max new token	Meta-V
Prompt	.mp4	.jpg	Layout	Number of frames	Max new token	Small	Medium	Long	Avg.
$P0$ (default)	✓	✗	–	4	default	26.7	22.7	25.1	24.9 -2.1
$P1$ (think step-by-step)	✓	✗	–	4	1024	17.8	27.7	30.7	24.8 -2.26
$P^{*}$ (single letter only)	✓	✗	–	4	default	26.0	27.4	27.9	27.0
$P0$ (default)	✗	✓	VT	4	default	30.7	26.5	27.9	28.5 -1.23
$P1$ (think step-by-step)	✗	✓	VT	4	1024	21.4	31.6	26.8	26.4 -3.36

Table 5: Meta-Video : Prompt and input modality ablation (.mp4 video vs. .jpg frames) with 4 frames. Average is the weighted mean of short, medium and long splits. Evaluation card is provided in Table 12 .

Prompt	Layout	Number of frames	Max new token	Meta-V
Prompt	Layout	Number of frames	Max new token	Small	Medium	Long	Avg.
$P0$ (default)	VT	32	default	39.8	36.4	34.2	37.1 -1.3
$P1$ (think step-by-step)	VT	32	1024	28.9	34.9	27.5	30.5 -7.9
$P^{*}$ (single letter only)	VT	32	default	37.9	42.3	34.2	38.4
$P0$ (default)	TVT	32	default	41.8	37.9	30.7	37.4 -0.7
$P1$ (think step-by-step)	TVT	32	1024	28.2	32.8	31.0	30.6 -7.5

Table 6: Meta-Video : Prompt and layout ablation across 32, 64 and 128 frames. All runs use .jpg frames as input; V and T denote the position of the visual input and the text prompt, respectively. Average is the weighted mean of short, medium and long splits; arrows show the difference to

P^{*}

within each block. Evaluation card is provided in Table 13 .

Figure 3: Reported vs. reproduced MMVU accuracy. Left: paper-reported scores (dark) vs. our reproductions at 32/64/128 frames (light). Right: prompt sensitivity of Qwen2.5-VL-7B across six prompts. The shaded band is the range of reproduced RL results, which falls within the base model’s prompt variance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Batch size	HF	vLLM original	vLLM fixed	$\Delta$
1	24.81	31.00	26.25	$-4.75$
4	24.34	30.52	26.16	$-4.35$
8	24.01	30.62	26.86	$-3.77$
16	24.61	30.40	26.66	$-3.74$
32	24.62	30.47	27.22	$-3.26$

Appendix

Table 7: In this table, we provide the scores of Meta-V after re-evaluating with the fixed vLLM, and compare it with the existing vLLM scores.

Figure 4: Blind baseline performance vs. four uniformly sampled frames, for Qwen2.5-VL-7b

Model	Video-	Long-	Temp-	MMVU	LV-	MV-	VideoMMMU			SciVideo-
Model	MME	VideoBench	Compass	(Val)	Bench	Bench	Perc.	Comp.	Adapt.	Bench
#Q	2700	1337	1580	1000	1549	4000	300	300	300	1000
Qwen2.5-VL-7B	42.7	41.8	44.9	35.0	24.9	40.3	37.3	28.7	22.3	17.3
Qwen3-VL-8B	39.0	41.6	37.9	39.7	15.2	31.2	26.7	29.7	26.3	23.1
InternVL3.5-8B	43.6	45.9	41.7	39.4	26.7	35.3	35.0	31.0	30.3	23.9

Appendix

Table 8: Blind baseline. Scores when every input frame is replaced by a black frame, so the only usable signal is the question and answer text. All runs use 4 frames, HuggingFace inference, batch size 1, temperature 0; max_pixels

=602{,}112

for the Qwen models and max_patches

=12

for InternVL3.5. #Q is the number of questions evaluated.

	#Frames
Model	4	8	16	32	64	128
Qwen2.5-VL-7B	50.1	53.4	58.1	61.4	63.5	65.6
Video-R1-7B	48.5 (-1.6)	52.0 (-1.4)	56.3 (-1.8)	60.0 (-1.4)	61.5 (-2.0)	62.5 (-3.1)
VideoChat-R1.5-7B	51.7 (+1.6)	54.1 (+0.7)	58.8 (+0.7)	61.4 (0.0)	63.4 (-0.1)	63.7 (-1.9)
Video-R2-7B	50.1 (0.0)	53.4 (0.0)	57.6 (-0.5)	61.3 (-0.1)	63.4 (-0.1)	64.9 (-0.7)

Appendix

Table 9: VideoMME (2,700 questions) accuracy (%) vs. number of uniformly sampled frames. All reasoning models are fine-tuned from Qwen2.5-VL-7B-Instruct; parentheses give the difference to the base model.

		MMVU			SciVideoBench
Model	Prompt	32	64	128	32	64	128
Qwen2.5-VL-7B	default	63.36	65.92	64.64	26.1	26.9	27.9
	CoT	66.24	65.28	68.96	24.4	23.6	25.6
	video r1	64.00	64.96	63.04	23.4	26.1	26.7
	video r2	62.08	61.28	63.36	23.6	25.3	25.5
	video chat r15 think	59.20	59.52	60.64	24.0	27.3	26.7

Appendix

Table 10: Accuracy (%) on MMVU (validation split, MCQ, 625 items) and SciVideoBench (test split, 1000 items) with 32, 64, and 128 input frames. Best result per column in bold.

Figure 5: Video duration of VideoMME [ 21 ] , LongVideoBench [ 65 ] , LVBench [ 61 ] .

Evaluation Card (Table 4 )
Scope	Models	InternVL3.5-4B, Qwen3-VL-4B, Qwen2.5-VL-7B, InternVL3.5-8B, Qwen3-VL-8B
	Datasets	Video-MME, LongVideoBench, TempCompass, MMVU (val), LVBench, VSI-Bench, MLVU, Meta-V (short / medium / long)
Runtime	Inference backend	HF
	Batch size	per no. of frames: 4 $\to$ 16, 8 $\to$ 8, 16 $\to$ 4, 32 $\to$ 4, 64 $\to$ 2, 128 $\to$ 1
Input	Input modality	.mp4
	No. of frames	$\{$ 4, 8, 16, 32, 64, 128 $\}$

Appendix

Table 11: Evaluation card for Table 4 . Options in bold are the ones used; a set

\{\cdot\}

denotes a parameter sweep (one run per value).

Evaluation Card (Table 5 )
Scope	Models	Qwen2.5-VL-7B
	Datasets	Meta-V (short / medium / long)
Runtime	Inference backend	HF
	Batch size	1 (.mp4; .jpg with vt) ; 16 (.jpg with tvt, vtv)
Input	Input modality	$\{$ .mp4, .jpg $\}$
	No. of frames	4

Appendix

Table 12: Evaluation card for Table 5 . Options in bold are the ones used; a set

\{\cdot\}

denotes a parameter sweep (one run per value).

Evaluation Card (Table 6 )
Scope	Models	Qwen2.5-VL-7B
	Datasets	Meta-V (short / medium / long)
Runtime	Inference backend	HF
	Batch size	16
Input	Input modality	.jpg (frames passed as images)
	No. of frames	$\{$ 32, 64, 128 $\}$

Appendix

Table 13: Evaluation card for Table 6 . Options in bold are the ones used; a set

\{\cdot\}

denotes a parameter sweep (one run per value).

		4 frames					8 frames					16 frames				32 frames		64 frames	128 frames
Model	Dataset	1	2	4	8	12	1	2	4	8	12	1	2	4	8	1	2	1	1
InternVL3.5-4B	Video-MME	54.6	53.4	53.4	53.2	53.2	57.7	58.1	58.2	56.5	56.5	60.7	60.8	60.8	58.7	63.1	63.0	64.8	64.1
	MVBench	67.0	67.9	67.8	66.5	66.7	70.0	70.6	70.5	68.2	68.0	71.4	71.7	71.8	67.1	71.9	71.5	71.3	69.4
	TempCompass (MC)	69.2	68.2	68.2	68.2	68.2	68.9	68.8	68.8	68.8	68.8	71.1	70.2	70.2	70.2	71.4	69.4	69.9	68.9
	MLVU (dev)	59.6	59.7	59.5	58.2	57.0	60.6	61.1	61.1	60.8	60.1	63.4	63.3	62.8	62.6	67.1	67.2	67.7	68.5
	LongVideoBench (val)	52.1	52.7	52.6	52.3	52.3	54.5	54.4	54.4	54.3	53.9	56.8	57.6	57.6	55.9	58.3	57.8	59.2	58.9

Appendix

Table 14: InternVL3.5 accuracy (%) versus number of input frames and maximum number of tiles per frame ( max_num_patches ; second header row). Input: raw .mp4 decoded to exactly

N

frames, tiled video path ( hf_tiled ). HF backend, fixed seed. Bold: best setting per row.

		4 frames					8 frames					16 frames				32 frames		64 frames	128 frames
Model	Dataset	1	2	4	8	12	1	2	4	8	12	1	2	4	8	1	2	1	1
InternVL3.5-4B	TempCompass (MC)	64.6	65.4	65.4	65.4	65.4	66.8	67.0	67.0	67.0	67.0	68.4	66.6	66.6	66.6	68.4	66.3	68.4	66.1
	MMVU (val)	46.6	46.4	46.4	47.6	44.0	46.1	47.1	47.1	47.1	45.1	47.1	47.9	47.9	48.2	48.2	47.9	47.8	47.5
InternVL3.5-8B	TempCompass (MC)	65.0	65.3	65.3	65.3	65.3	66.8	66.8	66.8	66.8	66.8	68.2	67.6	67.6	67.6	69.1	67.7	67.8	66.8
	MMVU (val)	54.1	57.4	57.4	55.1	54.4	55.6	56.3	56.3	56.3	55.7	56.4	55.8	55.8	55.6	55.4	56.1	54.6	55.1

Appendix

Table 15: InternVL3.5 accuracy (%) versus number of input frames and maximum number of tiles per frame ( max_num_patches ; second header row). Input:

N

pre-extracted .jpg frames passed as images, stock HF image tiler ( hf ). HF backend, fixed seed. Bold: best setting per row.

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 103 papers

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Apr 6, 2026Chaoyou Fu, Haozhi Yuan, Yuhao Dong +16Video Understanding Video-Language Model Evaluation
Video-Oasis: Rethinking Evaluation of Video Understanding
Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM Evaluation Temporal Video Understanding
Sejong University, South Korea · NAVER Cloud, South Korea
Video-Index: A Curated Meta-Benchmark for Video Understanding
Oct 1, 2026Enxin Song, Yinuo Xu, Shusheng Yang +2Video Understanding Video-Language Model Evaluation
University of Pennsylvania · New York University · Princeton University

Apr 6, 2026J/K move · Enter open · S save

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang+15

Video Understanding Video-Language Model Evaluation VLM Robustness Multimodal Reasoning Temporal Video Understanding

Open paper AskPDF arXiv

With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.

Abstract

Video-Language Model Evaluation

Explore similar work

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

Video-Oasis: Rethinking Evaluation of Video Understanding

Video-Index: A Curated Meta-Benchmark for Video Understanding