With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
Figures & tables
Figure 1: Left : The three-level capability hierarchy of Video-MME-v2: distribution of capability dimensions across Level 1 (information retrieval and aggregation), Level 2 (temporal understanding), and Level 3 (complex reasoning). Right : Models are ranked by their group-based non-linear scores, while average accuracy is provided for reference only. Due to API limitations, Gemini models are tested by extracting and compressing video frames to 60M, while GPT-5 is tested with an input of 50 frames.
Figure 2
Model
Frames
Non-Lin Score
Level 1
Level 2
Level 3
Consistency
Coherence
Avg Acc
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
Human Baseline
Human Expert
-
90.7
-
94.8
-
91.1
-
87.9
-
91.7
-
88.9
-
94.9
-
Commercialization Models
Gemini-3-Pro [ 10 ]
1fps
49.4
38.2
64.0
43.2
50.0
45.4
40.6
30.2
50.8
39.7
47.0
35.4
66.1
56.8
Doubao-Seed-2.0-Pro-260215 [ 3 ]
1fps
43.3
35.2
54.4
41.3
47.0
42.6
34.1
26.2
43.7
37.1
42.5
31.6
60.5
53.1
Table 1: Main Results on the Leaderboard. w. sub = with subtitle/audio; wo sub = without subtitle/audio (visual frames only). For Omni models that accept raw audio (e.g., MiMo-v2-Omni, Gemini-3-Pro), as well as the human baseline, we report results with raw audio input under the w. sub setting, and results with visual frames only under the wo sub setting. Sorting : models are ranked by their w. sub Non-Lin Score. Frames = video sampling rate (e.g., 1fps) or maximum sampled frames (e.g., 64, 512). Consistency = Capability consistency score. Coherence = Reasoning coherence score. Avg Acc = average accuracy across all QA pairs. We argue that the proposed group-based nonlinear scoring ( Non-Lin Score ) more faithfully reflects true model performance than the traditional per-question average accuracy (Avg Acc).
Model
Avg Acc (%)
Non-Lin Score (Ours)
Non-Lin / Avg Acc
Gemini-3-Pro
66.1
49.4
74.7%
Doubao-Seed-2.0-Pro-260215
60.5
43.3
71.6%
Gemini-3-Flash
61.1
42.5
69.6%
Qwen3.5-397B-A17B-Think (512)
55.9
39.1
69.9%
MiMo-v2-Omni
56.1
38.6
68.8%
GPT-5
55.6
37.0
66.5%
Table 2: Avg Acc vs. Non-Lin Score. The gap and the ratio reflect robustness: whether a model can answer multiple correlated questions correctly within the same group.
Figure 6: Q1–Q4 Accuracy Trends and Stability. Trends under (a) Capability consistency, (b) Reasoning coherence, and (c) mean/variance statistics under capability consistency.
Figure 7: Effect of Thinking Mode on Video-MME-v2. Performance changes induced by enabling Thinking for instruction-tuned baseline models, evaluated under both wo. subtitle and w. subtitle settings.
Figure 8: Capability Radar Across Video-MME-v2 Dimensions. For a wider range of models, please visit our project page, where you can select to view the radar chart performance of different models.
Lv
Parent Category
Leaf Category
Definition (detailed)
L1
Frame Only
Visual Recognition
Recognize static visual information from a single frame or a set of frames without requiring correct temporal order, including object/entity recognition, attribute recognition, scene understanding, and OCR-based text reading.
L1
Frame Only
Basic Counting
Count the number of target objects/entities appearing in the video (static counting). Aggregation across frames is allowed, but the correct answer should not depend on frame ordering.
L1
Frame Only
Numerical Calculation
Read numerical information from the visual stream (e.g., house numbers, clocks/timestamps shown on screen, displayed formulas) and perform simple arithmetic/comparisons. The task should not involve multi-hop reasoning beyond straightforward computation.
L1
Frames & Audio
Cross-Modal Semantic Consistency
Judge whether visual content and audio are semantically/affectively consistent (e.g., whether the perceived mood, event type, or semantic content matches). The question should not rely on temporal-change reasoning that belongs to Level 2.
L1
Frames & Audio
Audio-Guided Visual Description
Use an audio cue to temporally localize a moment, then answer what is shown visually at that moment. Distractors should be time-near and/or visually similar to enforce precise alignment.
L1
Frames & Audio
Vision-Guided Audio Description
Use a visual cue to temporally localize a moment, then answer what is said/heard (speech content, sound event, or audio attribute) at that moment.
Table 4: Video-MME v2 leaf-level task definitions (full version). This table provides detailed, reproducible descriptions for every leaf category in our Level 1–3 taxonomy, including the intended evidence type (visual/audio), whether temporal order is required, and the type of inference expected. These definitions are used as annotation guidelines so that each Group-QA can be assigned to a single leaf category with clear boundaries, enabling fine-grained analysis of capability consistency and reasoning coherence across models.