With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
Figures & tables
Figure 1: Left : The three-level capability hierarchy of Video-MME-v2: distribution of capability dimensions across Level 1 (information retrieval and aggregation), Level 2 (temporal understanding), and Level 3 (complex reasoning). Right : Models are ranked by their group-based non-linear scores, while average accuracy is provided for reference only. Due to API limitations, Gemini models are tested by extracting and compressing video frames to 60M, while GPT-5 is tested with an input of 50 frames.
Figure 2
Model
Frames
Non-Lin Score
Level 1
Level 2
Level 3
Consistency
Coherence
Avg Acc
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
w. sub
wo sub
Human Baseline
Human Expert
-
90.7
-
94.8
-
91.1
-
87.9
-
91.7
-
88.9
-
94.9
-
Commercialization Models
Gemini-3-Pro [ 10 ]
1fps
49.4
38.2
64.0
43.2
50.0
45.4
40.6
30.2
50.8
39.7
47.0
35.4
66.1
56.8
Doubao-Seed-2.0-Pro-260215 [ 3 ]
1fps
43.3
35.2
54.4
41.3
47.0
42.6
34.1
26.2
43.7
37.1
42.5
31.6
60.5
53.1
Table 1: Main Results on the Leaderboard. w. sub = with subtitle/audio; wo sub = without subtitle/audio (visual frames only). For Omni models that accept raw audio (e.g., MiMo-v2-Omni, Gemini-3-Pro), as well as the human baseline, we report results with raw audio input under the w. sub setting, and results with visual frames only under the wo sub setting. Sorting : models are ranked by their w. sub Non-Lin Score. Frames = video sampling rate (e.g., 1fps) or maximum sampled frames (e.g., 64, 512). Consistency = Capability consistency score. Coherence = Reasoning coherence score. Avg Acc = average accuracy across all QA pairs. We argue that the proposed group-based nonlinear scoring ( Non-Lin Score ) more faithfully reflects true model performance than the traditional per-question average accuracy (Avg Acc).
Model
Avg Acc (%)
Non-Lin Score (Ours)
Non-Lin / Avg Acc
Gemini-3-Pro
66.1
49.4
74.7%
Doubao-Seed-2.0-Pro-260215
60.5
43.3
71.6%
Gemini-3-Flash
61.1
42.5
69.6%
Qwen3.5-397B-A17B-Think (512)
55.9
39.1
69.9%
MiMo-v2-Omni
56.1
38.6
68.8%
GPT-5
55.6
37.0
66.5%
Table 2: Avg Acc vs. Non-Lin Score. The gap and the ratio reflect robustness: whether a model can answer multiple correlated questions correctly within the same group.
Figure 6: Q1–Q4 Accuracy Trends and Stability. Trends under (a) Capability consistency, (b) Reasoning coherence, and (c) mean/variance statistics under capability consistency.
Figure 7: Effect of Thinking Mode on Video-MME-v2. Performance changes induced by enabling Thinking for instruction-tuned baseline models, evaluated under both wo. subtitle and w. subtitle settings.
Figure 8: Capability Radar Across Video-MME-v2 Dimensions. For a wider range of models, please visit our project page, where you can select to view the radar chart performance of different models.
Lv
Parent Category
Leaf Category
Definition (detailed)
L1
Frame Only
Visual Recognition
Recognize static visual information from a single frame or a set of frames without requiring correct temporal order, including object/entity recognition, attribute recognition, scene understanding, and OCR-based text reading.
L1
Frame Only
Basic Counting
Count the number of target objects/entities appearing in the video (static counting). Aggregation across frames is allowed, but the correct answer should not depend on frame ordering.
L1
Frame Only
Numerical Calculation
Read numerical information from the visual stream (e.g., house numbers, clocks/timestamps shown on screen, displayed formulas) and perform simple arithmetic/comparisons. The task should not involve multi-hop reasoning beyond straightforward computation.
L1
Frames & Audio
Cross-Modal Semantic Consistency
Judge whether visual content and audio are semantically/affectively consistent (e.g., whether the perceived mood, event type, or semantic content matches). The question should not rely on temporal-change reasoning that belongs to Level 2.
L1
Frames & Audio
Audio-Guided Visual Description
Use an audio cue to temporally localize a moment, then answer what is shown visually at that moment. Distractors should be time-near and/or visually similar to enforce precise alignment.
L1
Frames & Audio
Vision-Guided Audio Description
Use a visual cue to temporally localize a moment, then answer what is said/heard (speech content, sound event, or audio attribute) at that moment.
Table 4: Video-MME v2 leaf-level task definitions (full version). This table provides detailed, reproducible descriptions for every leaf category in our Level 1–3 taxonomy, including the intended evidence type (visual/audio), whether temporal order is required, and the type of inference expected. These definitions are used as annotation guidelines so that each Group-QA can be assigned to a single leaf category with clear boundaries, enabling fine-grained analysis of capability consistency and reasoning coherence across models.
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
Geuntaek Lim, Sungjune Park, Jaeyun Lee +5
Sejong University, South Korea · NAVER Cloud, South Korea
The rapid progress of Large Language Models (LLMs) has spurred growing interest in Multi-modal LLMs (MLLMs) and motivated the development of benchmarks to evaluate their perceptual and comprehension abilities. Existing benchmarks, however, are limited to static images or single videos, overlooking the complex interactions across multiple videos. To address this gap, we introduce the Multi-Video Perception Evaluation Benchmark (MVPBench), a new benchmark featuring 14 subtasks across diverse visual domains designed to evaluate models on extracting relevant information from video sequences to make informed decisions. MVPBench includes 5K question-answering tests involving 2.7K video clips sourced from existing datasets and manually annotated clips. Extensive evaluations reveal that current models struggle to process multi-video inputs effectively, underscoring substantial limitations in their multi-video comprehension. We anticipate MVPBench will drive advancements in multi-video perception.
Purui Bai, Tao Wu, Jiayang Sun +3
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · SIST, ShanghaiTech University, Shanghai, China
Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in video understanding, existing benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world video understanding involves more challenging tasks that require multi-hop multimodal reasoning, and there is a critical absence of video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop video question answering benchmark focused on evaluating the spatial, temporal, and causal reasoning capabilities of MLLM agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our benchmark as a holistic testbed for future research on agentic video understanding. Code and datasets are available at https://github.com/krafton-ai/agentvidbench and https://huggingface.co/datasets/agentvidbench/agentvidbench.