VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
Authors: Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu, Zheng Lian, Xinyu Geng, Jingdong Chen, Yi Yuan, +1 more
Organizations: The Chinese University of Hong Kong · Ant Group · Tsinghua University · Tongji University · The Hong Kong University of Science and Technology
Video understanding is a fundamental task for evaluating the capabilities of multimodal large language models (MLLMs). However, existing leading models have already achieved approximately 90% accuracy on the Video-MME leaderboard, suggesting that conventional single-turn video understanding tasks are becoming increasingly saturated and insufficient for assessing the intelligence of advanced MLLMs. Towards this end, we introduce VideoGAIA, an agentic video understanding benchmark for general artificial intelligence (AI) assistants. Moving beyond one-shot video question answering, VideoGAIA formulates video understanding as a multi-turn, tool-augmented interaction process, where models must iteratively perceive videos, invoke external tools, gather complementary information, and integrate multimodal evidence across turns. VideoGAIA contains 271 model-human co-designed tasks covering diverse and complex real-world scenarios. Each video-question-answer instance is independently verified by three human experts to ensure both correctness and appropriate difficulty. All evaluated MLLMs, including frontier models such as GPT-5.5 and Kimi-K3, achieve less than 60% accuracy on VideoGAIA, highlighting its value as a high-quality and timely benchmark for evaluating next-generation MLLMs. We hope that VideoGAIA will facilitate the transition from conventional video understanding toward agentic video understanding.
Figures & tables
Figure 1: VideoGAIA focuses on agentic video understanding, which is more challenging compared with conventional video understanding benchmarks.
Figure 2: (a) Real-World Tasks. VideoGAIA spans 6 key domains and 39 subtasks. (b) Leaderboard . Performance comparison of 20 frontier MLLMs is shown under the agent loop setting.
Figure 3: The benchmark construction pipeline of VideoGAIA.
Figure 4: Overview of VideoGAIA dataset statistics, including category and difficulty distributions, video duration, question and answer token lengths, and a question-answer word cloud.
Model
Culture
Geography
Daily Life
History
Technology
Economics
Overall
Agent Loop
Qwen3.5-397B Qwen Team (2026a)
43.10
52.38
55.74
45.45
56.41
33.33
50.18
Qwen3.5-Plus Qwen Team (2026a)
46.55
47.62
54.10
63.64
56.41
16.67
52.03
Qwen3.6-Plus Qwen Team (2026b)
46.55
46.03
50.82
54.55
48.72
50.00
49.08
Qwen3.7-Plus Qwen Team (2026c)
50.00
58.73
57.38
56.82
58.97
50.00
56.09
Qwen3.8-Max Qwen Team (2026d)
44.83
42.86
62.30
50.00
51.28
50.00
50.18
Table 1: The pass@1 (%) results of various models under the agent loop setting on VideoGAIA.
Figure 5: Performance comparison among various frontier models under two difficulty levels.
Figure 6: Tool-use distributions of five representative models under the agent loop setting.
Figure 7: Distribution of dominant error causes among incorrect predictions from five models.
Figure 8: The visualization of correct and incorrect video agent trajectories on VideoGAIA.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Overall accuracy versus response length and output cost across 20 models on VideoGAIA.
Figure 10: Radar plot comparison of 20 video agents across six categories on VideoGAIA.
Figure 11: Tool-use distributions of 15 frontier models under the agent loop setting. Percentages are computed over all explicit tool calls on both correct and incorrect trajectories.
Figure 12: Distribution of dominant error causes among incorrect predictions from 15 frontier models. Each failed trajectory is assigned exactly one category.
Figure 13: An example from the Culture domain of VideoGAIA.
Figure 14: An example from the Geography domain of VideoGAIA.
Figure 15: An example from the Daily Life domain of VideoGAIA.
Figure 16: An example from the History domain of VideoGAIA.
Figure 17: An example from the Technology domain of VideoGAIA.
Figure 18: An example from the Economics domain of VideoGAIA.
Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in video understanding, existing benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world video understanding involves more challenging tasks that require multi-hop multimodal reasoning, and there is a critical absence of video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop video question answering benchmark focused on evaluating the spatial, temporal, and causal reasoning capabilities of MLLM agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our benchmark as a holistic testbed for future research on agentic video understanding. Code and datasets are available at https://github.com/krafton-ai/agentvidbench and https://huggingface.co/datasets/agentvidbench/agentvidbench.
Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a group-based non-linear evaluation strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by 3,300 human-hours and up to 5 rounds of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.