cs.CVOct 1, 2026

Video-Index: A Curated Meta-Benchmark for Video Understanding

Authors: Enxin Song, Yinuo Xu, Shusheng Yang, Wenhao Chai, Jiatao Gu

Organizations: University of Pennsylvania · New York University · Princeton University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index

Figures & tables

Figure 1: From pool to Video-Index. We filter the 505,518 items in four stages and keep the 840 hardest.
Figure 2: Options and question text replace the video on 30 benchmarks. Each benchmark is placed by the full video’s gain over options alone and over the question text, and shaded bands mark breaks.
Figure 3: Remembering earlier answers narrows the video gap. Each column is a benchmark, with its gap to the strongest pool attacker after 25 to 100% of the items.
Figure 4: Captions recover more of the video score than a single frame. Each column is a benchmark, sorted by its caption gap, with gaps under no video, one frame, and captions.
Figure 5: Much of the score survives temporal perturbation. Each dot is a benchmark, placed by the share of full-video accuracy that each perturbation retains.
Figure 6: Newer benchmarks break earlier. a , Benchmarks that reach and break at each level. b , Breaking levels by release year, with the share breaking before visual input in red.
Figure 7: Each capability group breaks at its own levels. Bars split the benchmarks of each group by breaking level, and segment labels give counts.
Figure 8: Eleven benchmarks gain more from pixels and three from frames. The curve is the density of pixel gain minus frame gain, and dots mark survivors with both ladders.
Figure 9: Most long videos gain from larger budgets. a , Scores per budget with means and quartiles, blue if the reference still gains at 600 frames. b , The six largest gains, 512 to 1024.
Figure 10: Errors concentrate in few capabilities. Columns are the 76 pool-level survivors with 20 or more agreed attributions, grouped by their first breaking level, each summing to 100%.
Figure 11: Near-duplicates shrink effective size. a , Effective items per 200, red where at least half are near-duplicates. b , The largest near-duplicate shares across benchmarks.
Figure 12: Agents differ in accuracy, runtime, and inspection. a , Accuracy and median runtime, Q1–Q3. b , Operations per answer. c , Image medians by duration on shared log axes.
\toprule ModelVideoBlindGainPercep.Temp.SpatialReason.
\midrule VideoChat3-4B ( li2026videochat3 )9.43.85.79.512.46.29.5
Qwen3-VL-8B ( bai2025qwen3 )9.51.48.19.011.46.711.0
Cambrian-S-7B ( yang2026cambrian )11.35.36.012.410.03.819.0
InternVideo2.5-8B ( wang2025internvideo2 )12.47.54.912.911.08.117.6
VideoLLaMA3-7B ( zhang2025videollama )12.45.27.111.918.68.610.5
Video-XL-2 ( qin2025video )12.47.35.112.415.79.012.4
Table 1: Agents lead open models on Video-Index. Accuracy in %, fixed input then agent tools. Blind sees text only and Gain is Video minus Blind. Highest in bold. Protocols in the appendix.

Explore similar work

CardsList

  1. Video-Oasis: Rethinking Evaluation of Video Understanding

    Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5Video Understanding

    Sejong University, South Korea · NAVER Cloud, South Korea

  2. Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models

    Sep 15, 2026Rwiddhi Chakraborty, Yinong, Wang +7Video UnderstandingText Analysis and Detection

    Oliver · University of Copenhagen · Carnegie Mellon University +4

  3. VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing

    Aug 6, 2026Ziyun Zeng, Zixuan Wang, Yongsheng Yu +2Long VideosVideo Generation

    University of Rochester