cs.CVSep 18, 2026

VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration

Authors: Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim

Organizations: Korea University · Sungkyunkwan University · The University of Tokyo · OMRON SINIC X Corporation

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understanding remains challenging. Existing benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a benchmark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs frequently generate hallucinated descriptions in dense video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bottlenecks in current Video-LLMs.

Figures & tables

Figure 1: Top: Previous benchmarks typically rely on MCQA or sparse GT captions, allowing models to succeed with limited temporal evidence. Bottom: In VidOmni-Bench, models must verify each event in dense captions against the video, enabling a more rigorous assessment of fine-grained video understanding.
Figure 2: We introduce VidOmni-Bench, a novel benchmark designed to evaluate how well Video-LLMs distinguish between correct and incorrect events across five types of spatio-temporal complexity and diverse video durations. This structured framework enables a rigorous assessment of whether Video-LLMs can truly ground their reasoning in the visual evidence across diverse scales.
Figure 3: Construction pipeline of VidOmni-Bench. We collect videos from public datasets and YouTube along with two axes. Then, we utilize diverse Video-LLMs for dense caption generation. These video and dense caption pairs then undergo a multi-stage annotation phase for sentence-wise error (hallucination) labeling, where annotators label sentence-level errors to construct hard negatives. Finally, the data filtering stage for a robust evaluation.
Figure 4: Our dataset consists of 500 videos, with exactly 100 videos allocated to each of the five complexity types. Within each type, we maintain a balanced distribution across Short, Medium, and Long duration.
 KeyeVL 1.5InternVL 3.5Qwen3-VLvideo-SALMONN 2+Gemini 3
Params.8B38B32B72BN/A
Total Sentences18974554211868189749485
Video Incorrect Rate (%)92.685.694.092.291.1
Incorrect / Caption (%)40.639.526.544.924.1
Table 1: Statistics of incorrect sentence in generated captions. VidOmni-Bench provides a large set of annotated incorrect sentences for evaluating sentence-wise event verification. Sentences labeled as Unknown are excluded.
Figure 5: Word-level error statistics per category. The consistent high frequency of temporal errors in dense video captioning highlights a fundamental bottleneck in video understanding.
Figure 6: Effect of video complexity and duration on hallucination. Plots show the average sentence-level Incorrect Rate per caption. Left : across durations for Type 5 videos. Right : across complexity types within Long videos. Dashed lines indicate overall trends.
Figure 7: Caption characteristics across complexity types. Higher-complexity videos lead to longer captions and a higher Video Incorrect Rate for temporal error categories, while also requiring broader video evidence in sentence-wise event verification of VidOmni-Bench.
BenchmarkEvaluation TaskDurationCert. Cov. (%)
TimeBlind Li et al. (2026)MCQA≤\leq 25s32.3
NOAH Lee et al. (2025)GT Matching / MCQA≤\leq 12m30.8
ARGUS Rawal et al. (2025)GT Matching≤\leq 3m25.0
HERBench Ben Ami et al. (2026)MCQA≤\leq 6m24.8
Video-MME Fu et al. (2025)MCQA≤\leq 60m3.4
CAPAbility Liu et al. (2026)GT Matching / MCQA≤\leq 50s2.0
Table 2: Comparison with other benchmarks in task design and temporal evidence. VidOmni-Bench evaluates sentence-wise event verification over longer videos and requires broader temporal evidence, indicated by higher Certificate Coverage (Cert. Cov.).
Video-LLMModalityVideo-MMEVideo-MMMULVBenchVidOmni-Bench
QA-Acc. (%)QA-Acc. (%)QA-Acc. (%)Precision (%)Recall (%)F1-Score (%)
Open-Source Models
Keye-VL 1.5 8Bvv73.066.035.1 †19.18.49.6
InternVL 3.5 2Bvv58.441.8 †30.5 †6.01.82.3
InternVL 3.5 8Bvv66.048.9 †34.2 †26.413.914.7
InternVL 3.5 14Bvv67.954.8 †35.1 †32.318.119.4
Table 3: Main results on VidOmni-Bench. We report verification Precision, Recall, and F1-Score for VidOmni-Bench, alongside QA Accuracy (QA-Acc) for Video-MME, Video-MMMU, and LVBench. Models are evaluated under video-only ( vv ) or video+audio ( v,av,a ) settings. †\dagger denotes reproduced performance.
Figure 8: Performance of Video-LLMs across error categories. This visualization highlights the relative strengths and weaknesses of current Video-LLMs in detecting specific static and temporal discrepancies.
Figure 9: Qualitative examples of sentence-wise event verification on a Qwen3-VL caption. Video-LLMs may produce the same prediction while relying on different reasoning, revealing varying degrees of video understanding.
Figure 10: F1-Score variations by video duration under fixed complexity. Each column represents a specific complexity type, illustrating how performance changes across Short, Medium, and Long durations.
Figure 11: Recall gain by error category as model size increases. The heatmap shows the improvement in Recall for InternVL3.5, Qwen3-VL, and video-SALMONN 2+ when scaling up parameters.
Figure 12: Self-preference bias in self-verification. Left: The heatmap shows F1-Scores for sentence-wise verification, with the y-axis representing the evaluated models and the x-axis representing the caption source models. Right: The bar chart compares the performance on cross-source verification versus self-verification.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

ComplexityNameVideo Source (Public Dataset)
Type 1Static WorldVideo-ACID Hosler et al. (2019) SpatialVID Wang et al. (2026) Mannequin Challenge Li et al. (2020)
Type 2Body MotionActivityNet Caba Heilbron et al. (2015)
Type 3Hand-Object InteractionAssembly101 Sener et al. (2022) Epic-Kitchens Damen et al. (2018) Ego4D Grauman et al. (2022)
Type 4Crowd ActionUCF-Crime Sultani et al. (2018) Sports-1M Karpathy et al. (2014)
Type 5Multi-Scene DynamicsVideo-MME Fu et al. (2025) XD-Violence Wu et al. (2020)
Appendix
Table 5: Data sources of each complexity type in VidOmni-Bench . Videos are primarily collected from public datasets. We further supplement underrepresented types and duration ranges with videos sourced from YouTube.
ModelProviderOpen/ClosedFPSMax FramesRelease
Gemini 3 FlashGoogleClosed1-2026/02
Qwen3-VL 32BAlibabaOpen220482026/01
InternVL 3.5 38BOpenGVLabOpen1322025/12
video-SALMONN 2+ 72BByteDanceOpen17682025/11
Keye-VL 1.5 8BKwaiOpen110242025/03
Appendix
Table 6: Details of Video-LLMs chosen as captioning engines. We use five Video-LLMs with diverse providers, accessibility, FPS, and maximum number of frames (Max Frames) to generate dense captions.
Figure 13: Prompt templates for dense video captioning. We use five prompt templates to generate dense captions for each video, reducing the potential instruction bias that may arise from relying on a single captioning prompt.
TypeDescriptionExample
ObjectMisdescriptions of the object itself including its attributes (color, text, count, etc.).There are two blue cars, but described as “three red cars.”
RelationMisdescriptions of spatial relationships between objects.A cup is under the table, but described as “on the table.”
LocationMisdescriptions of an object’s position.A cup is on the right side, but described as “on the left.”
DirectionMisdescriptions of orientation or direction of an object.A person is looking to the right, but described as “looking to the left.”
IllusionMentioning non-existent objects, events, or actions.No bird is present, but described as “a bird flying in the sky.”
ActionIncorrect or incomplete description of actions over time.A person is walking, but described as “running.”
Appendix
Table 7: Word-level error categories in VidOmni-Bench. We group word-level errors into five static-frame-related categories(Object, Relation, Location, Direction, and Illusion) and five temporal-understanding-related categories(Action, Order, Causality, Frequency, and State Change).
Figure 14: Sentence-wise event verification instruction. The prompt asks Video-LLMs to assign a binary label to each caption sentence before providing sentence-level reasoning.
ModelProviderOpen/ClosedFPSMax FramesRelease
Gemini 3 FlashGoogleClosed1-2026/02
GPT 5 miniOpenAIClosed-2562026/03
Qwen3-VL 32BAlibabaOpen220482026/01
InternVL 3.5 38BOpenGVLabOpen17682025/12
Video-SALMONN 2+ 72BByteDanceOpen1322025/11
Keye-VL 1.5 8BKwaiOpen110242025/03
Appendix
Table 8: Details of Video-LLMs evaluated on VidOmni-Bench. We list the provider, accessibility, FPS, and maximum number of input frames (Max Frames) for each Video-LLM evaluated on VidOmni-Bench.
BenchmarkAvg. WordsInfo CountNoun ChunksVerbsLLM Judge Win Rate (%) (vs. Ours)
ARGUS Rawal et al. (2025)477.5336.4134.363.14.0
NOAH Lee et al. (2025)48.034.213.88.00.6
CAPAbility Liu et al. (2026)3.22.41.10.30.4
VidOmni-Bench578.9400.6159.382.1–
Appendix
Table 10: Comparison of caption informativeness across benchmarks. VidOmni-Bench achieves substantially longer captions, higher information counts, and more noun chunks and verbs than prior benchmarks. The last column reports the percentage of pairwise LLM-as-a-judge comparisons in which the baseline caption is preferred over the VidOmni-Bench caption in terms of informativeness.
Figure 15: Prompt for pairwise caption informativeness judgment. We use Prometheus-2 7B to compare captions from VidOmni-Bench and other benchmarks under a predefined informativeness rubric. Given the captioning instruction, two captions, and the rubric, the judge selects the more informative caption.
Video-LLMEnsemble VerifierF1-Score (%)
Self-VerificationCross-Ensemble (+p.p)
Qwen3-VL 32B+ InternVL3.5 38B5.326.2 ( +20.9 )
InternVL3.5 38B+ Keye-VL 1.5 8B7.724.6 ( +16.9 )
Gemini 3 Flash+ GPT 5 mini21.925.1 ( +3.2 )
Appendix
Table 11: Effect of cross-model ensemble on self-verification. Cross-model ensemble verification consistently improves F1-Score over self-verification across caption sources. These gains show that combining judgments from other Video-LLMs can compensate for the low verification performance observed when models evaluate their own captions.
Video-LLMAudio CategoryF1-Score Gain (%)ttSignificant
Gemini 3 FlashSpeech+3.3+3.9
Sound Effect+4.2+3.6
Background−- 0.8−- 0.9n.s.
video-SALMONN 2+ 7BSpeech−- 0.3−- 1.7n.s.
Sound Effect−- 1.5−- 2.5
Background−- 0.9−- 2.7**
Appendix
Table 13: F1-Score gain when audio is added by audio category and model. For each model and audio category, we compute the F1-Score gain (with audio - without audio) and test whether this gain is significantly different from zero using a one-sample tt -test. * p<.05p<.05 , ** p<.01p<.01 , *** p<.001p<.001 , n.s.: not significant p≥.05p\geq.05 .
Figure 16: Performances of Video-LLMs across all complexity types and durations . The results reveal multidimensional and model-specific weaknesses, indicating that current Video-LLMs exhibit different failure patterns depending on both video dynamics and temporal scale.
Figure 17: Annotation interface for sentence-wise correctness (Stage 1) . Annotators label each caption sentence as Correct, Incorrect, or Unknown based on the video evidence. An additional memo field is provided to collect detailed notes and ambiguous cases during dataset construction.
Figure 18: Annotation interface for word-level error category (Stage 2). For each sentence labeled as Incorrect in Stage 1, annotators identify the erroneous words or phrases and assign them to predefined error categories, enabling fine-grained analysis of hallucination types.
More in this topic

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 103 papers

Explore similar work

CardsList
May 26, 2026

Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models

A key capability for video understanding is reliably linking subjects to events across time, yet whether Video Large Language Models (VideoLLMs) actually achieve this remains unclear. In this work, we introduce DistractionBench to evaluate whether VideoLLMs can robustly link subjects and events in the presence of unrelated video segments. Through controlled interventions, such as inserting short advertisement clips into longer videos, we show that VideoLLMs frequently hallucinate interactions between entities from different segments, incorrectly attributing actions from injected advertisements to subjects in the main video. We characterize this systematic hallucination as bag-of-events (BoE) behavior, where models process videos as collections of events rather than temporally structured sequences. Evaluating 11 popular VideoLLMs, we find that all models exhibit substantial BoE behavior. Our findings suggest that VideoLLMs lack reliable mechanisms for temporal grounding and motivate the development of models with more robust subject-event association.

Oscar Chew, Serhii Honcharenko, Qian-Hui Chen +4

Texas A&M University · National Taiwan University · Stanford University +1

Temporal Video GroundingVideo-Language Model EvaluationVideo-Language ModelsMultimodal Model EvaluationTemporal Video Understanding
AskPDF
Jun 1, 2026

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

Video multimodal large language models (MLLMs) have made rapid progress in general and long-form video understanding, yet their ability to capture and use brief, answer-critical visual evidence remains underexplored. Momentary actions or state transitions may determine an answer while occupying only a small fraction of the video. Performance on broad video understanding tasks alone does not establish whether models can reliably identify and use this temporally localized evidence. We introduce Moment-Video, a benchmark for evaluating temporal fidelity through momentary visual event understanding. Questions are constructed around visually observable events or event sequences, requiring models to notice, count, describe, or reason about transient evidence. The benchmark contains 1,000 human-verified video-QA pairs across seven domains and 25 subcategories, covering four tasks: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. We evaluate 34 proprietary and open-source MLLMs. Under the main evaluation settings, the best-performing model, Seed-2.0-Pro, achieves 39.6% overall accuracy, while most open-source models remain below 25%. Diagnostic analyses show that denser frame sampling improves some models, but substantial errors persist even when sampled frames cover all annotated necessary event intervals in an audited subset. Performance also declines on longer videos, suggesting additional temporal-localization challenges. By exposing these limitations, Moment-Video provides a diagnostic benchmark to guide the evaluation and development of video MLLMs that more reliably understand brief, answer-critical visual events.

Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9

Shandong University · Southeast University · Shanghai Jiao Tong University +1

VLM EvaluationVideo QAVideo-Language ModelsTemporal Video Understanding
AskPDF
Mar 31, 2026

Video-Oasis: Rethinking Evaluation of Video Understanding

The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.

Geuntaek Lim, Sungjune Park, Jaeyun Lee +5

Sejong University, South Korea · NAVER Cloud, South Korea

VLM EvaluationTemporal Video Understanding
AskPDF