cs.CVSep 18, 2026
SaveVidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration
Organizations: Korea University · Sungkyunkwan University · The University of Tokyo · OMRON SINIC X Corporation
Abstract
While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understanding remains challenging. Existing benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a benchmark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs frequently generate hallucinated descriptions in dense video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bottlenecks in current Video-LLMs.
Figures & tables
Figure 1: Top: Previous benchmarks typically rely on MCQA or sparse GT captions, allowing models to succeed with limited temporal evidence. Bottom: In VidOmni-Bench, models must verify each event in dense captions against the video, enabling a more rigorous assessment of fine-grained video understanding.
Figure 2: We introduce VidOmni-Bench, a novel benchmark designed to evaluate how well Video-LLMs distinguish between correct and incorrect events across five types of spatio-temporal complexity and diverse video durations. This structured framework enables a rigorous assessment of whether Video-LLMs can truly ground their reasoning in the visual evidence across diverse scales.
Figure 3: Construction pipeline of VidOmni-Bench. We collect videos from public datasets and YouTube along with two axes. Then, we utilize diverse Video-LLMs for dense caption generation. These video and dense caption pairs then undergo a multi-stage annotation phase for sentence-wise error (hallucination) labeling, where annotators label sentence-level errors to construct hard negatives. Finally, the data filtering stage for a robust evaluation.
Figure 4: Our dataset consists of 500 videos, with exactly 100 videos allocated to each of the five complexity types. Within each type, we maintain a balanced distribution across Short, Medium, and Long duration.
| KeyeVL 1.5 | InternVL 3.5 | Qwen3-VL | video-SALMONN 2+ | Gemini 3 | |
| Params. | 8B | 38B | 32B | 72B | N/A |
| Total Sentences | 18974 | 5542 | 11868 | 18974 | 9485 |
| Video Incorrect Rate (%) | 92.6 | 85.6 | 94.0 | 92.2 | 91.1 |
| Incorrect / Caption (%) | 40.6 | 39.5 | 26.5 | 44.9 | 24.1 |
Table 1: Statistics of incorrect sentence in generated captions. VidOmni-Bench provides a large set of annotated incorrect sentences for evaluating sentence-wise event verification. Sentences labeled as Unknown are excluded.
Figure 5: Word-level error statistics per category. The consistent high frequency of temporal errors in dense video captioning highlights a fundamental bottleneck in video understanding.
Figure 6: Effect of video complexity and duration on hallucination. Plots show the average sentence-level Incorrect Rate per caption. Left : across durations for Type 5 videos. Right : across complexity types within Long videos. Dashed lines indicate overall trends.
Figure 7: Caption characteristics across complexity types. Higher-complexity videos lead to longer captions and a higher Video Incorrect Rate for temporal error categories, while also requiring broader video evidence in sentence-wise event verification of VidOmni-Bench.
| Benchmark | Evaluation Task | Duration | Cert. Cov. (%) |
| TimeBlind Li et al. (2026) | MCQA | 25s | 32.3 |
| NOAH Lee et al. (2025) | GT Matching / MCQA | 12m | 30.8 |
| ARGUS Rawal et al. (2025) | GT Matching | 3m | 25.0 |
| HERBench Ben Ami et al. (2026) | MCQA | 6m | 24.8 |
| Video-MME Fu et al. (2025) | MCQA | 60m | 3.4 |
| CAPAbility Liu et al. (2026) | GT Matching / MCQA | 50s | 2.0 |
Table 2: Comparison with other benchmarks in task design and temporal evidence. VidOmni-Bench evaluates sentence-wise event verification over longer videos and requires broader temporal evidence, indicated by higher Certificate Coverage (Cert. Cov.).
| Video-LLM | Modality | Video-MME | Video-MMMU | LVBench | VidOmni-Bench | ||
| QA-Acc. (%) | QA-Acc. (%) | QA-Acc. (%) | Precision (%) | Recall (%) | F1-Score (%) | ||
| Open-Source Models | |||||||
| Keye-VL 1.5 8B | 73.0 | 66.0 | 35.1 † | 19.1 | 8.4 | 9.6 | |
| InternVL 3.5 2B | 58.4 | 41.8 † | 30.5 † | 6.0 | 1.8 | 2.3 | |
| InternVL 3.5 8B | 66.0 | 48.9 † | 34.2 † | 26.4 | 13.9 | 14.7 | |
| InternVL 3.5 14B | 67.9 | 54.8 † | 35.1 † | 32.3 | 18.1 | 19.4 | |
Table 3: Main results on VidOmni-Bench. We report verification Precision, Recall, and F1-Score for VidOmni-Bench, alongside QA Accuracy (QA-Acc) for Video-MME, Video-MMMU, and LVBench. Models are evaluated under video-only ( ) or video+audio ( ) settings. denotes reproduced performance.
Figure 8: Performance of Video-LLMs across error categories. This visualization highlights the relative strengths and weaknesses of current Video-LLMs in detecting specific static and temporal discrepancies.
Figure 9: Qualitative examples of sentence-wise event verification on a Qwen3-VL caption. Video-LLMs may produce the same prediction while relying on different reasoning, revealing varying degrees of video understanding.
Figure 10: F1-Score variations by video duration under fixed complexity. Each column represents a specific complexity type, illustrating how performance changes across Short, Medium, and Long durations.
Figure 11: Recall gain by error category as model size increases. The heatmap shows the improvement in Recall for InternVL3.5, Qwen3-VL, and video-SALMONN 2+ when scaling up parameters.
Figure 12: Self-preference bias in self-verification. Left: The heatmap shows F1-Scores for sentence-wise verification, with the y-axis representing the evaluated models and the x-axis representing the caption source models. Right: The bar chart compares the performance on cross-source verification versus self-verification.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Complexity | Name | Video Source (Public Dataset) |
| Type 1 | Static World | Video-ACID Hosler et al. (2019) SpatialVID Wang et al. (2026) Mannequin Challenge Li et al. (2020) |
| Type 2 | Body Motion | ActivityNet Caba Heilbron et al. (2015) |
| Type 3 | Hand-Object Interaction | Assembly101 Sener et al. (2022) Epic-Kitchens Damen et al. (2018) Ego4D Grauman et al. (2022) |
| Type 4 | Crowd Action | UCF-Crime Sultani et al. (2018) Sports-1M Karpathy et al. (2014) |
| Type 5 | Multi-Scene Dynamics | Video-MME Fu et al. (2025) XD-Violence Wu et al. (2020) |
Table 5: Data sources of each complexity type in VidOmni-Bench . Videos are primarily collected from public datasets. We further supplement underrepresented types and duration ranges with videos sourced from YouTube.
| Model | Provider | Open/Closed | FPS | Max Frames | Release |
| Gemini 3 Flash | Closed | 1 | - | 2026/02 | |
| Qwen3-VL 32B | Alibaba | Open | 2 | 2048 | 2026/01 |
| InternVL 3.5 38B | OpenGVLab | Open | 1 | 32 | 2025/12 |
| video-SALMONN 2+ 72B | ByteDance | Open | 1 | 768 | 2025/11 |
| Keye-VL 1.5 8B | Kwai | Open | 1 | 1024 | 2025/03 |
Table 6: Details of Video-LLMs chosen as captioning engines. We use five Video-LLMs with diverse providers, accessibility, FPS, and maximum number of frames (Max Frames) to generate dense captions.
Figure 13: Prompt templates for dense video captioning. We use five prompt templates to generate dense captions for each video, reducing the potential instruction bias that may arise from relying on a single captioning prompt.
| Type | Description | Example |
| Object | Misdescriptions of the object itself including its attributes (color, text, count, etc.). | There are two blue cars, but described as “three red cars.” |
| Relation | Misdescriptions of spatial relationships between objects. | A cup is under the table, but described as “on the table.” |
| Location | Misdescriptions of an object’s position. | A cup is on the right side, but described as “on the left.” |
| Direction | Misdescriptions of orientation or direction of an object. | A person is looking to the right, but described as “looking to the left.” |
| Illusion | Mentioning non-existent objects, events, or actions. | No bird is present, but described as “a bird flying in the sky.” |
| Action | Incorrect or incomplete description of actions over time. | A person is walking, but described as “running.” |
Table 7: Word-level error categories in VidOmni-Bench. We group word-level errors into five static-frame-related categories(Object, Relation, Location, Direction, and Illusion) and five temporal-understanding-related categories(Action, Order, Causality, Frequency, and State Change).
Figure 14: Sentence-wise event verification instruction. The prompt asks Video-LLMs to assign a binary label to each caption sentence before providing sentence-level reasoning.
| Model | Provider | Open/Closed | FPS | Max Frames | Release |
| Gemini 3 Flash | Closed | 1 | - | 2026/02 | |
| GPT 5 mini | OpenAI | Closed | - | 256 | 2026/03 |
| Qwen3-VL 32B | Alibaba | Open | 2 | 2048 | 2026/01 |
| InternVL 3.5 38B | OpenGVLab | Open | 1 | 768 | 2025/12 |
| Video-SALMONN 2+ 72B | ByteDance | Open | 1 | 32 | 2025/11 |
| Keye-VL 1.5 8B | Kwai | Open | 1 | 1024 | 2025/03 |
Table 8: Details of Video-LLMs evaluated on VidOmni-Bench. We list the provider, accessibility, FPS, and maximum number of input frames (Max Frames) for each Video-LLM evaluated on VidOmni-Bench.
| Benchmark | Avg. Words | Info Count | Noun Chunks | Verbs | LLM Judge Win Rate (%) (vs. Ours) |
| ARGUS Rawal et al. (2025) | 477.5 | 336.4 | 134.3 | 63.1 | 4.0 |
| NOAH Lee et al. (2025) | 48.0 | 34.2 | 13.8 | 8.0 | 0.6 |
| CAPAbility Liu et al. (2026) | 3.2 | 2.4 | 1.1 | 0.3 | 0.4 |
| VidOmni-Bench | 578.9 | 400.6 | 159.3 | 82.1 | – |
Table 10: Comparison of caption informativeness across benchmarks. VidOmni-Bench achieves substantially longer captions, higher information counts, and more noun chunks and verbs than prior benchmarks. The last column reports the percentage of pairwise LLM-as-a-judge comparisons in which the baseline caption is preferred over the VidOmni-Bench caption in terms of informativeness.
Figure 15: Prompt for pairwise caption informativeness judgment. We use Prometheus-2 7B to compare captions from VidOmni-Bench and other benchmarks under a predefined informativeness rubric. Given the captioning instruction, two captions, and the rubric, the judge selects the more informative caption.
| Video-LLM | Ensemble Verifier | F1-Score (%) | |
| Self-Verification | Cross-Ensemble (+p.p) | ||
| Qwen3-VL 32B | + InternVL3.5 38B | 5.3 | 26.2 ( +20.9 ) |
| InternVL3.5 38B | + Keye-VL 1.5 8B | 7.7 | 24.6 ( +16.9 ) |
| Gemini 3 Flash | + GPT 5 mini | 21.9 | 25.1 ( +3.2 ) |
Table 11: Effect of cross-model ensemble on self-verification. Cross-model ensemble verification consistently improves F1-Score over self-verification across caption sources. These gains show that combining judgments from other Video-LLMs can compensate for the low verification performance observed when models evaluate their own captions.
| Video-LLM | Audio Category | F1-Score Gain (%) | Significant | |
| Gemini 3 Flash | Speech | +3.3 | +3.9 | |
| Sound Effect | +4.2 | +3.6 | ||
| Background | 0.8 | 0.9 | n.s. | |
| video-SALMONN 2+ 7B | Speech | 0.3 | 1.7 | n.s. |
| Sound Effect | 1.5 | 2.5 | ||
| Background | 0.9 | 2.7 | ** |
Table 13: F1-Score gain when audio is added by audio category and model. For each model and audio category, we compute the F1-Score gain (with audio - without audio) and test whether this gain is significantly different from zero using a one-sample -test. * , ** , *** , n.s.: not significant .
Figure 16: Performances of Video-LLMs across all complexity types and durations . The results reveal multidimensional and model-specific weaknesses, indicating that current Video-LLMs exhibit different failure patterns depending on both video dynamics and temporal scale.
Figure 17: Annotation interface for sentence-wise correctness (Stage 1) . Annotators label each caption sentence as Correct, Incorrect, or Unknown based on the video evidence. An additional memo field is provided to collect detailed notes and ambiguous cases during dataset construction.
Figure 18: Annotation interface for word-level error category (Stage 2). For each sentence labeled as Incorrect in Stage 1, annotators identify the erroneous words or phrases and assign them to predefined error categories, enabling fine-grained analysis of hallucination types.
More in this topic
Video-Language Model Evaluation
Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.
See all 103 papers