OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

Xuyang Liu; Shihao Wang; Xinyu Wei; Minghan Li; Zeqing Wang; Zuhao Yang; Zichen Wen; Lei Zhang

cs.CVOct 13, 2026

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

Authors: Xuyang Liu, Shihao Wang, Xinyu Wei, Minghan Li, Zeqing Wang, Zuhao Yang, Zichen Wen, Lei Zhang

Organizations: The Hong Kong Polytechnic University · OPPO Research Institute · Colorado School of Mines · Nanyang Technological University · Shanghai Jiao Tong University

Abstract

With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, and three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.

Figures & tables

Figure 1: Overall model performance on OmniCompass-VL. Gemini-3.8-Flash and Kimi-K3 achieve the highest scores among proprietary and open-source models, respectively, at 42.8% and 39.6%.

Benchmarks

Target Specification

Video Evidence

Task Coverage

Video Duration

#Vid.

#Inst.

Anno.

Len.(s)

Text

Text+Image

Visual

Audio

Short

Med.

Long

Video Benchmarks

Charades-STA ( Gao et al., 2017 )

✓

✗

✓

✗

✓

✗

✓

✗

1,334

3,720

30.0

TVQA ( Lei et al., 2018 )

✓

✗

✓

✗

✓

✗

2,179

15,253

11.2

TVQA+ ( Lei et al., 2020 )

✓

✗

✓

✗

✓

✗

403

2,821

61.5

VidSTG ( Zhang et al., 2020 )

✓

✗

✓

✗

✓

✗

✓

✗

732

10,303

28.0

Table 1: Comparison of existing video understanding benchmarks. Target Specification indicates whether the target is specified using text alone ( i.e. , Text) or text with an image ( i.e. , Text+Image); Video Evidence indicates whether correct outputs depend on visual or native audio content. TG/SG/QA denote temporal grounding, spatial grounding, and question answering. Short/Med./Long denote video length of

<90

[90,300)

s, and

\geq 300

s. #Vid./#Inst. are video/instance counts; M/A/H denote manual/automatic/hybrid annotation; Len. is the average video length in seconds.

Figure 2: Overview of OmniCompass. The benchmark spans four dimensions: target specification, video evidence, task coverage, and video duration. Paired Text / Text+Image specifications are evaluated on the same targets. OmniCompass-VL contains 820 videos and 2,460 instances, while the 240-video OmniCompass-AV subset contains 720 instances with native audio. Both tracks cover temporal grounding, spatial grounding, and question answering.

Figure 3: Data construction pipeline. OmniCompass requires about 1,800 human-hours for rigorous annotation, verification, and revision. Stage 1: We screen 8,489 videos from nine public datasets, retain 6,946 after duration and referential-cue filtering, and select 580 to complement 240 self-collected videos. Stage 2: Four trained annotators independently construct paired Text and Text+Image TG, SG, and QA instances. AV cases include native audio. Stage 3: Annotators cross-verify and revise cases before final acceptance and correction by the lead expert, yielding 820 videos across seven domains, paired settings, two tracks, three tasks, and three duration groups.

Figure 4: Dataset statistics of OmniCompass. (a) 820 videos span 7 domains and 52 subcategories, each with at least ten videos. (b) Fifteen duration bins span short (30–90 s), medium (90–300 s), and long (

\geq 300

s) ranges. Stacked colors distinguish 580 VL-only and 240 shared VL–AV videos. (c) Bars report counts for eight OmniCompass-VL and five OmniCompass-AV QA types. Lighter and darker shades distinguish Perception and Reasoning. (d) Cumulative TG-span ratios show temporal selectivity. (e) Cumulative SG-area ratios show spatial granularity. Dots mark track means.

Models	Frames	Temporal Grounding			Spatial Grounding			Question Answering		Overall
Models	Frames	[email protected]	m tIoU	CS TG	[email protected]	m sIoU	CS SG	Accuracy	CS QA	Average	CS Avg
Proprietary MLLMs
Gemini-3.8-Flash	64	33.8 / 32.7	32.6 / 33.5	27.1	37.5 / 41.6	37.7/ 41.8	32.3	73.3/ 76.6	69.1	47.9 / 50.6	42.8
Gemini-3.1-Pro	64	24.9/24.9	25.6/26.6	19.3	34.9/35.2	36.5/37.3	31.8	71.4/73.6	65.7	44.5/45.9	38.9
Seed-2.1-Pro-260915	64	27.9/29.8	29.3/29.9	23.7	40.9 / 38.9	40.6 / 39.6	34.9	73.4/ 74.3	67.4	47.8 / 47.9	42.0
Seed-2.1-Turbo-260628	64	29.9/29.4	30.5/30.2	25.0	35.4/35.4	36.3/36.3	30.0	74.1 /74.1	67.7	47.0/46.9	40.9

Table 2: Results on OmniCompass-VL. We evaluate TG, SG, and QA without audio using uniformly sampled, timestamped frames. Performance is reported as Text / Text+Image , while CS requires paired predictions to be correct and consistent. Average is the mean of TG m tIoU , SG m sIoU , and QA Accuracy per specification, while CS Avg averages the three CS scores. Within each group, the best and second-best results are bold and underlined , respectively.

Models	Frames	Temporal Grounding			Spatial Grounding			Question Answering		Overall
Models	Frames	[email protected]	m tIoU	CS TG	[email protected]	m sIoU	CS SG	Accuracy	CS QA	Average	CS Avg
Proprietary Omni-modal MLLMs
Gemini-3.8-Flash	64	46.9/48.1	46.4/45.9	40.6	62.8/63.2	59.8/59.3	53.5	80.3/80.8	78.2	62.2/62.0	57.4
Gemini-3.1-Pro	64	15.1/17.6	16.6/17.7	9.0	54.8/56.1	52.0/52.1	45.8	72.4/71.1	64.9	47.0/47.0	39.9
Qwen3.8-Omni-Flash	64	26.3/28.5	29.2/30.9	21.8	60.1/59.5	56.8/56.0	50.9	75.9/78.9	73.2	54.0/55.3	48.6
Qwen3.5-Omni-Plus	64	18.0/14.9	21.0/21.1	14.0	61.8/60.8	55.5/55.6	50.5	71.1/73.2	68.4	49.2/50.0	44.3

Table 3: Results on the OmniCompass-AV. Models are evaluated on TG, SG, and QA using uniformly sampled, timestamped frames with synchronized audio.

Figure 5: Fine-grained QA performance across tracks. We report category-wise

\mathrm{CS}^{\mathrm{QA}}

scores in percent.

Figure 6: Impact of frame-level timestamps. Columns report

\mathrm{CS}^{\mathrm{TG}}

\mathrm{CS}^{\mathrm{SG}}

, and

\mathrm{CS}^{\mathrm{QA}}

in percent. Dashed links connect scores without and with timestamps. Axis ranges vary by panel.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Performance across video durations in OmniCompass-VL. Bars show task-specific CS in percent for short videos below 90 seconds, medium videos from 90 to below 300 seconds, and long videos of at least 300 seconds. Each model uses its own eligible pairs.

Figure A2 : Grounding across target granularity. Curves report

\mathrm{CS}^{\mathrm{TG}}

and

\mathrm{CS}^{\mathrm{SG}}

in percent across quartiles of target-to-video duration and mean endpoint box-to-frame area. Quartiles are track-specific; models share eligible pairs within each track and task.

Figure A3 : Performance across video domains in OmniCompass-VL. Bars report task-specific CS in percent for ten models. The shared eligible subsets contain 592 TG, 673 SG, and 689 QA pairs across the seven domains; comparisons use the same instances within each task.

Figure B1 : Impact of frame scaling. Each panel reports task-level paired CS (%) under 8, 16, 32, 64, and 128 uniformly sampled frames. Rows correspond to the VL and AV tracks, columns correspond to TG, SG, and QA, and curves distinguish the evaluated models.

Figure B2 : Effects of reference specification in OmniCompass-VL. Cells show Text+Image minus Text in percent for TG m tIoU , SG m sIoU , and QA accuracy on shared per-task subsets. Green denotes increases and blue denotes decreases; color does not encode significance.

Figure B3 : Consistency under paired reference specifications. Grounding compares CS with the lower setting-specific mean on identical eligible pairs. QA bars partition pairs by correctness in the two settings; blue denotes

\mathrm{CS}^{\mathrm{QA}}

. The right column reports correctness under exactly one setting. Bold indicates category maxima, not uniformly better performance.

Figure B4 : Qualitative Analysis on our OmniCompass-VL and OmniCompass-AV. Different models predictions on six cases in our two evaluation tracks.

Figure C1 : Seven high-level domains of videos in OmniCompass.

Source	Videos	Selected domains	License / Terms	Link
MultiSports ( Li et al., 2021 )	83	Sports	CC BY-NC 4.0	https://huggingface.co/datasets/MCG-NJU/SportsAction
LiveSports-3K ( Chen et al., 2025b )	92	Sports	Not specified	https://huggingface.co/datasets/stdKonjac/LiveSports-3K
UCF-Crime ( Sultani et al., 2018 )	111	Safety	CC0-1.0 on mirror	https://huggingface.co/datasets/AllenXeon/ucf_crime
CG-Bench ( Chen et al., 2025a )	58	Film, Variety, News, Life, Safety, Traffic	MIT; videos for academic research only	https://huggingface.co/datasets/CG-Bench/CG-Bench
MovieChat-1K-test ( Song et al., 2024 )	20	News	Not specified	https://huggingface.co/datasets/Enxin/MovieChat-1K-test
ActivityNetQA ( Yu et al., 2019 )	37	Life	MIT; non-commercial use	https://huggingface.co/datasets/lmms-lab/ActivityNetQA

Appendix

Table C1 : Nine public source datasets contributing to OmniCompass. Counts refer to selected videos, not complete source releases. Domains follow the final OmniCompass taxonomy; News includes documentary, nature, and knowledge content. Links point to downloads.

Panel of Figure 19 — Figure C2 : Structured video profiling prompt used with Qwen3-VL-8B-Instruct .

Component	Derived profiling fields	Computation	Intuition
Reference Necessity ( $R$ )	entity_count , action_count , attribute_count , scene_count , repeated actions	We first compute ambiguity and richness . Ambiguity increases with multiple candidate entities, repeated actions, scene variation, and potential attribute confusion. Richness increases with broader coverage over entity, action, attribute, and scene signals. We then combine them as $R=10\cdot\left(0.60A+0.20B+0.20AB\right),$ where $A$ and $B$ are the normalized ambiguity and richness scores.	Measures whether external referential cues are needed to uniquely specify the target.
Grounding Suitability ( $G$ )	entity_count , action_count , attribute_count , scene_count	We estimate temporal grounding potential from action and scene counts, spatial grounding potential from entity and attribute counts, and question answering potential from action and scene-related complexity. The final score is $G=10\cdot\mathrm{avg}(P_{\text{TG}},P_{\text{SG}},P_{\text{QA}}).$	Measures whether the video can support constructing TG, SG, and VideoQA instances.
Temporal-Spatial Utility ( $T$ )	video duration, entity_count , action_count , scene_count	We combine a duration term and a variation term: $T=10\cdot\left(0.60D+0.40V\right),$ where $D$ is a log-normalized duration score and $V$ is the average of normalized action, scene, and entity variation.	Measures whether the video provides sufficient temporal span and visual variation for evidence localization.
Evidence Diversity ( $E$ )	entity_count , action_count , attribute_count , scene_count	We compute normalized entropy over the four evidence counts: $E=10\cdot H_{\text{norm}}(c_{\text{ent}},c_{\text{act}},c_{\text{att}},c_{\text{scn}}).$	Measures whether referential evidence is distributed across multiple cue types rather than concentrated in a single source.
Final Score ( $S$ )	$R,G,T,E$	$S=0.35R+0.25G+0.25T+0.15E$	Encourages videos that require diverse referential cues, support TG/SG/VideoQA construction, and provide diverse evidence for annotation.

Appendix

Table C2 : Mapping from structured video profiling outputs to task-oriented scoring signals. We summarize how the structured video profiling outputs produced by Qwen3-VL-8B-Instruct are converted into intermediate statistics and then used to compute the four score components.

Figure C3 : QA taxonomy in our OmniCompass-VL and OmniCompass-AV.

Category	QA Task	Task Definition	Example
Perception	Fact Recognition	Identify entities or explicit facts associated with the referenced target.	In the video, when the man in a black suit sitting on the left side of the table is talking to two other people, what is the drink in front of the woman?
	Action Recognition	Identify an action performed by the referenced target.	In the video, what farewell gesture did the young man in the pink hoodie make to each of the two women inside the house before leaving?
	Attribute Perception	Identify the target’s appearance, attributes, or observable state.	In the video, when a lady wearing a red-brimmed straw hat dances with a group of people in yellow clothes, what is the color of the oil-paper umbrella held by the lady behind her?
	Visual Counting	Count referenced entities, actions, or event occurrences.	In the video, how many rotations did the athlete complete while raising one hand and high-fiving a partner?
	Spatial Perception	Determine scene context, locations, or spatial relations involving the referenced target.	In the video, where is the kettle relative to the man in a gray hat and light blue short-sleeved shirt, based on the direction he is facing while making tea?
Reasoning	Temporal Reasoning	Identify events or states before, during, or after a specific referenced event.	Before the man in a blue shirt enters the first elevator, what type of passenger is among the previous group entering the same elevator?

Appendix

Table C3 : QA taxonomy and examples in OmniCompass-VL. We group the eight QA types into Perception and Reasoning. Examples show only the question text from the Text target-specification setting; answer options are omitted.

Category	QA Task	Task Definition	Example
Perception	Speech Understanding	Understand explicit speech content, including facts, quantities, descriptions, and opinions.	How does the gray-haired man in a black leather jacket and white top summarize what the Nike swoosh represented to young people at the time?
	Event Understanding	Identify participants, objects, actions, states, or counts in audio-visual events.	When the woman with tied-up hair, a blue off-the-shoulder top, and brown overalls introduces Sean’s recently deceased father, which pair of objects appears beside the relevant framed photograph?
	Emotion Recognition	Recognize emotion, tone, attitude, or affective reactions from audio-visual evidence.	In the video, after repeated corrections from the man in red, how does the woman with shoulder-length auburn hair sound when asking whether he wants to tell the story?
Reasoning	Temporal Reasoning	Determine the order or temporal relations of utterances, events, and states.	In the video, before the blonde woman calls the grey bearded man arrogant, what is he doing?
	Causal Reasoning	Infer causes, motives, or consequences from audio-visual context.	Why could the lie told by the woman in a red choker and black top to avoid awkwardness with Joey end up costing her job?

Appendix

Table C4 : QA taxonomy and examples in OmniCompass-AV. We group the five QA types into Perception and Reasoning. Examples show only the question text from the Text target-specification setting; answer options are omitted.

Reference Type	Role in Target Identification and Typical Examples
Entity	Shows a particular person or object that the model must match across views, such as an athlete, actor, vehicle, or bag.
Action	Shows a recognizable motion or event pattern that defines the target behavior, such as a dunk, basketball shot, header, or tackle.
Attribute	Distinguishes the target through a visible property, appearance, or state, such as clothing color, vehicle type, shape, or pose.
Scene	Shows a place or surrounding visual context to match with a video segment, such as a street, room, stage, or stadium.

Appendix

Table C5 : Functional taxonomy of reference information in OmniCompass-VL. Each type states what the reference image contributes to target identification and gives representative examples.

Figure C4 : Reference composition across 2,460 VL task instances. Panel (a) shows overall cue-type frequencies, (b) shows their proportions by domain, and (c) compares video-internal and Internet sources. Counts are reported per task instance, so an image reused across tasks is counted more than once.

Panel of Figure 26 — Figure E1 : Temporal Grounding prompts for OmniCompass-VL. Both settings receive the same uniformly sampled, timestamped video frames without audio and predict the same event interval. Only the target specification changes from a textual description to a reference image.

Panel of Figure 27 — Figure E2 : Spatial Grounding prompts for OmniCompass-VL. Both settings receive the same uniformly sampled, timestamped video frames and the same temporal interval. They localize the same target at the two interval endpoints; only the target specification changes.

Panel of Figure 28 — Figure E3 : Question Answering prompts for OmniCompass-VL. Both settings receive the same uniformly sampled, timestamped video frames and retain the same answer choices and ground truth. Only the target specification changes.

Panel of Figure 29 — Figure E4 : Temporal Grounding prompts for OmniCompass-AV. Both settings receive the same timestamped video frames and corresponding audio, and predict the same audio-visual event interval. Only the target specification changes.

Panel of Figure 30 — Figure E5 : Spatial Grounding prompts for OmniCompass-AV. Both settings receive the same timestamped video frames, corresponding audio, and temporal interval. They localize the same target at the two interval endpoints; only the target specification changes.

Panel of Figure 31 — Figure E6 : Question Answering prompts for OmniCompass-AV. Both settings receive the same timestamped video frames, corresponding audio, and answer candidates. The correct answer is unchanged across target specifications.

Explore similar work

Mar 14, 2026

MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 20,000 carefully curated questions paired with 11877 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break.

Arushi Goel, Sreyan Ghosh, Vatsal Agarwal +16

NVIDIA, USA · University of Maryland, College Park, USA

Multimodal Understanding Multimodal Reasoning Long-Video Understanding Multimodal Model Evaluation Audio-Visual Understanding Multimodal QA

Ask PDF

Oct 8, 2026

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.

Zhongyu Yang, Jiale Tao, Ruitao Chen +9

Hunyuan, Tencent · Nanyang Technological University · Northumbria University

Multimodal Model Evaluation Audio-Visual Understanding

Ask PDF

Jun 1, 2026

AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

Recent advances in Omni-Multimodal Large Language Models (Omni-MLLMs) have enabled strong integration of vision, audio, and language. However, their audio-visual intelligence (AVI) remains insufficiently evaluated due to the lack of systematic and comprehensive benchmarks. We introduce AVI-Bench, a cognitively inspired benchmark that evaluates Omni-MLLMs across three stages, perception, understanding, and reasoning, through cross-modal tasks requiring joint audio-visual interpretation. This design enables fine-grained diagnosis of model capabilities and failure modes. To further assess robustness beyond familiar domains, we propose AVI-Bench-PriSe, an extension that probes models' primitive audio-visual sensation using unfamiliar, low-semantic stimuli, testing generalization beyond common training distributions. Extensive experiments on both open-source and closed-source models reveal substantial limitations in current Omni-MLLMs. Based on these findings, we present a four-level AVI taxonomy. Overall, AVI-Bench provides a principled evaluation framework to guide the development of more robust and generalizable AVI. Project website: https://fudancvl.github.io/AVI-Bench/

Yaoting Wang, Ziyi Zhang, Wenming Tu +10

Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Huazhong University of Science and Technology · Shanghai Jiaotong University +5

Audio-Visual Understanding Multimodal Robustness Multimodal Large Language Models Audio Understanding Multimodal Model Evaluation Multimodal Understanding

Ask PDF