LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
Authors: Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, +8 more
Organizations: Zhejiang University · Westlake University · Ant Group · Shanghai Innovation Institute · Independent Researcher · Shanghai Jiao Tong University · University of California Merced
Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily focus on short audio and video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new benchmark designed specifically for the cross-modal comprehension of long-form audio and video. This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275 videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%. We anticipate that this dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex cross-modal understanding problems within long-form audio-visual contexts.
Figures & tables
Figure 1: We introduce LVOmniBench to provide a rigorous assessment of the performance of OmniLLMs on long audio-visual inputs, comprising strictly manually curated videos and annotations. Each question is assigned a specific difficulty level to facilitate hierarchical evaluation of model performance. In the first example, the model should comprehend the entire audio-visual context. Solving this question initially requires cross-modal alignment to identify “Toby” and finally necessitates visual counting and scene recognition. Even SoTA models, such as Gemini 3 Pro, struggle to answer this question correctly. The second example presents questions at two additional difficulty tiers, demonstrating that a correct answer requires a combination of audio and visual.
Benchmark
Modality
Open Domain
Video Duration
Avg. Length
Anno.
Multi-Type
Multi-D
Answer Type
AVQA [ 97 ]
V+A
✗
10
10
M
✗
✗
MC
Music-AVQA [ 44 ]
V+A
✗
60
60
M
✗
✗
CLS
AVTrustBench [ 16 ]
V+A
✓
10\60
-
A&M
✗
✗
MC
AV-Odyssey [ 33 ]
I+A
✓
-
-
M
✓
✗
MC
AVHBench [ 74 ]
V+A
✓
10
10
A&M
✓
✗
CLS
OmniBench [ 47 ]
I+A
✓
-
-
M
✓
✗
MC
Table 1: Comparison of different benchmarks. The modalities V, I, and A denote video, image, and audio, respectively. The open-domain metric indicates whether the video sources span a broad range of real-world scenarios. The annotation method (Anno.) specifies whether the dataset was curated manually (M) or automatically (A). We also report the video duration and the average temporal duration for each dataset, both measured in seconds. Multi-dimension (Multi-D) indicates whether the benchmark questions are hierarchically stratified by varying levels of cognitive difficulty, and multi-type indicates whether the problem involves multiple aspects.
Figure 2: The construction of LVOmniBench follows a rigorous pipeline encompassing video collection, filtering, and question annotation. To guarantee both high-fidelity data quality and sufficient challenge for OmniLLMs, every component, from raw videos to the final questions, underwent meticulous manual selection and annotation.
Figure 3: Distribution of videos. Left: Videos in LVOmniBench span five primary categories encompassing 21 fine-grained subcategories. Each video is selected to ensure sufficient audio-visual information and dynamic variations. Right: The durations of the collected videos range from 10 to 90 minutes, with most between 20 and 50 minutes.
Figure 4: Distribution of questions. Left: LVOmniBench comprises nine question subcategories, with each demonstrating a balanced distribution across difficulty levels and video durations. Right: This panel illustrates the distribution of question difficulty and the corresponding audio types required to answer the questions.
Video Information
Question Information
Total Video
275
Total Questions
1014
Categories
21
Question Type
9
Audio Types
Sp+So+Mu
Difficulties (Low:Med:High)
314:441:259
Duration
613–5482
Avg Question Length
16.4
Avg. Duration
34:29
Multiple-Choice
4 (A/B/C/D)
Table 2: Key statistics of our LVOmniBench benchmark.
Model
Modality
Difficulty
Understanding
Perception
Inference
Logical
Avg.
Low
Medium
High
Proprietary Omnimodal LLMs
Gemini-3.0-Pro
A + V
79.3
68.1
45.0
73.3
60.1
65.4
67.5
65.8
Gemini-3.0-Flash
A + V
76.6
63.0
31.0
67.7
54.7
60.6
51.8
59.0
Gemini-2.0-Flash
A + V
57.0
48.9
29.8
41.4
38.5
49.1
42.1
42.9
Gemini-3.0-Flash
V
55.6
49.3
30.6
51.4
42.9
42.3
48.5
46.2
Table 3: The performance of models on LVOmniBench. We evaluate OmniLLMs, utilizing both Video LLMs and Audio LLMs as specialized baselines. During inference, each model is evaluated using the maximum permissible number of input frames. The results demonstrate that long-form audio-visual comprehension remains a significant challenge for current SoTA models.
Figure 5: Comparison between proprietary models and open-source models on LVOmniBench across different tasks. The blue, red, orange, and gray in the outer circle stands for perception, logical, inference, and understanding tasks. As can be observed, proprietary models demonstrate a substantial performance advantage over their open-source counterparts. Furthermore, across all models, significant vulnerabilities remain in specific sub-tasks, notably Counting and Music Perception.
Figure 6: Results across various audio categories and difficulty levels. The findings demonstrate that proprietary models consistently outperform open-source models across all evaluated dimensions. Furthermore, OmniLLMs generally face greater challenges in perceiving musical audio; specifically, at high difficulty levels, the performance of open-source models declines to a level approaching random chance.
Model
Modality
AP
Counting
EU
HU
LR
MP
SI
SpI
TI
Avg.
Qwen3-Omni-30B
Audio +Video
27.9 37.0
23.1 35.0
24.4 33.1
26.2 33.0
28.2 39.1
36.1 34.8
35.8 51.9
26.4 31.8
31.7 43.1
28.1 35.8+7.7
Gemini-3.0-Flash
A + V
69.7
45.5
64.6
71.6
51.8
43.8
66.7
55.8
63.8
59.0
Table 4: The impact of audio and video information. (a) The evaluation of performance across various capabilities. The abbreviations are defined as follows: AP : Attribute Perception, EU : Event Understanding, HU : Human-Centric Understanding, LR : Logical Reasoning, MP : Music Perception, SI : Sound Inference, SpI : Spatial Inference, and TI : Temporal Inference. (b) Experiments are conducted across three input configurations: video-only, video with ASR, and video with original audio.
Figure 7: Error distribution of six types in 153 annotated Gemini 3 Flash errors of LVOmniBench. Perception errors were divided into audio and visual categories, accounting for 60% and 40%, respectively.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Cases of perception errors. We present examples of failures that arise from the misperception of underlying visual or audio information by models.
Figure 9: Cases of localization errors. It is observed that models are unable to accurately locate the temporal occurrences of events within long-form audio and video inputs, leading to incorrect predictions.
Figure 10: Cases of cross-modal semantic gaps and reasoning errors. For complex problems, models fail to fully exploit audio and video information; instead, they often over-rely on a single modality during reasoning, hindering the alignment of cross-modal features. Addressing this limitation represents a critical direction for future enhancements to OmniLLMs. Furthermore, models frequently exhibit errors in reasoning tasks, such as spatial localization.