CineMME: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding
Authors: Jin Liu, Dawei Du, Yexiang Liu, Sijie Zhu, Fan Chen, Junxian Duan, Huaibo Huang, Ran He, Longyin Wen, Zhenfang Chen
Organizations: School of Information Science and Technology, ShanghaiTech University, Shanghai, China · MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · ByteDance, San Jose, USA. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Narrative video understanding requires more than recognizing visible events: models must extract evidence from dialogue, bind it to speakers, and use it to infer narrative meaning. Existing video benchmarks evaluate aspects of this problem, including video question answering, audio-visual reasoning, and temporal grounding, but rarely test whether these abilities form a coherent dialogue-to-plot evidence chain within the same clips. We introduce CineMME, a human-verified benchmark designed to diagnose dialogue-grounded plot understanding, supported by approximately 1,600 expert person-hours of annotation. CineMME contains 975 curated short narrative video clips and two linked tracks: CineMME-Grounding, with 13,554 timestamped speech segments and 33,083 speaker face boxes for temporal dialogue localization, transcription, and active-speaker grounding; and CineMME-Reasoning, with 1,600 multiple-choice and 316 open-ended questions spanning perception, audio, narrative, social, and scene-presentation understanding. Our main evaluation compares 20 MLLMs on reasoning and 9 omni-modal models on grounding. The results reveal a pronounced who-what-when decoupling: Gemini-3 Pro localizes dialogue in time (68.34% tIoU) and transcribes it with a WER of 28.99%, yet achieves only 7.74% vIoU for active-speaker grounding while achieving 67.66% MCQ accuracy on questions with extractable answers. Modality ablations further show that vision is not uniformly beneficial: it improves performance on perception-heavy questions but can hurt dialogue-centric reasoning. CineMME provides a compact diagnostic testbed for evaluating how multimodal video models connect dialogue, speakers, visual evidence, and narrative meaning.
Figures & tables
Reasoning Q&A
Free-Form Perception
Benchmark
Videos
Q&A
Anno.
Modality
MCQ
Open
Spatial- Grounding
Temporal- Grounding
Speech
Visual Benchmarks
Video-MME ( Fu et al., 2025 )
900
2.7K
M
V
✓
✗
✗
✗
✗
MLVU ( Zhou et al., 2025a )
1,730
3.1K
A+M
V
✓
✓
✗
✗
✗
Omni Benchmarks
LongVALE ( Geng et al., 2025 )
8.4K
–
A
V+A
✗
✗
✗
✓
✓
Table 1: Closest benchmark comparison. We keep the same comparison dimensions as the full benchmark table: dataset scale, annotation type, modality, reasoning formats, and free-form perception capabilities. The main paper reports representative benchmarks from four closely related categories; a broader comparison is provided in Appendix 25 . “Anno.” indicates annotation type (A: automatic, M: manual); in modality, “A” denotes audio, “V” video, and “I” image.
Figure 2: Annotation pipeline for CineMME. We transform raw short narrative clips into a rigorous benchmark through a multi-stage process. Raw data undergoes preprocessing before entering parallel tracks for speaker diarization and QA generation. Crucially, we employ an Adversarial QA Refinement strategy to enhance question difficulty, followed by a labor-intensive human verification process to guarantee annotation quality.
Figure 3: Dataset statistics highlighting the diagnostic characteristics of CineMME. (a) Short speech segments require models to localize dialogue with high temporal precision. (b) Tiny, small, and medium speaker face boxes make speaker-level grounding challenging, especially in multi-person narrative scenes. (c) Questions are balanced across five reasoning dimensions, supporting broad evaluation beyond a single capability. (d) A large fraction of questions are medium or long, with 76.1% exceeding 20 words, encouraging semantic understanding beyond keyword matching.
Figure 4: Frequent question keywords center on reasoning, sound, editing, and narrative concepts.
Models
LLM Params
Frame Rate
MCQs Accuracy (%)
OEQs
Narrative
Percept.
Scene
Social
Audio
Avg.
Open-source / open-weight MLLMs
Vision-Only
Phi-4-multimodal Microsoft et al. (2025)
5.6B
64 FPV
17.65
20.15
28.74
24.34
27.92
24.22
28.63
Kimi-VL Team et al. (2025b)
16B
128 FPV
18.04
20.90
20.92
23.87
27.83
22.56
37.82
InternVL3.5 Wang et al. (2025b)
38B
32 FPV
27.45
28.73
36.39
27.68
33.66
30.88
29.46
Table 2: Main Results on CineMME (Reasoning Track). We report MCQ accuracy (%) and OEQ scores (0–100). FPV : Frames Per Video; FPS : Frames Per Second. Best scores within each modality/model group are in bold . MiMo-v2.5 is evaluated via API. Scores use valid answers; effective sample counts are in Table 21 .
Model
Params
tIoU ↑
vIoU ↑
WER ↓
Open-source / open-weight omni-modal MLLMs
MiniCPM-o-2.6 Yao et al. (2024)
8B
34.32
0.24
96.76
OmniVinci Ye et al. (2025)
9B
5.13
0.09
98.91
Phi-4-multimodal Microsoft et al. (2025)
5.6B
16.96
0.09
98.22
Qwen2.5-Omni Xu et al. (2025a)
7B
24.72
0.38
93.31
Qwen3-Omni Xu et al. (2025b)
30B
49.25
3.18
75.79
Table 3: Zero-shot grounding results on CineMME-Grounding . All models predict transcripts, timestamps, and active-speaker face boxes. Scores are percentages; higher tIoU/vIoU and lower WER are better. MiMo-v2.5 is open-weight and evaluated via API.
Model
Input
CineMME-Reasoning MCQ Accuracy (%)
CineMME-Grounding
Narrative
Perception
Scene
Social
Audio
tIoU
vIoU
WER ↓
Gemini-3-Pro
A
69.02
67.54
63.61
70.88
74.43
68.64
3.46
27.42
Gemini-3-Pro
A+V
65.74
79.39
59.25
68.04
68.21
68.34
7.74
28.99
Qwen3-Omni
A
35.29
41.79
33.81
36.52
42.72
63.58
1.92
36.77
Qwen3-Omni
A+V (1 FPS)
30.98
47.57
37.54
37.47
38.51
49.25
3.18
75.79
Qwen3-Omni
A+V (256 FPV)
34.90
44.57
36.10
35.56
41.75
50.40
3.01
71.52
Table 4: Audio-only vs. audio+video ablation on Gemini-3 Pro Google (2025) and modality and visual sampling ablations on Qwen3-Omni Xu et al. (2025b) . Effective sample counts are in Table 22 .
Model
Split metric
Better
Worse
Gap
Gemini-3 Pro
tIoU ↑
51.55
44.71
+6.84
vIoU ↑
47.68
49.86
−2.17
WER ↓
52.43
43.43
+9.00
Qwen3-Omni
tIoU ↑
31.63
28.73
+2.91
vIoU ↑
27.71
32.59
−4.88
WER ↓
31.15
29.04
+2.12
Table 5: Grounding quality and open-ended reasoning on overlapping clips. Better and Worse denote the two halves ranked by each grounding metric. OEQ scores are on a 0–100 scale; grouping details are in Appendix G.5 .
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Adv. Round
Qwen2.5-VL
Qwen3-VL
GLM-4.1V
Initial
68.24
71.24
57.08
Round 1
46.78
44.64
35.19
Round 2
20.17
24.89
11.16
Appendix
Table 6: Adversarial refinement results. We report the zero-shot accuracy of three representative MLLMs across iterative optimization rounds. The decrease in accuracy summarizes the change in task difficulty for these models.
Stage
Scope
Avg. time / unit
Person-hours
Grounding
546 videos
∼ 1.56 h
∼ 850
Reasoning
667 videos
∼ 0.90 h
∼ 600
Final audit
975 clips
∼ 0.15 h
∼ 150
Total
975 clips
∼ 1.64 h
∼ 1,600
Appendix
Table 7: Annotation effort estimated from platform logs. Track averages are computed over videos in each track; the total average is computed over unique clips.
Item
N
Agreement
Cohen’s κ
Final flag rate
ASR text
21,771
95.01%
0.8998
51.52% non-Yes
Timestamp
21,833
94.56%
0.8910
55.25% non-Yes
Track ID
21,742
97.69%
0.9415
32.18% non-Yes
Discard decision
23,603
98.10%
0.8895
10.19% discard
Appendix
Table 8: Verification-decision agreement between annotation and review. N counts paired, non-missing decisions for each field. Final flag rates use all 23,603 candidate speech segments in the final logs, as detailed below.
Comparison
Pearson
Spearman
QWK
MAE
± 1
Human 1 vs. Human 2
0.752
0.775
0.745
1.15
71.8%
LLM judge vs. human mean
0.758
0.758
0.721
1.28
58.2%
LLM judge vs. Human 1
0.753
0.756
0.745
1.20
63.6%
LLM judge vs. Human 2
0.659
0.673
0.638
1.55
51.8%
Appendix
Table 9: Human calibration on 110 matched responses. Human mean averages two scores. QWK uses quadratic weights and rounds scores to the nearest integer.
Figure 5: Human calibration of the OEQ judge. Left: LLM judge scores against the average of two human expert scores. Right: agreement between the two human experts. Marker size indicates repeated score pairs; the dashed diagonal shows exact agreement.
Figure 6: Consistency check of LLM-as-a-Judge. OEQ scores assigned by Qwen2.5-72B-Instruct and Qwen3-30B-A3B show strong agreement (Pearson r=0.808 ).
Category
What It Measures
Representative Question Coverage
Perception
Extraction of objective audiovisual facts under visual complexity or short temporal windows.
Questions may ask about subtle visual details, dynamic counts, spoken content, speaker identity, or where and when an event occurs.
Audio
Interpretation of spoken and non-speech acoustic evidence, including subtle, masked, or off-screen cues.
Questions may focus on dialogue, vocal delivery, background sounds, subjective sound effects, or off-screen events that cannot be answered from frames alone.
Narrative
Reasoning about story structure, causal dependencies, chronology, and information asymmetry.
Questions may examine causal chains, temporal order, narrative structure, privileged information, or how one event changes the interpretation of another.
Social
Inference over intentions, beliefs, relationships, cultural context, and behavior that contradicts generic priors.
Questions may ask models to infer goals, false beliefs, implicit relationships, culturally grounded meanings, or evidence that overrides a generic social prior.
Scene Presentation
Understanding how visual composition and audio-visual presentation cues convey narrative, psychological, or thematic meaning.
Questions may connect framing, editing rhythm, sound design, lighting, props, or scene layout to their narrative or emotional effects.
Appendix
Table 10: Category scope of CineMME-Reasoning. The main text uses the five concise category names; this table clarifies the scope of each category with representative question coverage.
Category
Human (%)
Gemini-3 Pro (%)
Difference
Audio
75.70
68.21
+7.49
Social
78.26
68.04
+10.22
Narrative
76.47
65.74
+10.73
Perception
89.19
79.39
+9.80
Scene
66.67
59.25
+7.42
Overall
77.64
67.66
+9.98
Appendix
Table 11: Human MCQ performance on 161 questions answered by six volunteers. Gemini-3 Pro scores on the full benchmark are from Table 2 . Differences are percentage-point gaps.
Input
Audio
Social
Narrative
Perception
Scene
Overall
Question/options only
39.16
32.46
31.76
30.22
32.09
33.19
Vision only
51.46
42.00
36.47
47.01
44.99
44.44
Audio only
57.93
47.97
48.63
58.21
42.69
50.56
Full A/V
57.28
53.94
48.63
64.18
51.58
54.94
Vision + shuffled audio
49.19
38.90
40.39
43.45
45.56
43.34
Appendix
Table 12: MiMo-v2.5 MCQ input controls. Accuracy (%) is averaged over 1,600 scored responses per condition, except shuffled audio (1,599). Question/options only supplies no media; shuffled audio pairs the target video with an unrelated audio track.
Input
Audio
Social
Narrative
Perception
Scene
Overall
N
Question only
31.21
32.06
20.00
14.69
41.86
27.09
313
Vision only
31.36
46.47
33.33
26.22
58.57
38.33
311
Audio only
38.79
48.53
37.50
26.53
46.57
38.17
311
Full A/V
41.06
59.12
35.62
32.35
57.86
43.75
312
Vision + shuffled audio
34.09
38.24
34.79
28.78
51.43
37.07
314
Appendix
Table 13: Gemini-3 Pro OEQ input controls, scored by NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with the reference-based rubric and rescaled to 0–100. N counts valid judge scores out of 316 questions.
Model
Input
Audio
Social
Narrative
Perception
Scene
Overall
MiMo-v2.5
Q/options only
39.16
32.46
31.76
30.22
32.09
33.19
Correct A/V
57.28
53.94
48.63
64.18
51.58
54.94
Gain
+18.12
+21.48
+16.86
+33.96
+19.48
+21.75
MiniCPM-o-4.5
Q/options only
28.48
20.53
18.43
13.43
28.08
22.19
Correct A/V
33.33
27.92
31.37
37.69
29.51
31.50
Gain
+4.85
+7.40
+12.94
+24.25
+1.43
+9.31
Appendix
Table 14: Question-only and correct-clip MCQ results from the supplementary control runs (1,600 scored responses per condition). Values are accuracy (%); gains are percentage points computed before rounding.
Model
Media
MCQ accuracy (%)
OEQ score (0–100)
Q
Correct
Same
Different
Correct
Same
Different
MiniCPM-o-4.5
A/V
22.19
31.50
22.78
22.45
28.79
23.31
18.76
Qwen3-Omni
A/V
26.88
45.25
32.00
29.39
28.15
16.12
13.14
Qwen3-VL-30B-A3B
Vision
28.00
35.62
32.97
30.37
30.35
21.01
19.76
Appendix
Table 15: Source-recall controls. Q denotes question/options only; Correct is the target clip; Same and Different are wrong clips from the same and a different source narrative. Q and Correct use the full benchmark; the two wrong-clip conditions share 922 MCQs and 170 OEQs. OEQ uses the Nemotron judge specified in Table 13 .
Task Type
Question
Option
Answer
Narrative
25.6
23.7
19.0
Perception
24.5
13.2
10.1
Scene
26.6
28.3
22.3
Social
24.1
23.8
16.4
Audio
27.3
25.0
19.2
Overall
25.6
23.2
17.5
Appendix
Table 16: Mean whitespace-separated word counts, excluding option labels. Questions and gold answers cover all 1,916 MCQ/OEQ records; options cover all 9,600 MCQ options. Overall means weight individual records or options equally.
Track
N
Min
Median
Mean
P90
Max
MCQ
1600
52.1
151.1
156.0
205.2
299.1
OEQ
316
22.1
148.1
148.0
190.2
274.1
Grounding
546
22.1
147.1
148.0
189.1
300.1
Pooled task records
2462
22.1
151.1
153.2
202.1
300.1
Appendix
Table 17: Clip durations in seconds. Each MCQ or OEQ contributes one observation, so clips with multiple questions occur repeatedly. Grounding contributes one observation per video. The pooled row combines task records rather than 975 unique clips. P90 is the 90th percentile.
Position
A
B
C
D
E
F
Count
275
252
267
296
276
234
Share (%)
17.19
15.75
16.69
18.50
17.25
14.63
Appendix
Table 18: Correct-answer positions in the 1,600-question MCQ set. A chi-square goodness-of-fit test against uniform positions gives χ2(5)=8.623 , p=0.125 .
Track
Audio
Social
Narrative
Perception
Scene
Total
Retained
MCQ
300/309
417/419
249/255
263/268
346/349
1575/1600
98.44%
OEQ
64/66
33/34
45/48
94/98
66/70
302/316
95.57%
Grounding
–
–
–
–
–
513/546
93.96%
Appendix
Table 19: Benchmark items retained after excluding the 50 sources that failed the July 2026 availability audit. Entries are retained/original counts. MCQ and OEQ counts refer to questions; Grounding counts refer to videos.
Model
Scored questions
Accuracy
95% CI
Phi-4-multimodal
1598
24.22
[22.15, 26.35]
Kimi-VL
1600
22.56
[20.56, 24.62]
InternVL3.5
1600
30.88
[28.62, 33.12]
Ministral-3
1600
22.69
[20.62, 24.69]
MiMo-VL
1600
27.88
[25.69, 30.12]
GLM-4.1V
1600
23.25
[21.12, 25.37]
Appendix
Table 20: 95% percentile bootstrap confidence intervals for MCQ accuracy, computed from 10,000 resamples of each model’s scored questions.
Model
nMCQ
nOEQ
nG
nv
Phi-4-multimodal
1598
315
79
79
Kimi-VL
1600
316
–
–
InternVL3.5
1600
316
–
–
Ministral-3
1600
316
–
–
MiMo-VL
1600
315
–
–
GLM-4.1V
1600
316
–
–
Appendix
Table 21: Effective sample counts for the main results. MCQ and OEQ counts are scored questions out of 1,600 and 316; nG is parsed videos out of 546; nv is parsed videos with annotated face boxes out of 540. – denotes an unavailable count or unevaluated task.
Input
nMCQ
nG
nv
Gemini-3 Pro
A only
1600
541
535
A+V
1574
546
540
Qwen3-Omni
A+V (1 FPS)
1599
433
429
A+V (256 FPV)
1599
440
436
Appendix
Table 22: Effective sample counts for Gemini-3 Pro and Qwen3-Omni modality and sampling ablations. Counts use the same populations as Table 21 .
Visual
Audio
Frames
CineMME-Reasoning
CineMME-Grounding
Avg.
Narr.
Perc.
Scene
Soci.
Audi.
tIoU
vIoU
WER ↓
✓
✓
1 FPS
38.34
30.98
47.57
37.54
37.47
38.51
49.25
3.18
75.79
✓
✓
256
38.27
34.90
44.57
36.10
35.56
41.75
50.40
3.01
71.52
✓
✓
128
39.40
36.47
44.57
37.54
36.99
42.72
48.20
3.49
74.93
✓
✓
64
37.31
32.55
44.03
34.38
35.56
41.10
50.48
3.67
75.92
✓
✓
32
38.78
32.68
46.62
34.96
38.04
42.39
52.91
3.63
76.42
Appendix
Table 23: Ablation Study on Visual Sampling Density and Modality. We evaluate Qwen3-Omni with varying frame counts and audio/video inputs. Effective sample counts are in Table 22 .
Model
Split metric
MCQs / videos
Better
Worse
Gap
Gemini-3 Pro
tIoU ↑
166 / 144
73.81
78.05
−4.24
vIoU ↑
164 / 142
75.32
75.86
−0.54
WER ↓
166 / 144
76.47
75.31
+1.16
Qwen3-Omni
tIoU ↑
141 / 121
33.33
43.94
−10.61
vIoU ↑
140 / 120
32.86
44.29
−11.43
WER ↓
141 / 121
28.38
49.25
−20.88
Appendix
Table 24: Grounding quality and MCQ accuracy on overlapping clips. Better and Worse denote the two halves ranked by each grounding metric, using the same procedure as Table 5 .
Reasoning Q&A
Free-Form Perception
Benchmark
# Videos
# Q&A
Anno.
Modality
MCQ
Open
Spatial- Grounding
Temporal- Grounding
Speech
Visual Benchmarks:
MV-Bench ( Li et al., 2024 )
3,641
4.0 K
A+M
V
✓
✗
✗
✗
✗
EgoSchema ( Mangalam et al., 2023 )
5,063
5.0 K
A+M
V
✓
✗
✗
✗
✗
MLVU ( Zhou et al., 2025a )
1,730
3.1 K
A+M
V
✓
✓
✗
✗
✗
Video-MME ( Fu et al., 2025 )
900
2.7 K
M
V
✓
✗
✗
✗
✗
Appendix
Table 25: Comparison between CineMME and existing video understanding benchmarks. CineMME jointly supports reasoning, speech understanding, temporal grounding, and spatial grounding in the narrative-video domain. “A” means audio, “V” means video, “I” means image. “Anno.” is the annotation type (A: automatic, M: manual).
Models
Organization
Release
Version
Video Support
Audio Support
Inference Deployment
Open-source / open-weight MLLMs
Phi-4-multimodal Microsoft et al. (2025)
Microsoft
2025-3
Phi-4-multimodal-instruct
✗
✓
vLLM
Kimi-VL Team et al. (2025b)
Moonshot
2025-4
Kimi-VL-A3B-Thinking
✗
✗
vLLM
InternVL3.5 Wang et al. (2025b)
OpenGVLab
2025-8
InternVL3_5-38B-Instruct
✗
✗
vLLM
Ministral-3 Liu et al. (2026)
Mistral
2025-12
Ministral-3-14B-Reasoning-2512
✗
✗
vLLM
MiMo-VL Li et al. (2025b)
Xiaomi
2025-6
MiMo-VL-7B-RL-2508
✓
✗
vLLM
Appendix
Table 26: Model identifiers and inference backends. Frame-sampling settings are in Table 2 . HF denotes Hugging Face.
Figure 7: An example of an MCQ in the Audio category of a CineMME-Reasoning shows the Question, Options, and highlights the correct answer in green. Below, an example of correct Reasoning and an example of incorrect Reasoning is illustrated.
Figure 8: An example of an MCQ in the Narrative category of a CineMME-Reasoning shows the Question, Options, and highlights the correct answer in green. Below, an example of correct Reasoning and an example of incorrect Reasoning is illustrated.
Figure 9: An example of an MCQ in the Perception category of a CineMME-Reasoning shows the Question, Options, and highlights the correct answer in green. Below, an example of correct Reasoning and an example of incorrect Reasoning is illustrated.
Figure 10: An example of an MCQ in the Scene Presentation category of a CineMME-Reasoning shows the Question, Options, and highlights the correct answer in green. Below, an example of correct Reasoning and an example of incorrect Reasoning is illustrated.
Figure 11: An example of an MCQ in the Social category of a CineMME-Reasoning shows the Question, Options, and highlights the correct answer in green. Below, an example of correct Reasoning and an example of incorrect Reasoning is illustrated.
Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entities' contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.
Current video moment retrieval excels at action-centric tasks but struggles with narrative content. Models can see what is happening but fail to reason why it matters. This semantic gap stems from the lack of Theory of Mind (ToM): the cognitive ability to infer implicit intentions, mental states, and narrative causality from surface-level observations. We introduce StoryTR, the first video moment retrieval benchmark requiring ToM reasoning, comprising 8.1k samples from narrative short-form videos (shorts/reels). These videos present an ideal testbed. Their high information density encodes meaning through subtle multimodal cues. For instance, a glance paired with a sigh carries entirely different semantics than the glance alone. Yet multimodal perception alone is insufficient; ToM is required to decode that a character smiling'' may actually be concealing hostility.'' To teach models this reasoning capability, we propose an Agentic Data Pipeline that generates training data with explicit three-tier ToM chains (intent decoding, narrative reasoning, boundary localization). Experiments reveal the severity of the reasoning gap: Gemini-3.0-Pro achieves only 0.53 Avg IoU on StoryTR. However, our 7B Shorts-Moment model, trained on ToM-guided data, improves +15.1% relative IoU over baselines, demonstrating that narrative reasoning capability matters more than parameter scale.
Xuanyue Zhong, Yuqiang Xie, Guanqun Bi +2
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong · SkyReels, Skywork AI · Independent Researcher
Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.
Zhaoyang Wei, Zipeng Wang, Yushe Cao +10
University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China