cs.CVNov 11, 2026

CineMME: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding

Authors: Jin Liu, Dawei Du, Yexiang Liu, Sijie Zhu, Fan Chen, Junxian Duan, Huaibo Huang, Ran He, 

Organizations: School of Information Science and Technology, ShanghaiTech University, Shanghai, China · MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · ByteDance, San Jose, USA. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Abstract

Narrative video understanding requires more than recognizing visible events: models must extract evidence from dialogue, bind it to speakers, and use it to infer narrative meaning. Existing video benchmarks evaluate aspects of this problem, including video question answering, audio-visual reasoning, and temporal grounding, but rarely test whether these abilities form a coherent dialogue-to-plot evidence chain within the same clips. We introduce CineMME, a human-verified benchmark designed to diagnose dialogue-grounded plot understanding, supported by approximately 1,6001,600 expert person-hours of annotation. CineMME contains 975975 curated short narrative video clips and two linked tracks: CineMME-Grounding, with 13,55413,554 timestamped speech segments and 33,08333,083 speaker face boxes for temporal dialogue localization, transcription, and active-speaker grounding; and CineMME-Reasoning, with 1,6001,600 multiple-choice and 316316 open-ended questions spanning perception, audio, narrative, social, and scene-presentation understanding. Our main evaluation compares 2020 MLLMs on reasoning and 99 omni-modal models on grounding. The results reveal a pronounced who-what-when decoupling: Gemini-3 Pro localizes dialogue in time (68.34%68.34\% tIoU) and transcribes it with a WER of 28.99%28.99\%, yet achieves only 7.74%7.74\% vIoU for active-speaker grounding while achieving 67.66%67.66\% MCQ accuracy on questions with extractable answers. Modality ablations further show that vision is not uniformly beneficial: it improves performance on perception-heavy questions but can hurt dialogue-centric reasoning. CineMME provides a compact diagnostic testbed for evaluating how multimodal video models connect dialogue, speakers, visual evidence, and narrative meaning.

Figures & tables

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Jan 3, 2026

NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entities' contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.
Apr 25, 2026

StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning

Current video moment retrieval excels at action-centric tasks but struggles with narrative content. Models can see what is happening but fail to reason why it matters. This semantic gap stems from the lack of Theory of Mind (ToM): the cognitive ability to infer implicit intentions, mental states, and narrative causality from surface-level observations. We introduce StoryTR, the first video moment retrieval benchmark requiring ToM reasoning, comprising 8.1k samples from narrative short-form videos (shorts/reels). These videos present an ideal testbed. Their high information density encodes meaning through subtle multimodal cues. For instance, a glance paired with a sigh carries entirely different semantics than the glance alone. Yet multimodal perception alone is insufficient; ToM is required to decode that a character smiling'' may actually be concealing hostility.'' To teach models this reasoning capability, we propose an Agentic Data Pipeline that generates training data with explicit three-tier ToM chains (intent decoding, narrative reasoning, boundary localization). Experiments reveal the severity of the reasoning gap: Gemini-3.0-Pro achieves only 0.53 Avg IoU on StoryTR. However, our 7B Shorts-Moment model, trained on ToM-guided data, improves +15.1% relative IoU over baselines, demonstrating that narrative reasoning capability matters more than parameter scale.
Sep 15, 2026

Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.