CineMME: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding
Organizations: School of Information Science and Technology, ShanghaiTech University, Shanghai, China · MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · ByteDance, San Jose, USA. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Abstract
Narrative video understanding requires more than recognizing visible events: models must extract evidence from dialogue, bind it to speakers, and use it to infer narrative meaning. Existing video benchmarks evaluate aspects of this problem, including video question answering, audio-visual reasoning, and temporal grounding, but rarely test whether these abilities form a coherent dialogue-to-plot evidence chain within the same clips. We introduce CineMME, a human-verified benchmark designed to diagnose dialogue-grounded plot understanding, supported by approximately expert person-hours of annotation. CineMME contains curated short narrative video clips and two linked tracks: CineMME-Grounding, with timestamped speech segments and speaker face boxes for temporal dialogue localization, transcription, and active-speaker grounding; and CineMME-Reasoning, with multiple-choice and open-ended questions spanning perception, audio, narrative, social, and scene-presentation understanding. Our main evaluation compares MLLMs on reasoning and omni-modal models on grounding. The results reveal a pronounced who-what-when decoupling: Gemini-3 Pro localizes dialogue in time ( tIoU) and transcribes it with a WER of , yet achieves only vIoU for active-speaker grounding while achieving MCQ accuracy on questions with extractable answers. Modality ablations further show that vision is not uniformly beneficial: it improves performance on perception-heavy questions but can hurt dialogue-centric reasoning. CineMME provides a compact diagnostic testbed for evaluating how multimodal video models connect dialogue, speakers, visual evidence, and narrative meaning.
Figures & tables
| Reasoning Q&A | Free-Form Perception | ||||||||
| Benchmark | Videos | Q&A | Anno. | Modality | MCQ | Open | Spatial- Grounding | Temporal- Grounding | Speech |
| Visual Benchmarks | |||||||||
| Video-MME ( Fu et al., 2025 ) | 900 | 2.7K | M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| MLVU ( Zhou et al., 2025a ) | 1,730 | 3.1K | A+M | V | ✓ | ✓ | ✗ | ✗ | ✗ |
| Omni Benchmarks | |||||||||
| LongVALE ( Geng et al., 2025 ) | 8.4K | – | A | V+A | ✗ | ✗ | ✗ | ✓ | ✓ |
| Models | LLM Params | Frame Rate | MCQs Accuracy (%) | OEQs | |||||
| Narrative | Percept. | Scene | Social | Audio | Avg. | ||||
| Open-source / open-weight MLLMs | |||||||||
| Vision-Only | |||||||||
| Phi-4-multimodal Microsoft et al. (2025) | 5.6B | 64 FPV | 17.65 | 20.15 | 28.74 | 24.34 | 27.92 | 24.22 | 28.63 |
| Kimi-VL Team et al. (2025b) | 16B | 128 FPV | 18.04 | 20.90 | 20.92 | 23.87 | 27.83 | 22.56 | 37.82 |
| InternVL3.5 Wang et al. (2025b) | 38B | 32 FPV | 27.45 | 28.73 | 36.39 | 27.68 | 33.66 | 30.88 | 29.46 |
| Model | Params | tIoU | vIoU | WER |
| Open-source / open-weight omni-modal MLLMs | ||||
| MiniCPM-o-2.6 Yao et al. (2024) | 8B | 34.32 | 0.24 | 96.76 |
| OmniVinci Ye et al. (2025) | 9B | 5.13 | 0.09 | 98.91 |
| Phi-4-multimodal Microsoft et al. (2025) | 5.6B | 16.96 | 0.09 | 98.22 |
| Qwen2.5-Omni Xu et al. (2025a) | 7B | 24.72 | 0.38 | 93.31 |
| Qwen3-Omni Xu et al. (2025b) | 30B | 49.25 | 3.18 | 75.79 |
| Model | Input | CineMME-Reasoning MCQ Accuracy (%) | CineMME-Grounding | ||||||
| Narrative | Perception | Scene | Social | Audio | tIoU | vIoU | WER | ||
| Gemini-3-Pro | A | 69.02 | 67.54 | 63.61 | 70.88 | 74.43 | 68.64 | 3.46 | 27.42 |
| Gemini-3-Pro | A+V | 65.74 | 79.39 | 59.25 | 68.04 | 68.21 | 68.34 | 7.74 | 28.99 |
| Qwen3-Omni | A | 35.29 | 41.79 | 33.81 | 36.52 | 42.72 | 63.58 | 1.92 | 36.77 |
| Qwen3-Omni | A+V (1 FPS) | 30.98 | 47.57 | 37.54 | 37.47 | 38.51 | 49.25 | 3.18 | 75.79 |
| Qwen3-Omni | A+V (256 FPV) | 34.90 | 44.57 | 36.10 | 35.56 | 41.75 | 50.40 | 3.01 | 71.52 |
| Model | Split metric | Better | Worse | Gap |
| Gemini-3 Pro | tIoU | 51.55 | 44.71 | |
| vIoU | 47.68 | 49.86 | ||
| WER | 52.43 | 43.43 | ||
| Qwen3-Omni | tIoU | 31.63 | 28.73 | |
| vIoU | 27.71 | 32.59 | ||
| WER | 31.15 | 29.04 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Adv. Round | Qwen2.5-VL | Qwen3-VL | GLM-4.1V |
| Initial | 68.24 | 71.24 | 57.08 |
| Round 1 | 46.78 | 44.64 | 35.19 |
| Round 2 | 20.17 | 24.89 | 11.16 |
| Stage | Scope | Avg. time / unit | Person-hours |
| Grounding | 546 videos | 1.56 h | 850 |
| Reasoning | 667 videos | 0.90 h | 600 |
| Final audit | 975 clips | 0.15 h | 150 |
| Total | 975 clips | 1.64 h | 1,600 |
| Item | Agreement | Cohen’s | Final flag rate | |
| ASR text | 21,771 | 95.01% | 0.8998 | 51.52% non-Yes |
| Timestamp | 21,833 | 94.56% | 0.8910 | 55.25% non-Yes |
| Track ID | 21,742 | 97.69% | 0.9415 | 32.18% non-Yes |
| Discard decision | 23,603 | 98.10% | 0.8895 | 10.19% discard |
| Comparison | Pearson | Spearman | QWK | MAE | 1 |
| Human 1 vs. Human 2 | 0.752 | 0.775 | 0.745 | 1.15 | 71.8% |
| LLM judge vs. human mean | 0.758 | 0.758 | 0.721 | 1.28 | 58.2% |
| LLM judge vs. Human 1 | 0.753 | 0.756 | 0.745 | 1.20 | 63.6% |
| LLM judge vs. Human 2 | 0.659 | 0.673 | 0.638 | 1.55 | 51.8% |
| Category | What It Measures | Representative Question Coverage |
| Perception | Extraction of objective audiovisual facts under visual complexity or short temporal windows. | Questions may ask about subtle visual details, dynamic counts, spoken content, speaker identity, or where and when an event occurs. |
| Audio | Interpretation of spoken and non-speech acoustic evidence, including subtle, masked, or off-screen cues. | Questions may focus on dialogue, vocal delivery, background sounds, subjective sound effects, or off-screen events that cannot be answered from frames alone. |
| Narrative | Reasoning about story structure, causal dependencies, chronology, and information asymmetry. | Questions may examine causal chains, temporal order, narrative structure, privileged information, or how one event changes the interpretation of another. |
| Social | Inference over intentions, beliefs, relationships, cultural context, and behavior that contradicts generic priors. | Questions may ask models to infer goals, false beliefs, implicit relationships, culturally grounded meanings, or evidence that overrides a generic social prior. |
| Scene Presentation | Understanding how visual composition and audio-visual presentation cues convey narrative, psychological, or thematic meaning. | Questions may connect framing, editing rhythm, sound design, lighting, props, or scene layout to their narrative or emotional effects. |
| Category | Human (%) | Gemini-3 Pro (%) | Difference |
| Audio | 75.70 | 68.21 | +7.49 |
| Social | 78.26 | 68.04 | +10.22 |
| Narrative | 76.47 | 65.74 | +10.73 |
| Perception | 89.19 | 79.39 | +9.80 |
| Scene | 66.67 | 59.25 | +7.42 |
| Overall | 77.64 | 67.66 | +9.98 |
| Input | Audio | Social | Narrative | Perception | Scene | Overall |
| Question/options only | 39.16 | 32.46 | 31.76 | 30.22 | 32.09 | 33.19 |
| Vision only | 51.46 | 42.00 | 36.47 | 47.01 | 44.99 | 44.44 |
| Audio only | 57.93 | 47.97 | 48.63 | 58.21 | 42.69 | 50.56 |
| Full A/V | 57.28 | 53.94 | 48.63 | 64.18 | 51.58 | 54.94 |
| Vision + shuffled audio | 49.19 | 38.90 | 40.39 | 43.45 | 45.56 | 43.34 |
| Input | Audio | Social | Narrative | Perception | Scene | Overall | N |
| Question only | 31.21 | 32.06 | 20.00 | 14.69 | 41.86 | 27.09 | 313 |
| Vision only | 31.36 | 46.47 | 33.33 | 26.22 | 58.57 | 38.33 | 311 |
| Audio only | 38.79 | 48.53 | 37.50 | 26.53 | 46.57 | 38.17 | 311 |
| Full A/V | 41.06 | 59.12 | 35.62 | 32.35 | 57.86 | 43.75 | 312 |
| Vision + shuffled audio | 34.09 | 38.24 | 34.79 | 28.78 | 51.43 | 37.07 | 314 |
| Model | Input | Audio | Social | Narrative | Perception | Scene | Overall |
| MiMo-v2.5 | Q/options only | 39.16 | 32.46 | 31.76 | 30.22 | 32.09 | 33.19 |
| Correct A/V | 57.28 | 53.94 | 48.63 | 64.18 | 51.58 | 54.94 | |
| Gain | +18.12 | +21.48 | +16.86 | +33.96 | +19.48 | +21.75 | |
| MiniCPM-o-4.5 | Q/options only | 28.48 | 20.53 | 18.43 | 13.43 | 28.08 | 22.19 |
| Correct A/V | 33.33 | 27.92 | 31.37 | 37.69 | 29.51 | 31.50 | |
| Gain | +4.85 | +7.40 | +12.94 | +24.25 | +1.43 | +9.31 |
| Model | Media | MCQ accuracy (%) | OEQ score (0–100) | |||||
| Q | Correct | Same | Different | Correct | Same | Different | ||
| MiniCPM-o-4.5 | A/V | 22.19 | 31.50 | 22.78 | 22.45 | 28.79 | 23.31 | 18.76 |
| Qwen3-Omni | A/V | 26.88 | 45.25 | 32.00 | 29.39 | 28.15 | 16.12 | 13.14 |
| Qwen3-VL-30B-A3B | Vision | 28.00 | 35.62 | 32.97 | 30.37 | 30.35 | 21.01 | 19.76 |
| Task Type | Question | Option | Answer |
| Narrative | 25.6 | 23.7 | 19.0 |
| Perception | 24.5 | 13.2 | 10.1 |
| Scene | 26.6 | 28.3 | 22.3 |
| Social | 24.1 | 23.8 | 16.4 |
| Audio | 27.3 | 25.0 | 19.2 |
| Overall | 25.6 | 23.2 | 17.5 |
| Track | N | Min | Median | Mean | P90 | Max |
| MCQ | 1600 | 52.1 | 151.1 | 156.0 | 205.2 | 299.1 |
| OEQ | 316 | 22.1 | 148.1 | 148.0 | 190.2 | 274.1 |
| Grounding | 546 | 22.1 | 147.1 | 148.0 | 189.1 | 300.1 |
| Pooled task records | 2462 | 22.1 | 151.1 | 153.2 | 202.1 | 300.1 |
| Position | A | B | C | D | E | F |
| Count | 275 | 252 | 267 | 296 | 276 | 234 |
| Share (%) | 17.19 | 15.75 | 16.69 | 18.50 | 17.25 | 14.63 |
| Track | Audio | Social | Narrative | Perception | Scene | Total | Retained |
| MCQ | 300/309 | 417/419 | 249/255 | 263/268 | 346/349 | 1575/1600 | 98.44% |
| OEQ | 64/66 | 33/34 | 45/48 | 94/98 | 66/70 | 302/316 | 95.57% |
| Grounding | – | – | – | – | – | 513/546 | 93.96% |
| Model | Scored questions | Accuracy | 95% CI |
| Phi-4-multimodal | 1598 | 24.22 | [22.15, 26.35] |
| Kimi-VL | 1600 | 22.56 | [20.56, 24.62] |
| InternVL3.5 | 1600 | 30.88 | [28.62, 33.12] |
| Ministral-3 | 1600 | 22.69 | [20.62, 24.69] |
| MiMo-VL | 1600 | 27.88 | [25.69, 30.12] |
| GLM-4.1V | 1600 | 23.25 | [21.12, 25.37] |
| Model | ||||
| Phi-4-multimodal | 1598 | 315 | 79 | 79 |
| Kimi-VL | 1600 | 316 | – | – |
| InternVL3.5 | 1600 | 316 | – | – |
| Ministral-3 | 1600 | 316 | – | – |
| MiMo-VL | 1600 | 315 | – | – |
| GLM-4.1V | 1600 | 316 | – | – |
| Input | |||
| Gemini-3 Pro | |||
| A only | 1600 | 541 | 535 |
| A+V | 1574 | 546 | 540 |
| Qwen3-Omni | |||
| A+V (1 FPS) | 1599 | 433 | 429 |
| A+V (256 FPV) | 1599 | 440 | 436 |
| Visual | Audio | Frames | CineMME-Reasoning | CineMME-Grounding | |||||||
| Avg. | Narr. | Perc. | Scene | Soci. | Audi. | tIoU | vIoU | WER | |||
| ✓ | ✓ | 1 FPS | 38.34 | 30.98 | 47.57 | 37.54 | 37.47 | 38.51 | 49.25 | 3.18 | 75.79 |
| ✓ | ✓ | 256 | 38.27 | 34.90 | 44.57 | 36.10 | 35.56 | 41.75 | 50.40 | 3.01 | 71.52 |
| ✓ | ✓ | 128 | 39.40 | 36.47 | 44.57 | 37.54 | 36.99 | 42.72 | 48.20 | 3.49 | 74.93 |
| ✓ | ✓ | 64 | 37.31 | 32.55 | 44.03 | 34.38 | 35.56 | 41.10 | 50.48 | 3.67 | 75.92 |
| ✓ | ✓ | 32 | 38.78 | 32.68 | 46.62 | 34.96 | 38.04 | 42.39 | 52.91 | 3.63 | 76.42 |
| Model | Split metric | MCQs / videos | Better | Worse | Gap |
| Gemini-3 Pro | tIoU | 166 / 144 | 73.81 | 78.05 | |
| vIoU | 164 / 142 | 75.32 | 75.86 | ||
| WER | 166 / 144 | 76.47 | 75.31 | ||
| Qwen3-Omni | tIoU | 141 / 121 | 33.33 | 43.94 | |
| vIoU | 140 / 120 | 32.86 | 44.29 | ||
| WER | 141 / 121 | 28.38 | 49.25 |
| Reasoning Q&A | Free-Form Perception | ||||||||
| Benchmark | # Videos | # Q&A | Anno. | Modality | MCQ | Open | Spatial- Grounding | Temporal- Grounding | Speech |
| Visual Benchmarks: | |||||||||
| MV-Bench ( Li et al., 2024 ) | 3,641 | 4.0 K | A+M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| EgoSchema ( Mangalam et al., 2023 ) | 5,063 | 5.0 K | A+M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| MLVU ( Zhou et al., 2025a ) | 1,730 | 3.1 K | A+M | V | ✓ | ✓ | ✗ | ✗ | ✗ |
| Video-MME ( Fu et al., 2025 ) | 900 | 2.7 K | M | V | ✓ | ✗ | ✗ | ✗ | ✗ |
| Models | Organization | Release | Version | Video Support | Audio Support | Inference Deployment |
| Open-source / open-weight MLLMs | ||||||
| Phi-4-multimodal Microsoft et al. (2025) | Microsoft | 2025-3 | Phi-4-multimodal-instruct | ✗ | ✓ | vLLM |
| Kimi-VL Team et al. (2025b) | Moonshot | 2025-4 | Kimi-VL-A3B-Thinking | ✗ | ✗ | vLLM |
| InternVL3.5 Wang et al. (2025b) | OpenGVLab | 2025-8 | InternVL3_5-38B-Instruct | ✗ | ✗ | vLLM |
| Ministral-3 Liu et al. (2026) | Mistral | 2025-12 | Ministral-3-14B-Reasoning-2512 | ✗ | ✗ | vLLM |
| MiMo-VL Li et al. (2025b) | Xiaomi | 2025-6 | MiMo-VL-7B-RL-2508 | ✓ | ✗ | vLLM |
Video-Language Model Evaluation
Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.
See all 115 papers