DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas
Organizations: Huawei Technologies Ltd
Abstract
Despite recent progress in long-form video understanding, current benchmarks mostly focus on short clips or hour-scale videos. This leaves a major gap in evaluating ultra-long video understanding, where videos often last tens of hours. To bridge this gap, we introduce DramaVQA, a large-scale benchmark built upon long-form TV dramas. Containing approximately 20K question-answering (QA) pairs, the scale of DramaVQA exceeds previous long-video datasets, such as Video-MME and LVBench, by an order of magnitude. Our benchmark has two main features that make it stand out from existing datasets. First, we establish a scalable, LLM-assisted annotation pipeline that uses multimodal large language models (MLLMs) to generate complex questions, followed by strict human checking to ensure high data quality. Second, DramaVQA introduces a multi-level time structure; queries are designed across different video lengths, with the hardest questions requiring overall understanding across the entire drama. Extensive evaluations show that state-of-the-art MLLMs face major challenges when tested on these very long videos, highlighting the need for agentic AI algorithms capable of managing long-term memory. The project is available at https://github.com/xinyuehuo/DramaVQA
Figures & tables
| Benchmark | #QAs [per video] | Avg Dura. | MG | VM | TG | Annotation |
|---|---|---|---|---|---|---|
| TVQA-test Lei et al. (2018) | 15,253 [1.0] | 0.003h | ✗ | ✗ | – | Manual |
| ActivityNet-QA Yu et al. (2019) | 800 [1.0] | 0.03h | ✗ | ✗ | – | Manual |
| MVBench Li et al. (2024c) | 4,000 [1.1] | 0.04h | ✗ | ✗ | – | Auto |
| EgoSchema Mangalam et al. (2023) | 5,063 [1.0] | 0.05h | ✗ | ✗ | – | Auto+Manual |
| MovieQA Tapaswi et al. (2016) | 6,462 [1.0] | 0.06h | ✗ | ✗ | – | Manual |
| LongVideoBench Wu et al. (2024) | 6,678 [1.8] | 0.13h | ✗ | ✓ | – | Manual |
| Model | Fr. | Th. | Medium (L3) Mini [Full] (time) | Long (L4) Mini [Full] (time) | Extra-long (L5) Mini [Full] (time) | Ultra-long (L6) Mini [Full] (time) |
|---|---|---|---|---|---|---|
| Human | – | – | 81.0 ( 215 ) | – | – | – |
| No-vision Models | ||||||
| Random Guess | 0.0 | ✗ | 16.2 [16.2] (0.0) | 16.2 [16.2] (0.0) | 16.2 [16.2] (0.0) | 16.2 [16.2] (0.0) |
| Language Only (Q3) | 256 | ✗ | 21.5 [22.2] (0.1) | 21.5 [22.2] (0.1) | 21.5 [22.2] (0.1) | 21.5 [22.2] (0.1) |
| Language Only (Q3.5) | 256 | ✓ | 23.1 [23.3] (33.9) | 23.1 [23.3] (33.9) | 23.1 [23.3] (33.9) | 23.1 [23.3] (33.9) |
| Proprietary Models | ||||||
| Model | Multi-Point (1,510) | Multi-Hop (1,200) | ||||
|---|---|---|---|---|---|---|
| TTC-2 | TTC-3 | CECI | Average | Single-QA | Multi-Hop-QA | |
| Random | 50.0 | 16.7 | 50.0 | 39.6 | 13.7 | 16.0 |
| Qwen3-VL-8B Bai et al. (2025a) | 47.4 | 18.6 | 49.2 | 38.2 | 39.5 | 38.4 |
| Qwen3-VL-32B Bai et al. (2025a) | 46.4 | 19.9 | 43.4 | 36.1 | 44.8 | 42.2 |
| GLM-4.6V-106B Hong et al. (2025) | 47.4 | 18.8 | 48.2 | 37.8 | 48.4 | 46.0 |
| Qwen3.5-27B Team (2026a) | 66.3 | 57.9 | 50.6 | 57.3 | 61.0 | 56.8 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | #Vids | Dura. | #QAs | #Faces | #Spkrs | Sub Len. | Cap Len. | QA Len. |
|---|---|---|---|---|---|---|---|---|
| Video-MME-S | 300 | 0.02h | 3.0 | N/A | N/A | 198.6 | N/A | 28.7 |
| Video-MME-M | 300 | 0.14h | 3.0 | N/A | N/A | 1425.6 | N/A | 32.8 |
| Video-MME-L | 300 | 0.69h | 3.0 | N/A | N/A | 6515.6 | N/A | 45.6 |
| LVBench | 103 | 1.14h | 15.0 | N/A | N/A | N/A | N/A | 45.4 |
| DramaVQA -M | 3,699 | 0.13h | 5.4 | 14,594.7 | 126.3 | 829.0 | 19,098.5 | 80.5 |
| DramaVQA -L | 649 | 0.72h | 30.8 | 83,183.1 | 720.0 | 4,724.7 | 109,286.1 | 80.5 |
| Major Category | QA Number (Full/Mini) | Subcategory | QA Number (Full) |
| Visual Perception | 4966 / 475 | Character Clothing (CCI) | 1397 |
| Character Hairstyle (CH) | 306 | ||
| Distinctive Features (DF) | 399 | ||
| Object Appearance (OA) | 1168 | ||
| Object Presence (OP) | 437 | ||
| Season and Weather (SAW) | 155 |
| Face | Subtitle | Speaker | Acc. |
|---|---|---|---|
| bounding boxes | hard-rendered | ✗ | 50.0 |
| bounding boxes | text | ✗ | 51.0 |
| character gallery | text | ✗ | 49.4 |
| bounding boxes | ✗ | ✗ | 38.6 |
| bounding boxes | text | ✓ | 53.0 |
| Model | TP | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Acc. | time | Acc. | time | Acc. | time | Acc. | time | ||
| Open-source Native MLLMs | |||||||||
| Qwen3-VL-32B † ‡ | C/B | 51.0 | 22.6 | 35.8 | 36.4 | 28.2 | 37.6 | 34.1 | 30.4 |
| Open-source Agentic MLLMs | |||||||||
| Naive-RAG (Q3.8) | C | 60.1 | 293.0 | 56.7 | 1,664.6 | 37.0 | 1,955.5 | 31.5 | 1,955.0 |
| B | 34.9 | 34.1 | 4.9 | 1.4 | |||||
Video Understanding
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.
See all 122 papers