Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they consistently fail to outperform naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding.
Figures & tables
Figure 1: Existing scene boundaries fail to support movie RAG. (a) Scene boundary comparison: Standard scene segmentation boundaries often follow visually salient transitions, which can create spurious cuts and miss valid boundaries, fragmenting a continuous narrative event. In contrast, our definition groups shots by shared narrative goal and preserves coherent context. (b) Movie RAG example: This difference directly affects retrieval-based movie claim verification. Fragmented visual chunks can miss relevant evidence and lead to an incorrect answer, while narrative scene boundaries retrieve a coherent event for movie understanding.
Figure 2: Movie RAG pipeline with scene segmentation-based retrieval. A movie is first partitioned into chunks (segments), segmentation representations are retrieved against a text query, and the retrieved segment is served as evidence for a downstream task.
Table 1: Comparison of scene segmentation baselines on the temporal grounding benchmark MAD [ 29 ] . We evaluate two recent temporal grounding models (TimeLens, Vidi) under the same movie RAG pipeline. Fixed-length uniform chunks provide strong baselines, while legacy scene segmentation methods consistently underperform.
Condition
Avg. Length (s)
Qwen3-VL (4B) [ 1 ]
Qwen3-VL (8B) [ 1 ]
InternVL3.5 8B [ 34 ]
InternVL3.5 14B [ 34 ]
QA Acc. (%)
QA Acc. (%)
QA Acc. (%)
QA Acc. (%)
K =1
K =3
K =5
K =1
K =3
K =5
K =1
K =3
K =5
K =1
K =3
K =5
Full-length Movie
–
47.2
48.6
51.9
55.2
Fixed-Length Retrieval Chunks
Uniform Split (2 min)
120.0
40.2
65.7
75.0
40.4
66.8
75.6
52.3
72.0
78.5
54.5
72.0
78.4
Uniform Split (3 min)
180.0
43.8
68.1
77.9
42.6
65.2
75.6
51.1
69.4
76.3
55.4
72.1
77.8
Table 2: Comparison of scene segmentation baselines on movie question answering results on MovieStory101 [ 12 ] . We evaluate downstream QA models including Qwen3-4B, Qwen3-8B, InternVL3.5 8B, and InternVL3.5 14B while varying the retrieved scene units.
Condition
Avg. Length (s)
Qwen3-VL (4B) [ 1 ]
Qwen3-VL (8B) [ 1 ]
Pairwise Acc. (%)
Pairwise Acc. (%)
K =1
K =2
K =3
K =5
K =1
K =2
K =3
K =5
Full-length Movie
–
12.56
23.77
Fixed-Length Retrieval Chunks
Uniform Split (2 min)
120.0
12.90
20.51
25.23
33.29
9.45
16.71
22.85
29.26
Uniform Split (3 min)
180.0
12.79
20.07
25.35
32.14
8.41
16.82
22.70
28.00
Table 3: Impact of segmentation methods on movie fact verification in MF 2 [ 41 ] . Whole-movie rows feed the entire video directly into the MLLM without retrieval. All other rows use Qwen3-VL-Embedding retrieval and pass the top- K retrieved chunks to the downstream model. Uniform splitting remains a strong baseline, while comparisons between length-controlled and semantically grounded chunks reveal that narrative boundary alignment contributes beyond chunk length alone.
Figure 3: Three levels of narrative-centric scene transitions. We characterize scene segmentation candidate boundaries by changes in physical setting, character configuration, and narrative focus. A valid narrative scene boundary must include Level 3 change, while Level 1 and Level 2 cues alone are not sufficient.
Attribute
Value
Level 1
80.4
Level 2
31.6
Level 3
11.4
None
15.7
Table 4: Audit of MovieNet-SSeg under our EST-based scene definition. Values are percentages. Level 1, Level 2, and Level 3 are non-exclusive because a boundary may satisfy multiple levels. None denotes a benchmark boundary that satisfies none of the three criteria.
Action
Count
Rate
Confirm
1684
58.4%
Relabel
1027
35.6%
Delete
99
3.4%
Move
34
1.2%
Insert
30
1.0%
Hard
8
0.3%
Table 5: Cross-verification in NarraScene.
Condition
Qwen3-VL (4B)
Qwen3-VL (8B)
K =1
K =2
K =3
K =5
K =1
K =2
K =3
K =5
Uniform Split (2 min)
12.90
20.51
25.23
33.29
9.45
16.71
22.85
29.26
Uniform Split (3 min)
12.79
20.07
25.35
32.14
8.41
16.82
22.70
28.00
Uniform Split (4 min)
12.56
19.93
24.19
30.65
8.45
16.59
22.35
29.26
GenreDur [ 10 ]
12.90
21.14
24.53
36.98
9.56
17.40
23.96
33.41
NarraScene
18.66
22.24
30.53
43.43
19.59
22.35
28.34
41.47
Table 6: NarraScene improves pairwise accuracy on movie claim verification in MF 2 . All rows use the same RAG pipeline with a Qwen3-VL-Embedding retriever and Qwen3 QA models, differing only in the retrieval units used under each condition. We compare fixed-length uniform chunks against semantic oracle boundaries annotated under our EST-grounded definition.
Figure 4: Qualitative example of retrieved chunks under different scene segmentation strategies. We compare fixed-length chunking, a standard scene segmentation method (BaSSL), and NarraScene on a movie claim verification example from Murder by Contract . The claim asks whether Claude wanted the job to get more money to buy a house. Uniform chunking and BaSSL retrieve incomplete scene segments and lead to an incorrect answer, while the narrative scene retrieved from NarraScene preserves the full event and contains the decisive evidence, namely Claude’s explicit statement that he wants to buy a certain house on the Ohio River.
Setting
Top-1
Best Overlap
Uniform 3 min
41.6
60.9
GenreDur
36.2
50.4
Table 7: Retrieval quality analysis on MovieStory101. We compare actual top-1 retrieval against the chunk with the largest overlap with the annotated 3-minute evidence clip under Qwen3-VL 8B.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Prompt template used to elicit a single-choice answer from the QA model on MovieStory101. The placeholder {question} is filled per instance.
Figure B : Prompt templates used by the two MAD localizers. One prompt is issued per retrieved scene. The placeholders {query} and {video_length} are filled per query with the MAD caption and the per-clip duration in seconds, respectively. TimeLens-8B emits boundaries directly in seconds. Vidi-7B emits a normalized start%--end% range, which we remap to seconds using the clip duration before mapping to the MAD movie timeline.
Figure C : Prompt template used by the MF 2 verifier. One instance is run independently per retrieved scene and combined with an OR ensemble across the top- K candidates. The placeholders {transcripts} and {claim} are filled with the ASR transcript of the scene and the claim under evaluation, respectively.
Film
# Boundaries
L1
L2
L3
None
The Terminator (1984)
48
89.6 (43)
54.2 (26)
10.4 (5)
10.4 (5)
Sherlock Holmes (2009)
214
78.5 (168)
33.2 (71)
9.3 (20)
17.3 (37)
Ring (1998)
81
84.0 (68)
33.3 (27)
12.3 (10)
11.1 (9)
Gravity (2013)
46
80.4 (37)
4.3 (2)
19.6 (9)
13.0 (6)
The Curious Case of Benjamin Button (2008)
185
72.4 (134)
25.9 (48)
13.0 (24)
26.5 (49)
Insidious (2010)
121
90.1 (109)
33.9 (41)
10.7 (13)
9.1 (11)
Appendix
Table A: Film-wise audit of MovieNet scene boundaries using our cognitive taxonomy. Values are percentages, with raw boundary counts in parentheses.
Figure D : Custom annotation interfaces used in the three-stage NarraScene pipeline. (a) In Stage 1, a single annotator watches the full movie, records scene boundary timestamps, assigns one or more narrative levels, and leaves a memo when needed. (b) In Stage 2, a second annotator reviews local clips and applies actions such as confirm, relabel, move, delete, insert, or hard marking for ambiguous cases that should be revisited in consensus. (c) In Stage 3, all four annotators compare the original and revised annotations and resolve disagreements through consensus. Zoom in for details.
# Movies
Hours
Mean movie
Median movie
# Boundaries
# Scenes
Mean scenes
Mean scene
Median scene
dur. (min)
dur. (min)
per movie
dur. (sec)
dur. (sec)
53
76.99
87.16
87.00
2371
2424
45.74
116.90
98.00
Appendix
Table B: Detailed statistics of the MF 2 -aligned portion of NarraScene.
Label combination
Count
Rate
L1+L2+L3
1275
53.75%
L1+L3
489
20.62%
L2+L3
403
17.00%
L3
204
8.60%
L1 presence rate
74.02%
L2 presence rate
70.35%
Appendix
Table C: Detailed statistics of scene-boundary labels in the MF 2 -aligned portion of NarraScene.
Figure E : Statistical overview of the MF 2 -aligned portion of NarraScene. The distribution of scene durations (a) shows that the dataset spans a broad range of narrative unit lengths, with a mean of 1.97 minutes and a median of 1.65 minutes. The distribution of scene-boundary annotations across cognitive-level combinations (b) shows that the dominant boundary types include Level 3 semantics, confirming that NarraScene emphasizes semantic scene transitions beyond low-level visual discontinuities. Zoom in for details.
Table D : Scene segmentation performance on NarraScene
Figure F : Qualitative example on MovieStory101 Movie QA. We compare retrieval results produced by different chunking strategies on a multiple-choice movie question from Chasing The Dragon . The figure shows the oracle evidence clip together with the top retrieved chunk from each method and the corresponding QA prediction. Uniform chunking retrieves a clip that better matches the annotated evidence, whereas scene-based baselines may retrieve distractor clips that lead to incorrect answers or fallback predictions.
Figure G : Qualitative example on MAD Movie Temporal Grounding. We compare retrieval and temporal grounding outputs under different chunking strategies for a query from Fargo . The figure shows the ground-truth timeline together with the retrieved chunk and predicted temporal span from each method. Uniform chunking retrieves a candidate that more tightly covers the target moment, while scene-based baselines produce less accurate candidates and correspondingly weaker temporal localization.
Recent progress in multimodal large language models (MLLMs) has led to a surge of benchmarks for long-video reasoning. However, most existing benchmarks rely on localized cues and fail to capture narrative reasoning, the ability to track intentions, connect distant events, and reconstruct causal chains across an entire movie. We introduce NA-VQA, a benchmark designed to evaluate deep temporal and narrative reasoning in long-form videos. NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far to assess long-range dependencies. By requiring generative, multi-scene answers, NA-VQA tests whether models can integrate dispersed narrative information rather than rely on shallow pattern matching. To address the limitations of existing approaches, we propose Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured memory for retrieval during reasoning. Extensive experiments show that state-of-the-art MLLMs perform poorly on questions requiring far-range evidence, highlighting the need for explicit narrative modeling. Video-NaRA improves long-range reasoning performance by up to 3 percent, demonstrating its effectiveness in handling complex narrative structures. We will release NA-VQA upon publication.
Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception. This leads us to a key question: can current VLMs reason effectively over long, scene-level contexts? To answer this, we introduce a new benchmark, SceneBench, designed to provide scene-level challenges. Our evaluation reveals a sharp drop in accuracy when VLMs attempt to answer scene-level questions, indicating significant forgetting of long-range context. To further validate these findings, we propose Scene Retrieval-Augmented Generation (Scene-RAG), which constructs a dynamic scene memory by retrieving and integrating relevant context across scenes. This Scene-RAG improves VLM performance by +2.50%, confirming that current models still struggle with long-context retention. We hope SceneBench will encourage future research toward VLMs with more robust, human-like video comprehension.
Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of ⟨query, evidence chunk, answer⟩ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.
Yuho Lee, Jisu Shin, Nicole Hee-Yeon Kim +5
KAIST, YH, Seoul, Republic of Korea · Qualcomm AI Research, Qualcomm Korea, YH, Seoul, Republic of Korea