cs.CVAug 27, 2026
SaveBeyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG
Organizations: Korea University · KT · KAIST
Abstract
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they consistently fail to outperform naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding.
Figures & tables
Figure 1: Existing scene boundaries fail to support movie RAG. (a) Scene boundary comparison: Standard scene segmentation boundaries often follow visually salient transitions, which can create spurious cuts and miss valid boundaries, fragmenting a continuous narrative event. In contrast, our definition groups shots by shared narrative goal and preserves coherent context. (b) Movie RAG example: This difference directly affects retrieval-based movie claim verification. Fragmented visual chunks can miss relevant evidence and lead to an incorrect answer, while narrative scene boundaries retrieve a coherent event for movie understanding.
Figure 2: Movie RAG pipeline with scene segmentation-based retrieval. A movie is first partitioned into chunks (segments), segmentation representations are retrieved against a text query, and the retrieved segment is served as evidence for a downstream task.
| Condition | TimeLens-8B [ 43 ] | Vidi-7B [ 32 ] | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| [email protected] | [email protected] | [email protected] | [email protected] | [email protected] | [email protected] | Avg. | [email protected] | [email protected] | [email protected] | [email protected] | [email protected] | [email protected] | Avg. | |
| Full-length Movie | 0.11 | - | 0.00 | - | 0.00 | - | 0.03 | 2.01 | - | 1.20 | - | 0.62 | - | 1.27 |
| Fixed-Length Retrieval Chunks | ||||||||||||||
| Uniform Split (2 min) | 12.12 | 27.42 | 7.80 | 17.59 | 4.25 | 9.45 | 13.10 | 7.14 | 20.48 | 5.82 | 12.75 | 3.28 | 8.10 | 9.59 |
| Uniform Split (3 min) | 11.00 | 26.56 | 7.27 | 17.82 | 4.11 | 10.36 | 12.85 | 5.31 | 17.59 | 3.38 | 10.91 | 1.48 | 7.36 | 7.67 |
| Uniform Split (4 min) | 10.93 | 26.28 | 6.70 | 16.19 | 3.59 | 8.65 | 12.05 | 5.28 | 15.26 | 2.91 | 10.52 | 1.42 | 6.15 | 6.92 |
Table 1: Comparison of scene segmentation baselines on the temporal grounding benchmark MAD [ 29 ] . We evaluate two recent temporal grounding models (TimeLens, Vidi) under the same movie RAG pipeline. Fixed-length uniform chunks provide strong baselines, while legacy scene segmentation methods consistently underperform.
| Condition | Avg. Length (s) | Qwen3-VL (4B) [ 1 ] | Qwen3-VL (8B) [ 1 ] | InternVL3.5 8B [ 34 ] | InternVL3.5 14B [ 34 ] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| QA Acc. (%) | QA Acc. (%) | QA Acc. (%) | QA Acc. (%) | ||||||||||
| =1 | =3 | =5 | =1 | =3 | =5 | =1 | =3 | =5 | =1 | =3 | =5 | ||
| Full-length Movie | – | 47.2 | 48.6 | 51.9 | 55.2 | ||||||||
| Fixed-Length Retrieval Chunks | |||||||||||||
| Uniform Split (2 min) | 120.0 | 40.2 | 65.7 | 75.0 | 40.4 | 66.8 | 75.6 | 52.3 | 72.0 | 78.5 | 54.5 | 72.0 | 78.4 |
| Uniform Split (3 min) | 180.0 | 43.8 | 68.1 | 77.9 | 42.6 | 65.2 | 75.6 | 51.1 | 69.4 | 76.3 | 55.4 | 72.1 | 77.8 |
Table 2: Comparison of scene segmentation baselines on movie question answering results on MovieStory101 [ 12 ] . We evaluate downstream QA models including Qwen3-4B, Qwen3-8B, InternVL3.5 8B, and InternVL3.5 14B while varying the retrieved scene units.
| Condition | Avg. Length (s) | Qwen3-VL (4B) [ 1 ] | Qwen3-VL (8B) [ 1 ] | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Pairwise Acc. (%) | Pairwise Acc. (%) | ||||||||
| =1 | =2 | =3 | =5 | =1 | =2 | =3 | =5 | ||
| Full-length Movie | – | 12.56 | 23.77 | ||||||
| Fixed-Length Retrieval Chunks | |||||||||
| Uniform Split (2 min) | 120.0 | 12.90 | 20.51 | 25.23 | 33.29 | 9.45 | 16.71 | 22.85 | 29.26 |
| Uniform Split (3 min) | 180.0 | 12.79 | 20.07 | 25.35 | 32.14 | 8.41 | 16.82 | 22.70 | 28.00 |
Table 3: Impact of segmentation methods on movie fact verification in MF 2 [ 41 ] . Whole-movie rows feed the entire video directly into the MLLM without retrieval. All other rows use Qwen3-VL-Embedding retrieval and pass the top- retrieved chunks to the downstream model. Uniform splitting remains a strong baseline, while comparisons between length-controlled and semantically grounded chunks reveal that narrative boundary alignment contributes beyond chunk length alone.
Figure 3: Three levels of narrative-centric scene transitions. We characterize scene segmentation candidate boundaries by changes in physical setting, character configuration, and narrative focus. A valid narrative scene boundary must include Level 3 change, while Level 1 and Level 2 cues alone are not sufficient.
| Attribute | Value |
|---|---|
| Level 1 | 80.4 |
| Level 2 | 31.6 |
| Level 3 | 11.4 |
| None | 15.7 |
Table 4: Audit of MovieNet-SSeg under our EST-based scene definition. Values are percentages. Level 1, Level 2, and Level 3 are non-exclusive because a boundary may satisfy multiple levels. None denotes a benchmark boundary that satisfies none of the three criteria.
| Action | Count | Rate |
|---|---|---|
| Confirm | 1684 | 58.4% |
| Relabel | 1027 | 35.6% |
| Delete | 99 | 3.4% |
| Move | 34 | 1.2% |
| Insert | 30 | 1.0% |
| Hard | 8 | 0.3% |
Table 5: Cross-verification in NarraScene.
| Condition | Qwen3-VL (4B) | Qwen3-VL (8B) | ||||||
|---|---|---|---|---|---|---|---|---|
| =1 | =2 | =3 | =5 | =1 | =2 | =3 | =5 | |
| Uniform Split (2 min) | 12.90 | 20.51 | 25.23 | 33.29 | 9.45 | 16.71 | 22.85 | 29.26 |
| Uniform Split (3 min) | 12.79 | 20.07 | 25.35 | 32.14 | 8.41 | 16.82 | 22.70 | 28.00 |
| Uniform Split (4 min) | 12.56 | 19.93 | 24.19 | 30.65 | 8.45 | 16.59 | 22.35 | 29.26 |
| GenreDur [ 10 ] | 12.90 | 21.14 | 24.53 | 36.98 | 9.56 | 17.40 | 23.96 | 33.41 |
| NarraScene | 18.66 | 22.24 | 30.53 | 43.43 | 19.59 | 22.35 | 28.34 | 41.47 |
Table 6: NarraScene improves pairwise accuracy on movie claim verification in MF 2 . All rows use the same RAG pipeline with a Qwen3-VL-Embedding retriever and Qwen3 QA models, differing only in the retrieval units used under each condition. We compare fixed-length uniform chunks against semantic oracle boundaries annotated under our EST-grounded definition.
Figure 4: Qualitative example of retrieved chunks under different scene segmentation strategies. We compare fixed-length chunking, a standard scene segmentation method (BaSSL), and NarraScene on a movie claim verification example from Murder by Contract . The claim asks whether Claude wanted the job to get more money to buy a house. Uniform chunking and BaSSL retrieve incomplete scene segments and lead to an incorrect answer, while the narrative scene retrieved from NarraScene preserves the full event and contains the decisive evidence, namely Claude’s explicit statement that he wants to buy a certain house on the Ohio River.
| Setting | Top-1 | Best Overlap |
|---|---|---|
| Uniform 3 min | 41.6 | 60.9 |
| GenreDur | 36.2 | 50.4 |
Table 7: Retrieval quality analysis on MovieStory101. We compare actual top-1 retrieval against the chunk with the largest overlap with the annotated 3-minute evidence clip under Qwen3-VL 8B.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Prompt template used to elicit a single-choice answer from the QA model on MovieStory101. The placeholder {question} is filled per instance.
Figure B : Prompt templates used by the two MAD localizers. One prompt is issued per retrieved scene. The placeholders {query} and {video_length} are filled per query with the MAD caption and the per-clip duration in seconds, respectively. TimeLens-8B emits boundaries directly in seconds. Vidi-7B emits a normalized start%--end% range, which we remap to seconds using the clip duration before mapping to the MAD movie timeline.
Figure C : Prompt template used by the MF 2 verifier. One instance is run independently per retrieved scene and combined with an OR ensemble across the top- candidates. The placeholders {transcripts} and {claim} are filled with the ASR transcript of the scene and the claim under evaluation, respectively.
| Film | # Boundaries | L1 | L2 | L3 | None |
|---|---|---|---|---|---|
| The Terminator (1984) | 48 | 89.6 (43) | 54.2 (26) | 10.4 (5) | 10.4 (5) |
| Sherlock Holmes (2009) | 214 | 78.5 (168) | 33.2 (71) | 9.3 (20) | 17.3 (37) |
| Ring (1998) | 81 | 84.0 (68) | 33.3 (27) | 12.3 (10) | 11.1 (9) |
| Gravity (2013) | 46 | 80.4 (37) | 4.3 (2) | 19.6 (9) | 13.0 (6) |
| The Curious Case of Benjamin Button (2008) | 185 | 72.4 (134) | 25.9 (48) | 13.0 (24) | 26.5 (49) |
| Insidious (2010) | 121 | 90.1 (109) | 33.9 (41) | 10.7 (13) | 9.1 (11) |
Table A: Film-wise audit of MovieNet scene boundaries using our cognitive taxonomy. Values are percentages, with raw boundary counts in parentheses.
Figure D : Custom annotation interfaces used in the three-stage NarraScene pipeline. (a) In Stage 1, a single annotator watches the full movie, records scene boundary timestamps, assigns one or more narrative levels, and leaves a memo when needed. (b) In Stage 2, a second annotator reviews local clips and applies actions such as confirm, relabel, move, delete, insert, or hard marking for ambiguous cases that should be revisited in consensus. (c) In Stage 3, all four annotators compare the original and revised annotations and resolve disagreements through consensus. Zoom in for details.
| # Movies | Hours | Mean movie | Median movie | # Boundaries | # Scenes | Mean scenes | Mean scene | Median scene |
| dur. (min) | dur. (min) | per movie | dur. (sec) | dur. (sec) | ||||
| 53 | 76.99 | 87.16 | 87.00 | 2371 | 2424 | 45.74 | 116.90 | 98.00 |
Table B: Detailed statistics of the MF 2 -aligned portion of NarraScene.
| Label combination | Count | Rate |
|---|---|---|
| L1+L2+L3 | 1275 | 53.75% |
| L1+L3 | 489 | 20.62% |
| L2+L3 | 403 | 17.00% |
| L3 | 204 | 8.60% |
| L1 presence rate | 74.02% | |
| L2 presence rate | 70.35% | |
Table C: Detailed statistics of scene-boundary labels in the MF 2 -aligned portion of NarraScene.
Figure E : Statistical overview of the MF 2 -aligned portion of NarraScene. The distribution of scene durations (a) shows that the dataset spans a broad range of narrative unit lengths, with a mean of 1.97 minutes and a median of 1.65 minutes. The distribution of scene-boundary annotations across cognitive-level combinations (b) shows that the dominant boundary types include Level 3 semantics, confirming that NarraScene emphasizes semantic scene transitions beyond low-level visual discontinuities. Zoom in for details.
| Method | mIoU | [email protected] | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|---|
| Uniform Heuristics (Training-Free) | ||||||
| Uniform Split (2 min) | 49.3 | 98.8 | 82.9 | 48.5 | 15.3 | 1.7 |
| Uniform Split (3 min) | 44.5 | 97.1 | 74.0 | 38.8 | 12.5 | 1.8 |
| Uniform Split (4 min) | 38.8 | 94.1 | 61.8 | 28.0 | 10.3 | 1.0 |
| Previous Scene Segmentation Methods | ||||||
| PySceneDetect [ 7 ] | 34.92 | 90.40 | 51.04 | 22.47 | 9.06 | 1.92 |
Table D : Scene segmentation performance on NarraScene
Figure F : Qualitative example on MovieStory101 Movie QA. We compare retrieval results produced by different chunking strategies on a multiple-choice movie question from Chasing The Dragon . The figure shows the oracle evidence clip together with the top retrieved chunk from each method and the corresponding QA prediction. Uniform chunking retrieves a clip that better matches the annotated evidence, whereas scene-based baselines may retrieve distractor clips that lead to incorrect answers or fallback predictions.
Figure G : Qualitative example on MAD Movie Temporal Grounding. We compare retrieval and temporal grounding outputs under different chunking strategies for a query from Fargo . The figure shows the ground-truth timeline together with the retrieved chunk and predicted temporal span from each method. Uniform chunking retrieves a candidate that more tightly covers the target moment, while scene-based baselines produce less accurate candidates and correspondingly weaker temporal localization.