How Well Do Vision-Language Models Understand Movies?
Authors: Emmanouil Zaranis, António Farinhas, Saul Santos, Beatriz Canaverde, Miguel Moura Ramos, Aditya K Surikuchi, André Viveiros, Baohao Liao, Elena Bueno-Benito, Nithin Sivakumaran, Pavlo Vasylenko, Shoubin Yu, Sonal Sannigrahi, Wafaa Mohammed, Ben Peters, Danae Sánchez Villegas, Elias Stengel-Eskin, Giuseppe Attanasio, Jaehong Yoon, Stella Frank, Alessandro Suglia, Chrysoula Zerva, Desmond Elliott, Mariella Dimiccoli, Mohit Bansal, Oswald Lanz, Raffaella Bernardi, Raquel Fernández, Sandro Pezzelle, Vlad Niculae, André F. T. Martins
Organizations: Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ILLC, University of Amsterdam · Language Technology Lab, University of Amsterdam · Institut de Robòtica i Informàtica Industrial, CSIC-UPC · UNC Chapel Hill · University of Copenhagen · Pioneer Center for AI · Heriot-Watt University · ELLIS Unit Lisbon · ELLIS Unit Barcelona · Free University of Bozen-Bolzano · ELLIS Unit Trento · ELLIS Unit Amsterdam · Unbabel
Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos--but can they understand the narrative structure of movies as humans do? To investigate this question, we introduce MF2 (Movie Facts and Fibs), a fully manually curated benchmark designed to evaluate whether models can identify, recall, and comprehend information central to the movie's storyline, using both the visual and textual modalities. MF2 includes over 50 open-licensed movies, each paired with manually constructed sets of claim pairs: one true claim (fact) and one plausible but false (fib), totaling over 850 pairs. Our analysis across 20 open-source and 2 proprietary models reveals a substantial gap relative to human performance. While subtitles provide a strong signal for narrative understanding, other textual sources (e.g., synopses) are less informative. In contrast, when visual information is distributed across the entire movie and jointly embedded with subtitles, models struggle to identify and effectively use the relevant content. However, when the visual input is restricted to claim-relevant visual content, models improve significantly, indicating substantial room for improvement in processing large multimodal contexts and distinguishing relevant from irrelevant information. Overall, our findings highlight persistent challenges in multimodal narrative understanding for current VLMs. We release our data and code to support further research in this direction.
Figures & tables
Figure 1: Illustration of two claim pairs (each with a fact and a fib ) from the movie “The Little Princess” (1939). Our claims target memorable events, focusing on key points of the narrative such as emotional arcs and causal relationships between characters, and require reasoning across different granularities ( single-scene and multi-scene/global ).
Figure 2: Dataset construction process involving three main stages: movie collection, data annotation, and quality control.
Method
#Params
Pairwise Accuracy (%)
Accuracy (%)
Subs
Video
Video+Subs
Subs
Video
Video+Subs
Baselines
Random
-
25.0
25.0
25.0
50.0
50.0
50.0
Human
-
–
–
84.1
–
–
90.5
Closed Models
GPT-4o
-
48.8
19.5
49.8
72.9
55.5
73.2
Table 1: Performance (%) of open-weight and closed models on MF 2 under different input modalities. Best values overall are bolded ; best per group are underlined . Asterisk (*) denotes models that failed to follow the instruction prompt in a substantial fraction of samples and their results are therefore not reported. Numbers in parentheses indicate the number of active parameters.
Model
Video-only
Video+Subs
(full)
(targeted)
(full)
(targeted)
Gemini 2.5 Pro
38.6
49.4 (+10.8)
59.4
74.0 (+14.6)
Gemma3-27B
27.2
36.9 (+9.7)
42.7
49.5 (+6.8)
Qwen3VL-30B
12.6
22.3 (+9.7)
40.8
52.4 (+11.6)
InternVL3.5-38B
29.1
41.7 (+12.6)
51.5
53.4 (+1.9)
Qwen2.5VL-72B
34.0
29.1 (-4.9)
46.6
53.4 (+6.8)
Table 2: Pairwise accuracy when models are provided with “targeted” frames as visual input versus frames sampled uniformly from the entire movie. “Targeted” frames refer to frames extracted from the annotated time intervals that contain the visual content relevant to each claim. Results are reported on a single-scene claim subset under video-only and video with subtitles settings. Values in parentheses indicate the performance difference relative to the full-movie setting. Positive and negative deltas denote performance gains and drops, respectively.
Input Modality
Metric
Video
Subs
Video+Subs
Synopsis
Movie Title
Pairwise Acc.
37.9
58.9
63.8
27.9
45.4
Table 3: Performance of Gemini 2.5 Pro across different input modalities. Video uses only the visual stream; Subs includes only subtitle information; Synopsis relies only on the synopsis of the movie obtained from Wikipedia; Video+Subs combines both visual and subtitles inputs; and Movie Title uses only the claim, along with the movie title and release year, without access to movie content.
Figure 3: Pairwise accuracy for Gemini 2.5 Pro per comprehension dimension and reasoning granularity when varying the input modalities.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Number of claim pairs per movie.
Comprehension Dimension
Definition
Event/Entity Understanding
Involves identifying key entities (e.g., people, places, or objects) and understanding the events they participate in. This includes tracking entities across scenes, interpreting their roles, and recognizing their interactions and relationships throughout the narrative.
Temporal Perception
Requires reasoning about the timeline of events—determining whether actions occur before, after, or simultaneously—and may also include counting or sequencing events. The focus is on broader temporal relationships within the narrative.
Emotion Understanding
Involves recognizing the emotional states of characters and interpreting how these emotions evolve throughout the story.
Causal Reasoning
Focuses on identifying cause-and-effect relationships between events or actions, including both explicit and implicit dependencies that may span multiple scenes.
Appendix
Table 4: Definitions of comprehension dimensions.
Model
Pairwise Acc
Average Acc
Acc on Facts
Acc on Fibs
Video
Gemini 2.5 Pro
37.9
64.4
47.4
81.5
Gemma3-27B
31.5
61.2
57.4
65.1
Qwen3VL-IT-30B
19.2
56.9
24.3
89.5
InternVL3.5-38B
26.6
60.0
38.8
81.1
Qwen2.5VL-72B
29.7
58.8
45.4
72.1
Appendix
Table 5: Reporting pairwise accuracy, average accuracy, accuracy on facts and accuracy on fibs for some open-weight models and Gemini2.5. Results highlight that simple accuracy on its own can be misleading.
Model
Movie Title
Synopsis
Subs
Video
Video+Subs
Gemini 2.5 Pro
45.4
27.9
58.9
37.9
63.8
Gemma3-27B
28.1
29.6
42.3
31.5
39.7
Qwen3VL-IT-30B
12.9
26.6
33.5
19.2
43.8
InternVL3.5-38B
27.0
31.8
47.5
26.6
46.2
Qwen2.5-VL-72B
15.2
25.7
37.6
29.7
47.5
LLaVA-Video-72B
17.1
38.7
41.7
15.6
41.8
Appendix
Table 6: Extended version of Table 3 reporting pairwise performance for Gemini 2.5 Pro and mid- to large-scale open-weight models across different input modalities. Video uses only the visual stream; Subs includes only subtitle information; Synopsis relies only on the synopsis of the movie obtained from Wikipedia; Video+Subs combines both visual and subtitles inputs; and Movie Title uses only the claim, along with the movie title and release year, without access to movie content.
Figure 5: Pairwise accuracy for large-scale models with video and subtitles, and human baseline per comprehension dimension and reasoning granularity.
Model
#Params
#Frames
HF Checkpoint
Closed Models
GPT-4o ( OpenAI et al., 2024 )
-
50
–
Gemini 2.5 Pro ( Team et al., 2023 )
-
120
–
Open-weight Models
mPLUG-Owl3* ( Ye et al., 2025a )
7B
–
mPLUG/mPLUG-Owl3-7B-240728
VideoLLaMA3 ( Zhang et al., 2025 )
7B
180
DAMO-NLP-SG/VideoLLaMA3-7B
Appendix
Table 7: Overview of the evaluated vision-language models, including parameter scale, number of input frames, and hf checkpoint.
Figure 6: Guidelines provided for the data annotation procedure (Part 1).
Figure 7: Guidelines provided for the data annotation procedure (Part 2).
Figure 8: Guidelines provided for the data annotation procedure (Part 3). This part of the guidelines provides examples given to annotators to illustrate how to assign reasoning granularity labels. While more examples were shared during the annotation process, we include a selection here for illustrative purposes.
Figure 9: Guidelines provided for the data annotation procedure (Part 4). This part of the guidelines provides examples given to annotators to illustrate how to assign comprehension dimension labels. While more examples were shared during the annotation process, we include a selection here for illustrative purposes.
Figure 10: Guidelines provided for human evaluation (Part 1).
Figure 11: Guidelines provided for human evaluation (Part 2).
Figure 12: Direct prompt template used for open-weight models.
Figure 13: Explanation prompt template used for closed models.
Movie (Year)
Genre (IMDB)
Lang.
Duration
The Last Chance (1945)
Drama, War
en, it
93.84
They Made Me a Criminal (1939)
Boxing, Film Noir, Crime, Drama, Sport
en
91.21
Tokyo After Dark (1959)
Drama
en
81.23
The Sadist (1963)
Horror, Thriller
en
91.63
Suddenly (1954)
Film Noir, Psychological Thriller, Crime, Drama, Thriller
en
76.71
Sabotage (Hitchcock) (1936)
Psychological Thriller, Spy, Crime, Thriller
en
75.92
Appendix
Table 8: Metadata of collected movies (Part 1).
Movie (Year)
Genre (IMDB)
Lang.
Duration
The Emperor Jones (1933)
Drama, Music
en
76.29
The Deadly Companions (1961)
Adventure, Drama, Western
en
93.62
The Red House (1947)
Film Noir, Drama, Mystery, Thriller
en
100.39
Trapped (1949)
Film Noir, Crime, Drama, Thriller
en
79.4
City of Fear (1959)
Crime, Drama, Thriller
en
75.18
Kid Monk Baroni (1952)
Action, Drama, Sport
en
79.56
Appendix
Table 9: Metadata of collected movies (Part 2).
Example 1 (Part 1) - Movie: "Suddenly"
Pidge brings the water Setting up the circuit Pop spills the water The electrocution
Fact ( ✓ ): The grandfather, Peter Benson, intentionally spilled the water on the floor.
Fib ( ✗ ): The grandfather, Peter Benson, accidentally spilled the water on the floor.
Granularity : Multi-scene/Global
Appendix
Table 10: Illustrative Example 1 from the movie “Suddenly” . Gemini fails to predict both claims correctly in every modality setting.
Example 1 (Part 2) - Movie: "Suddenly"
Video only setting
Prediction on Fact : The grandfather, Peter Benson, is seen at 2:17 reaching for a glass of water. He then appears to deliberately drop the glass, causing the water to spill on the floor. This creates a distraction, allowing the woman to crawl towards a gun. The context of the scene, where the family is being held hostage and looking for an opportunity to fight back, combined with the grandfather’s actions and the subsequent events, strongly suggests the spill was intentional. TRUE (Ground-truth: TRUE)
Prediction on Fib : The video introduces an older character who is later identified as Peter Benson (around 0:46-0:47). Much later in the video, during a tense hostage situation, Peter Benson is seen sitting in an armchair holding a glass of water. At timestamp 2:18, he is startled and gets up abruptly, causing the water in his glass to spill onto the floor. The context of the scene, where he reacts to the unfolding events, suggests the spill was accidental due to his sudden movement or reaction. TRUE (Ground-truth: FALSE)
Appendix
Table 11: Illustrative Example 1 from the movie “Suddenly” . Gemini fails to predict both claims correctly in every modality setting.
Example 2 (Part 1) - Movie: "Sabotage (Hitchcock)"
Aquarium Planning the sabotage Meet in cinema Kid enters the bus The time arrives Bus explosion People in the cinema Explosion in cinema
Fact ( ✓ ): The second bomb exploded in the cinema.
Fib ( ✗ ): There was a second bomb about to explode, but it didn’t.
Granularity : Global
Context ( Spoiler alert ! ) : Mr. Verloc and the terrorist meet at the aquarium, where the terrorist first mentions the plan to explode a bomb (he mentions “a parcel of fireworks” and asks Mr. Verloc to meet his friend the bombmaker). The terrorists meet again in the cinema, to discuss about the sabotage. Later in the movie the explosion happens inside the bus. While the second explosion happens inside the cinema. Gemini fails to predict both claims correctly in every modality setting.
Appendix
Table 12: Illustrative Example 2 from “Sabotage (Hitchcock)” . Gemini predicts the pair of claims incorrectly in every modality setting.
Example 2 (Part 2) - Movie: "Sabotage"
Prediction on Fib : The transcript describes two main bomb-related incidents. The first is the explosion that kills Stevie. The second occurs at the cinema where the old man (the bomb maker) threatens to “blow the place sky high.” The transcript later confirms an explosion there: “The roof’s gone. The whole of the back of the place - clean blown off.” This indicates the second bomb did explode. Therefore, the statement that a second bomb was “about to explode, but it didn’t” is false because it did explode. FALSE (Ground-truth: FALSE)
Video-only setting
Prediction on Fact : The video shows one clear explosion involving a bus on a street around the 1:46 mark. Following this, there are various scenes, including one where people are watching a cartoon in a cinema. However, there is no depiction or indication of a second bomb exploding, nor is there any explosion shown occurring inside the cinema. FALSE (Ground-truth: TRUE)
Appendix
Table 13: Illustrative Example 2 from “Sabotage (Hitchcock)” . Gemini predicts the pair of claims incorrectly in every modality setting.
Example 2 (Part 3) - Movie: "Sabotage"
Video w/ subs setting
Prediction on Fact : The first act of sabotage was the power station blackout. The second planned act of sabotage, referred to as “a parcel of fireworks”, was to be left in the cloakroom at Piccadilly Circus underground station […] This parcel, carried by Stevie on a bus, explodes on the bus at 1:45. A newspaper headline later confirms “WEST END BOMB SENSATION. Fully-laden bus was blown up in Piccadilly this afternoon”. […] An explosion then occurs at the cinema, as described by an officer: “The roof’s gone. The whole of the back of the place - clean blown off”. This is a separate, third bomb event. Therefore, the second bomb (the one Stevie was carrying) exploded on the bus, not in the cinema. The explosion in the cinema was a distinct event caused by a different bomb wielded by the bird shop owner. FALSE (Ground-truth: TRUE)
Prediction on Fib : The statement claims: “There was a second bomb about to explode, but it didn’t.” Let’s analyze the bombs in the film: 1. The first act of sabotage is the power station blackout. While not explicitly shown as a bomb explosion, it is an act of “wilful destruction.” 2. The second major plot device is the bomb intended for Piccadilly Circus on Lord Mayor’s Show day. Verloc gives this bomb to Stevie. This bomb explodes prematurely on the bus, killing Stevie and many others […]. So this bomb did explode. 3. The third instance involving a bomb is at the cinema, where the Professor (the bomb maker) is cornered by the police. [..] This bomb does explode […]. In all instances where a bomb is primed or ready for its purpose within the narrative, it either explodes (bus bomb, Professor’s bomb in the cinema) or fulfills its sabotage (power station). There is no scene in the film where a bomb is “about to explode” and then “didn’t.” The bombs that are set up to go off, do go off. Therefore, the statement is false. FALSE (Ground-truth: FALSE)
Appendix
Table 14: Illustrative Example 2 from “Sabotage (Hitchcock)” . Gemini predicts the pair of claims incorrectly in every modality setting.
Example 3 (Part 1) - Movie: "Scandal Sheet"
News about the killer Sales increasing.. Sales increasing..
Fact ( ✓ ): The sales of the newspaper increased after it was revealed that the editor was the killer.
Fib ( ✗ ): The sales of the newspaper decreased after it was revealed that the editor was the killer.
Granularity : Single-scene
Appendix
Table 15: Illustrative Example 3 from “Scandal Sheet” . The pair of claims relies primarily on visual information. Gemini predicts them incorrectly when given only a single modality, but when provided with both the transcript and the video, it correctly identifies the pair of claims.
Example 3 (Part 2) - Movie: "Scandal Sheet"
Video only setting
Prediction on Fact : The video shows a chart of the “NET PAID CIRCULATION NEW YORK EXPRESS” at various points. Initially, at 0:10, the circulation is shown with an upward trend. After the editor is revealed to be the killer and is subsequently shot (around 2:26-2:28), the final shot of the movie (2:29) shows a newspaper headline “KILLER!!” with the byline “By Steve McCleary” superimposed over the same circulation chart. At this point, the line on the chart indicating sales has reached its highest point, implying a significant increase in sales coinciding with or immediately following the revelation and capture of the killer editor. TRUE (Ground-truth: TRUE)
Prediction on Fib : The movie shows a chart of the “New York Express” net paid circulation at 0:10, indicating an upward trend in sales from October to September, reaching 650,000. Later, after the editor (the man standing by the chart at 0:10) is revealed as the killer and is shot (around 2:27), the final shot at 2:29 shows a newspaper headline “KILLER!!” superimposed over the same circulation chart. The numbers and the upward trend on this chart are identical to the one shown at 0:10. The movie does not provide any information or visual evidence suggesting that the newspaper’s sales decreased after the editor was revealed as the killer. The chart displayed at the end is the same one that showed increasing sales. TRUE (Ground-truth: FALSE)
Appendix
Table 16: Illustrative Example 3 from “Scandal Sheet” . The pair of claims relies primarily on visual information. Gemini predicts them incorrectly when given only a single modality, but when provided with both the transcript and the video, it correctly identifies the pair of claims.
Example 4 (Part 1) - Movie: "Tight Spot"
Fact ( ✓ ): When Hallet shows Conley a photograph of an individual who was on the yacht, she immediately recognizes the person.
Fib ( ✗ ): When Hallet shows Conley a photograph of an individual who was on the yacht, she claims not to recognize the person.
Granularity : Single-scene
Context ( Spoiler alert ! ): This pair of claims to be answered correctly requires both the visual and text cues. Hallet shows Conley a photograph (visual information) and asks whether she recognizes the person (Senior Maracelli). Conley recognizes the person. When the model is given only the subtitles, it misclassifies both the fact and the fib (as visual cues are absent). In contrast, when provided with both the video and the subtitles, it predicts each one correctly.
Appendix
Table 17: Illustrative Example 4 from “Tight Spot” . Gemini fails to predict both claims correctly when given only the subtitles or only the visual cues. However, when provided with both modalities together, it correctly predicts both claims.
Example 4 (Part 2) - Movie: "Tight Spot"
Video only setting
Prediction on Fact : The statement asks whether Conley immediately recognizes the person in the photograph shown to her by Hallet. In the video, at 00:30, Hallet takes out a photograph. At 00:31, he shows it to Conley. Conley looks down at the photo, then at 00:32, she looks up at Hallet with an expression that seems more contemplative or questioning rather than one of instant, clear recognition. She then glances back at the photo before looking at Hallet again and starting to speak at 00:33. Her reaction involves a pause and a sequence of looks, which does not convey immediate recognition. “Immediate” suggests an instantaneous, reflexive understanding, which is not evident in her more measured response. FALSE (Ground-truth: TRUE)
Prediction on Fib : The statement claims that when Hallet shows Conley a photograph of an individual from a yacht, Conley claims not to recognize the person. Observing the video, there is a scene from approximately 0:29 to 0:36 where a man (presumably Hallet, played by Edward G. Robinson) shows a photograph to a woman (presumably Conley, played by Ginger Rogers). The woman looks at the photograph (0:31-0:32). However, the video segment does not include any audible dialogue or clear visual cue (like shaking her head definitively or mouthing “I don’t know”) from the woman that would confirm she “claims not to recognize the person.” Her facial expression is somewhat pensive or surprised but not an explicit denial of recognition. She then proceeds to make a phone call (0:34). The context that the photograph is of “an individual who was on the yacht” is also not explicitly provided by the visual information in this clip. Since the crucial part of the statement - her claiming not to recognize the person - is not depicted or audible in the provided video evidence, the statement cannot be verified as true based on the available information. FALSE (Ground-truth: FALSE)
Appendix
Table 18: Illustrative Example 4 from “Tight Spot” . Gemini fails to predict both claims correctly when given only the subtitles or only the visual cues. However, when provided with both modalities together, it correctly predicts both claims.
Example 5 (Part 1) - Movie: "Tight Spot"
Pete in taxi Walking the stairs of the church Pete being shot
Fact ( ✓ ): Pete Tinelli was shot.
Fib ( ✗ ): Pete Tinelli was poisoned.
Granularity : Multi-scene
Appendix
Table 19: Illustrative Example 5 from “Tight Spot” . Gemini fails to predict the pair of claims correctly, in every modality setting.
Example 5 (Part 2) - Movie: "Tight Spot"
Video only setting
Prediction on Fact : A man is visibly shot around the 1:57-1:58 mark in the video. In this scene, a man is held by two others, another man (resembling Edward G. Robinson) nods, a gunshot sound is heard, and the man being held slumps. However, throughout the provided video clip, there is no dialogue, on-screen text, or any other information that identifies the man who was shot as “Pete Tinelli.” […] FALSE (Ground-truth: TRUE)
Prediction on Fib : The video contains several scenes that lead to the conclusion that Pete Tinelli was poisoned. At 1:58, a group of men are shown in what appears to be a garage, and one man is forcibly made to drink something. He subsequently collapses at 2:01. Later, at 2:05, a woman asks, “What happened to Pete Tinelli?” and is told, “He won’t bother you anymore.” Shortly after, at 2:08, one man informs another, “Pete Tinelli’s been taken care of.” When asked “How?”, the response is “Poison.” This sequence of events and dialogue directly confirms that Pete Tinelli was poisoned. TRUE (Ground-truth: FALSE)
Appendix
Table 20: Illustrative Example 5 from “Tight Spot” . Gemini fails to predict the pair of claims correctly, in every modality setting.
Example 6 (Part 1) - Movie: “A Bucket of Blood”
Holding the cat sculpture Drops it The damage Moment of realization
Fact ( ✓ ): The gallery owner dropped the cat sculpture, causing minor damage.
Fib ( ✗ ): The gallery owner dropped the cat sculpture, but it remained intact.
Granularity : Single-scene
Appendix
Table 21: Illustrative Example 6 from the movie: “A Bucket of Blood” . Gemini fails to predict the pair of claims correctly, in every modality setting.
Example 6 (Part 2) - Movie: “A Bucket of Blood”
Video-only setting
Prediction on Fact : The statement is: “The gallery owner dropped the cat sculpture, causing minor damage.” To verify this, we need to identify a “gallery owner,” a “cat sculpture,” an act of “dropping” by the gallery owner, and resulting “minor damage.” 1. Gallery Owner: * Leonard de Santis is the owner of "The Yellow Dog" cafe, which also functions as an art gallery where poetry is read and art is displayed […] Walter Paisley creates a sculpture of a cat by encasing his landlady’s dead cat in clay (0:16-0:20). This is the most prominent cat sculpture. * At 0:29, Leonard and Carla are seen holding other, more stylized white cat sculptures, indicating these might be common art pieces in their circle. 3. Dropped the cat sculpture, causing minor damage: * At 0:29, Leonard (the cafe/gallery owner) is shown holding one of the stylized white cat sculptures. He does not drop it. […] No scene in the provided video shows a character identifiable as "the gallery owner" dropping a cat sculpture and causing minor damage. The closest event involves a police officer tapping Walter’s cat sculpture, causing a tiny chip. Therefore, the statement is false. FALSE
Prediction on Fib : The statement says: “The gallery owner dropped the cat sculpture, but it remained intact.” Let’s break this down: 1. Identity of the “gallery owner”: In the movie clip, the gallery owner is Maxwell Brock, often seen wearing a beret or a suit, and he runs “The Yellow Door” gallery. 2. Did the gallery owner drop a cat sculpture?: * At 0:27, the main character, Walter Paisley (a busboy who becomes an artist), is carrying a large abstract sculpture (not a cat) when he is bumped by Carla. Walter drops this sculpture, and it shatters. Walter is not the gallery owner. * Walter Paisley creates a sculpture of a cat (by encasing his landlady’s dead cat in clay) around 0:22. This sculpture is later handled but not dropped by the gallery owner. * At 0:50, Maxwell Brock (the gallery owner) is seen handling a different white, stylized cat sculpture. He picks it up, examines it carefully, and places it back down. He does not drop this sculpture. Since there is no instance in the provided video where the gallery owner (Maxwell Brock) drops any cat sculpture, the first part of the statement ("The gallery owner dropped the cat sculpture") is false. Therefore, the entire statement is false. FALSE
Appendix
Table 22: Illustrative Example 6 from the movie: “A Bucket of Blood” . Gemini fails to predict the pair of claims correctly, in every modality setting.
Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning about film history and the evolution of narrative techniques. But the creation of stable benchmarks built around Hollywood films is complicated by copyright protections. In this work, we address these concerns directly, by building a new collection of Hollywood films defined by two criteria: box office popularity (where we publish the first large-scale, open collection of weekly box office earnings reported by Variety magazine from 1922-1979); and likely public domain status (by researching copyright registrations and renewals in the US Catalog of Copyright Entries). We build a new multimodal MCQ benchmark on top of this collection that focuses on narrative elements that directly evaluate the abilities of models to inform meaningful research on film narrative; we find that many vision-language models struggle on this task (with many performing at near-chance levels of accuracy), while audio-visual models (including those that use audio in captioning scenes) reach a maximum accuracy of 61.1%, well below human-level performance.
David Bamman, Kent K. Chang, Allison Cooper +7
School of Information, UC Berkeley · Cinema Studies, Bowdoin College · UC Berkeley +1
Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotionally engage with video content. While Large Vision Language Models (LVLMs) have made remarkable progress in video question answering, existing benchmarks primarily focus on identifying low-level techniques rather than understanding their storytelling impact. To address this, we introduce CinematicVQA, the first-of-its-kind benchmark for cinematic video understanding that goes beyond technique recognition to evaluate film-grammar reasoning, utilizing our introduced Cinematic Scene Graph (CSG), a structured representation that links filming techniques to their perceptual effects and narrative functions. Through comprehensive evaluation of state-of-the-art LVLMs, we reveal a striking semantic gap: models consistently perform higher on describing visual presentations than on identifying the underlying techniques. Surprisingly, Chain-of-Thought prompting fails to provide consistent gains and degrades performance for most models, suggesting that current LVLMs lack sufficient cinematic domain knowledge to benefit from step-by-step reasoning. Fine-tuning on CinematicVQA-train yields consistent improvements, particularly for narrative function and multi-hop reasoning. Overall, CinematicVQA serves both as a rigorous benchmark for cinematic evaluation in LVLMs and as a practical dataset for training more film-aware video models.
Shuo Xing, Pooja Verlani, Balu Adsumilli +1
Texas A&M University, College Station, USA · Google Inc., Mountain View, USA
Recent progress in multimodal large language models (MLLMs) has led to a surge of benchmarks for long-video reasoning. However, most existing benchmarks rely on localized cues and fail to capture narrative reasoning, the ability to track intentions, connect distant events, and reconstruct causal chains across an entire movie. We introduce NA-VQA, a benchmark designed to evaluate deep temporal and narrative reasoning in long-form videos. NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far to assess long-range dependencies. By requiring generative, multi-scene answers, NA-VQA tests whether models can integrate dispersed narrative information rather than rely on shallow pattern matching. To address the limitations of existing approaches, we propose Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured memory for retrieval during reasoning. Extensive experiments show that state-of-the-art MLLMs perform poorly on questions requiring far-range evidence, highlighting the need for explicit narrative modeling. Video-NaRA improves long-range reasoning performance by up to 3 percent, demonstrating its effectiveness in handling complex narrative structures. We will release NA-VQA upon publication.