Evaluating Multimodal Narrative Understanding of Popular Hollywood Films
Authors: David Bamman, Kent K. Chang, Allison Cooper, Juishan Hsu, Reina Kushihashi, Madison Mar, Arnav Podichetty, Rachael Samberg, Ipek Nil Sancak, Yuhan Shao
Organizations: School of Information, UC Berkeley · Cinema Studies, Bowdoin College · UC Berkeley · Scholarly Communication and Information Policy, UC Berkeley
Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning about film history and the evolution of narrative techniques. But the creation of stable benchmarks built around Hollywood films is complicated by copyright protections. In this work, we address these concerns directly, by building a new collection of Hollywood films defined by two criteria: box office popularity (where we publish the first large-scale, open collection of weekly box office earnings reported by Variety magazine from 1922-1979); and likely public domain status (by researching copyright registrations and renewals in the US Catalog of Copyright Entries). We build a new multimodal MCQ benchmark on top of this collection that focuses on narrative elements that directly evaluate the abilities of models to inform meaningful research on film narrative; we find that many vision-language models struggle on this task (with many performing at near-chance levels of accuracy), while audio-visual models (including those that use audio in captioning scenes) reach a maximum accuracy of 61.1%, well below human-level performance.
Figures & tables
Figure 1: Weekly box office information from two issues of Variety , October 29, 1924 (left) and July 10, 1934 (right).
Figure 2: Example question ( His Girl Friday ): “How is the camera positioned as Hildy enters the room to talk to Earl Williams?” A .) On Hildy’s side of the bars to look through the grate at Earl. B .) The camera is at eye level and moves parallel to her. C .) On the inside of the cell to look out at Hildy. D .) High angle above Hildy and Earl.
Table 1: Accuracy on B . The largest 95% Wald confidence interval is ±3.9% . Bold indicates best overall; underline marks the within-subgroup leader significantly better than the runner-up (whose 95% CIs do not overlap).
Figure 3: Registration for Bells of St. Mary’s recorded in the Catalog of Copyright Entries, Cumulative Series 1940–1949
Figure 4: Registration renewal for Bells of St. Mary’s recorded in the Catalog of Copyright Entries, Third Series, Volume 27, Parts 12–13, Number 1: Motion Pictures (January–June 1973)
Category
Exclusion rate
B count
D count
Representation
0.389
22
36
Symbolism
0.241
22
29
Temporality
0.221
53
68
Plot
0.209
155
196
Object Identification
0.200
92
115
Character
0.167
135
162
Appendix
Table 4: Exclusion rate by category.
Backbone
N=64
N=128
N=256
End-to-end vision-language models
GPT-5-mini
30.6 [27.2–33.9]
32.8 [29.3–36.3]
34.2 [30.7–37.7]
Qwen3-VL-8B
25.3 [22.0–28.5]
29.8 [26.4–33.1]
28.8 [25.5–32.2]
GLM-4.1V-9B-Thinking
25.8 [22.6–29.0]
25.6 [22.5–28.8]
24.2 [21.2–27.4]
End-to-end long-video models
HourLLaVA
23.6 [20.4–26.8]
24.7 [21.5–27.9]
26.3 [23.1–29.5]
Appendix
Table 5: Frame-budget sweep N∈{64,128,256} for end-to-end vision-language and long-video backbones run at multiple input budgets.