cs.CVOct 30, 2026

LF-VideoQA: Evaluating Long-Video Understanding with Long-Form Theory-of-Mind Questions

Authors: Shaden Shaar, Bradon Thymes, Majd Aldaye, Claire Cardie, Bharath Hariharan

Organizations: Department of Computer Science Cornell University Ithaca, NY 14850, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The hardest part of understanding a long video is not pinpointing any one moment but synthesizing many, e.g., answering a question that requires tracking what develops across hours of content and articulating how the pieces fit together. To date, most long-video QA datasets require only a short answer or a multiple-choice selection, oversimplifying the long-video understanding task to retrieval. We introduce LF-VideoQA, a dataset for long-form, open-ended question answering over long narrative videos built on theory-of-mind questions: a character's belief depends on everything they have seen or been told, so each question demands assembling and reasoning from evidence distributed across the video. The dataset contains more than 6,300 questions on 94 recent films; each requires reasoning over scenes spanning an average of 23 minutes and comes with a question-specific rubric for recall and precision scoring. The dataset is created entirely automatically by annotating screenplays with scene events and each character's first-order beliefs and meta-beliefs, then generating (question, answer, rubric) triples through LLM-based question generation and verification; human studies validate their quality. We evaluate four families of state-of-the-art open-source VLMs across three settings and find the task highly challenging: (i) even an oracle with access to the screenplay falls well short of complete answers; (ii) the best non-oracle model attains barely half of the recall level of the oracle; (iii) unlike on prior long-video benchmarks, captions and subtitles underperform the video itself, indicating that the questions need visual evidence; and (iv) recall is the predominant bottleneck, with answers omitting many required rubric claims.

Figures & tables

Figure 1: Dataset Construction on an Example. This figure shows (question, answer, rubric) from Deadpool & Wolverine (2024). Preprocessing: screenplay scenes are time-stamped against the subtitles and decomposed into scene events. Character belief annotation: scene by scene, we extract and verify each character’s first-order beliefs and meta-beliefs. Question generation and verification: for a center character (Wolverine) and an anchor (scene 113), we generate the triple and verify every rubric claim against the screenplay.
Question typennCore claimsContext claimsQuestion tokensAnswer tokensClaim tokensCited scenesSpan (min)Anchor start end
Q1 Knowledge split2063.01.32377143.17 
Q2 Realization8571.51.82075174.124 
Q3 False belief3251.42.12073143.412 
Q4 Knowledge source1,4631.61.22069163.05 
Q5 First-order belief1281.31.32165163.29 
Q6 Second-order belief1121.21.22361173.05 
Table 1: Statistics per question type. Belief-state types ask about one moment; arc types ask about the whole film. All columns except nn are averages per question; span is the runtime between the earliest and latest cited scene; the last column shows where questions are anchored.
 OverallPer film
Films94–
Runtime182 h112 min
Scenes11,712118
Questions6,32760
Verified claims24,496250
Unique Characters1,62515.5
Table 2: Dataset Size
  LexicalLLM judge
SourceModelRLClmRL[email protected]Hedge ↓\downarrowRecPrecF1Contra ↓\downarrow
Gold answer1.000.686.975.007––––
Text-based
ScreenplayQwen3.6 35B/A3B.160.248.348.343.472.667.505.128
Qwen3.8 27B ⋆.170.243.325.216.509.703.544.103
Gemma4 26B/A4B.164.220.287.519.260.464.290.100
Table 3: Results on LF-VideoQA . Rows are answerer models; in the video-captioning answerer also write the captions. Rec, Prec, F1, and Contra follow Section 3.3 . Blue: five best values across all runs; red: five worst values of Hedge and Contra. Bold : highest value within a source.
  Belief & knowledgeNarrative 
ContextCaptioner / tierQ1Q2Q3Q4Q5Q6Q7Q8N1N2N3All
Screenplay–.64 ±\pm .05.39 ±\pm .03.61 ±\pm .04.60 ±\pm .03.31 ±\pm .10.20 ±\pm .07.61 ±\pm .02.33 ±\pm .03.35 ±\pm .03.49 ±\pm .04.55 ±\pm .03.50 ±\pm .01
Subtitles–.22 ±\pm .04.10 ±\pm .02.27 ±\pm .04.18 ±\pm .02.09 ±\pm .05.09 ±\pm .05.20 ±\pm .03.13 ±\pm .02.16 ±\pm .02.16 ±\pm .03.14 ±\pm .03.16 ±\pm .01
Video CaptioningQwen3.6 35B/A3B.02 ±\pm .01.01 ±\pm .01.00 ±\pm .01.01 ±\pm .01.00 ±\pm .00.00 ±\pm .00.01 ±\pm .00.02 ±\pm .01.04 ±\pm .01.03 ±\pm .01.03 ±\pm .01.02 ±\pm .00
Qwen3.8 27B.01 ±\pm .01.01 ±\pm .01.01 ±\pm .01.01 ±\pm .01.00 ±\pm .00.00 ±\pm .00.00 ±\pm .00.01 ±\pm .01.02 ±\pm .01.02 ±\pm .01.02 ±\pm .01.01 ±\pm .00
Gemma4 26B/A4B.01 ±\pm .01.00 ±\pm .01.00 ±\pm .00.01 ±\pm .01.00 ±\pm .00.00 ±\pm .00.00 ±\pm .00.01 ±\pm .01.01 ±\pm .01.02 ±\pm .01.01 ±\pm .01.01 ±\pm .00
Table 4: Qwen3.6 across question types. The answerer is always Qwen3.6; rows vary the source and the captioner. Cells are F1 (w/ 95% bootstrap interval). Blue: five highest per question type, darker for higher; bold : highest.
Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

BenchmarkVideo lengthFull filmsOpen-endedToMQ lenA len
General long video
ActivityNet-QA ( Yu et al., 2019 )3 min×\times✓\checkmark×\times8.71.9
NExT-QA ( Xiao et al., 2021 )44 s×\times✓\checkmark×\times11.62.6
EgoSchema ( Mangalam et al., 2023 )3 min×\times×\times×\times––
Video-MME ( Fu et al., 2024 )17 min×\times×\times×\times12.75.5
LongVideoBench ( Wu et al., 2024 )7.9 min×\times×\times×\times43.58.3
Appendix
Table 5: LF-VideoQA beside existing video QA benchmarks , as reported by each paper. Full films : questions are asked over entire feature films ( ∼\sim : clips or selected snippets of films, or a mix of films and shorter videos). Open-ended : answers are free text ( ∼\sim : some tasks). ToM : questions target characters’ mental states ( ∼\sim : some question categories). Q len and A len are mean words; where answers are not open-ended, A len is the length of the correct option; –: not reported. † Tokens, as reported by DeepMovieQA. Each LF-VideoQA question draws on 4.2 scenes spanning 23 minutes on average, and in o200k tokens its questions and answers average 21 and 75 (Table 1 ).
Figure 2: A knowledge-split (Q1) example from Sinners (2025), anchored at S82 with Annie as center character. At 01:18:22 Sammie glimpses Stack and Mary in the back room and tells the others that Stack is busy; Smoke, Delta Slim and Annie adopt the same false belief. Each then revises it at a different moment, depending on what they witness: Smoke when he sees Mary feeding (01:19:22), Delta Slim when she runs past him (01:20:03), Annie and Sammie when Stack dies (01:20:08). Meta-beliefs track who knows that Sammie missed each step. The question asks for this knowledge split at the anchor scene; its rubric separates core claims a correct answer must contain from supporting context claims.
Figure 3: A second-order belief (Q6) example from Oppenheimer (2023), whose center character is Strauss’s Senate aide and whose anchor scene is S261. At 00:11:50 Strauss watches Einstein walk past him after a private word with Oppenheimer and concludes that Einstein disapproves of him; by 02:41:48 this has hardened into the false belief that Oppenheimer turned Einstein against him. The aide models that belief (meta-beliefs, bottom lane), suggests a better explanation that Strauss rejects, and the film reveals what was actually said only at the very end. Strauss never revises the belief. The question asks what the aide takes Strauss to believe, a belief about a false belief.
Figure 4: An accepting-a-claim (Q7) example from Barbie (2023), with Sasha as center character and S51, Gloria’s car, as anchor scene. Told that a sad girl is causing her malfunction, Barbie misses the scene where Gloria shows her sad sketches (00:35:36) and wrongly concludes that the girl is Sasha; she revises only when Gloria explains in the car (00:52:19). Sasha hears the same true account and rejects it, while her own meta-belief records that Barbie accepts it. The question asks what Sasha makes of her mother’s explanation: noticing that Sasha rejects a true account, and that she is mistaken, is the task.
FieldContents
holderThe center character: the character the question is about; for Q1 a cohort of characters.
anchorThe scene the question points at, with its time interval and whether the interval is aligned to the subtitles or interpolated.
basisThe belief entries: the belief-record entries the type is defined over (Table 7 ); each cites the scene events it rests on. Meta-belief entries also carry the target character’s own beliefs at that scene, so that the question writer can see where the center character’s model of another character diverges from that character’s actual state.
scenesThe anchor scene and one scene either side, each with its summary, location and scene events with their participants and witnesses.
rosterThe roster entry of every character the belief entries or the scenes mention.
name_budgetNames the film has spoken aloud by the anchor.
Appendix
Table 6: Fields of a context-set, given to the question writer as JSON.
TypeAsksBelief entries in the context-setAnchor
Belief state — one context-set per (center character, scene)
Q1 Knowledge splitWhat each of several characters takes to be the case at one momentFor every character with a record at the scene: the events they witnessed and the false beliefs they hold; kept only if ≥3\geq 3 characters are in ≥2\geq 2 knowledge statesthe scene
Q2 RealizationWhen and through what a character’s understanding changedThe character’s revisions at the scene, each with the events that triggered itthe scene
Q3 False beliefWhat a character takes to be the case just before acting on itThe character’s false beliefs held at the scenethe scene
Q4 Knowledge sourceHow a character comes to know a factThe character’s meta-beliefs formed at the scene that carry a source (told, observed behavior, shared witnessing, absence), joined to the target’s own statethe scene
Q5 First-order beliefWhat XX takes YY to believeXX ’s first-order meta-beliefs about YY formed at the scene, joined to YY ’s own beliefs therethe scene
Appendix
Table 7: Belief entries of each question type. For Q1, the center character is a group of characters.
Judgesκ\kappaKrippendorff’s α\alpha
gpt-5.6-luna, gpt-5.6-terra0.7780.778
gpt-5.6-luna, gemini-3.8-flash0.7320.732
gpt-5.6-terra, gemini-3.8-flash0.7140.712
All three0.7420.742
Appendix
Table 8: Claim-level agreement between judges on 27,875 claim verdicts (asserted, contradicted, or omitted). κ\kappa is Cohen’s for a pair of judges and Fleiss’ for all three.
Q1 Knowledge split The Holdovers (2023) center character: Angus Tully 3 core / 5 claims evidence: 2 scenes over 10 min
question: At the Winning Ticket, what do Paul, Angus, and Miss Crane each understand about Christmas Eve? gold answer: At the Winning Ticket, Miss Crane knows that she has invited both Paul Hunham and Angus Tully to her Christmas Eve party. Paul hears the invitation. He knows it includes both Angus and himself. Angus is away from the table at the moment when Miss Crane extends that invitation, and he does not yet know about the invitation that she has made. ■\blacksquare 1 Miss Crane knows she invited Paul Hunham and Angus Tully to her Christmas Eve party. □\square 2 Paul Hunham hears Miss Crane’s invitation. ■\blacksquare 3 Paul Hunham knows Miss Crane’s invitation includes Paul Hunham and Angus Tully. □\square 4 Angus Tully is away from the table during Miss Crane’s invitation. ■\blacksquare 5 Angus Tully does not yet know about Miss Crane’s invitation. 54:46 1:03:23 ▲\blacktriangle start of film end of film, 2:11:23 evidence in the film: 54:46 The Winning Ticket (claims 1–5) 1:03:23 Kitchen (claim 5)
Q2 Realization Conclave (2024) center character: Joseph Tremblay 2 core / 5 claims evidence: 2 scenes over 71 min
question: In Casa Santa Marta’s courtyard corridor, when does Tremblay’s view of Lawrence’s methods change? gold answer: Tremblay’s view changes when Lawrence personally confronts him in the courtyard corridor. Until then, Tremblay had regarded Lawrence as guided by diplomatic protocol and courtesy. Lawrence directly demands that Tremblay withdraw and accuses him of sabotaging Adeyemi. The confrontation makes Tremblay realize that Lawrence will bypass protocol and directly attack a political rival. □\square 1 Joseph Tremblay had regarded Cardinal Lawrence as guided by diplomatic protocol and courtesy. ■\blacksquare 2 Cardinal Lawrence directly demands that Joseph Tremblay withdraw. ■\blacksquare 3 Cardinal Lawrence accuses Joseph Tremblay of sabotaging Adeyemi. □\square 4 Joseph Tremblay realizes Cardinal Lawrence will bypass diplomatic protocol. □\square 5 Joseph Tremblay realizes Cardinal Lawrence will directly attack a political rival. 6:00 1:15:19 ▲\blacktriangle start of film end of film, 1:52:57 evidence in the film: 6:00 Papal Suite / Sitting Room (claim 1) 1:15:19 Casa Santa Marta / Courtyard Corridor (claims 2–5)
Q3 False belief Nosferatu (2024) center character: Louise 3 core / 4 claims evidence: 3 scenes over 72 min
question: In the Harding nursery, what danger does Louise believe she and Clara face? gold answer: Louise believes that a monster is in the Harding nursery. She also believes that this monster threatens to kill her. Louise further believes that Clara faces the same lethal danger from the monster. The shadowy figure seen nearby, however, proves to be Ellen. Thus, in Louise’s understanding, both Louise and Clara are in danger of being killed by the monster. ■\blacksquare 1 Louise believes a monster is in the Harding nursery. ■\blacksquare 2 Louise believes the monster will kill Louise. ■\blacksquare 3 Louise believes the monster will kill Clara. □\square 4 The nearby shadowy figure proves to be Ellen. 11:40 1:21:24 ▲\blacktriangle start of film end of film, 2:05:11 evidence in the film: 11:40 Harding House. Rear Drawing Room (claim 1) 13:56 Front Drawing Room (claims 1–3) 1:21:24 Harding House. Upstairs Hallway / Nursery (claim 4)
Q4 Knowledge source Nickel Boys (2024) center character: Elwood 2 core / 2 claims evidence: 2 scenes over 4 min
Appendix
Table 9: Released questions, one per type: Belief-state types, Q1–Q4. Core claims are marked ■\blacksquare , context claims □\square ; ▲\blacktriangle marks the anchor scene.
Q5 First-order belief Oppenheimer (2023) center character: Gray 2 core / 2 claims evidence: 5 scenes over 22 min
question: During Oppenheimer’s security hearing, what does Gray understand about Robb’s view of Oppenheimer? gold answer: Gray understands that Robb doubts Oppenheimer told the whole truth. Gray therefore recognizes Robb’s distrust of Oppenheimer’s truthfulness. Gray also understands that Robb sees Oppenheimer as professionally willing to follow orders but morally inconsistent about the atomic and hydrogen bombs. In Gray’s understanding, Robb separates Oppenheimer’s willingness to perform assigned work from Oppenheimer’s differing moral positions on the two weapons. ■\blacksquare 1 Gray understands that Robb doubts Oppenheimer told the whole truth. ■\blacksquare 2 Gray understands that Robb sees Oppenheimer as professionally willing to follow orders but morally inconsistent about the atomic and hydrogen bombs. 2:25:13 2:38:29 ▲\blacktriangle start of film end of film, 2:53:24 evidence in the film: 2:25:13 Room 2022, Atomic Energy Commission (claim 1) 2:28:28 Room 2022, Atomic Energy Commission (claim 1) 2:29:39 Room 2022, Atomic Energy Commission (claim 1) 2:38:29 Room 2022, Atomic Energy Commission (claim 1) 2:45:37 Room 2022, Atomic Energy Commission (claim 2)
Q6 Second-order belief The Wild Robot (2024) center character: Fink 2 core / 3 claims evidence: 2 scenes over 5 min
question: In the lodge beside the woven nest, what does Fink take Brightbill to believe about Roz’s place in his past and family? gold answer: Fink understands that Brightbill holds Roz responsible for the accident that killed Brightbill’s biological family. Fink also understands that Brightbill no longer accepts Roz as his mother. More specifically, Fink understands that Brightbill rejects Fink’s defense that the accident was not Roz’s fault. Thus, in Fink’s view, Brightbill blames Roz for losing his original family and denies that Roz still occupies a maternal place in his family. ■\blacksquare 1 Fink understands that Brightbill holds Roz responsible for the accident that killed Brightbill’s biological family. ■\blacksquare 2 Fink understands that Brightbill no longer accepts Roz as his mother. □\square 3 Fink understands that Brightbill rejects Fink’s defense that the accident was not Roz’s fault. 35:46 ▲\blacktriangle start of film end of film, 1:41:36 evidence in the film: 35:46 Far Side Of The Lake (claim 1) 39:37 Lodge (claims 1–3)
Q7 Accepting a claim Sinners (2025) center character: Delta Slim 2 core / 4 claims evidence: 2 scenes over 6 min
question: At the railroad station, over the bottle of Chicago beer, what does Delta Slim make of Stack’s pitch? gold answer: Delta Slim does not dismiss Stack’s promised pay and beer as a bluff; he treats both parts of the offer seriously. His initial refusal reflects his commitment to play at Messengers, rather than disbelief that Stack will deliver. When Stack raises the offer, Delta Slim is won over. His performance with Sammie immediately afterward confirms that he accepted Stack’s deal. □\square 1 Delta Slim believes Stack’s promised pay and beer are genuine rather than a bluff. ■\blacksquare 2 Delta Slim initially refuses because of Delta Slim’s commitment to play at Messengers. ■\blacksquare 3 Stack’s raised offer wins Delta Slim over. □\square 4 Delta Slim’s performance with Sammie confirms that Delta Slim accepted Stack’s raised offer. 22:29 ▲\blacktriangle start of film end of film, 2:17:17 evidence in the film: 22:29 Railroad Station (claims 1–3) 25:56 Railroad Station (claims 3, 4)
Q8 Order of discovery Anatomy of a Fall (2023) center character: Sandra 3 core / 3 claims evidence: 4 scenes over 119 min
Appendix
Table 10: Released questions, one per type: Belief-state types, Q5–Q8. Core claims are marked ■\blacksquare , context claims □\square ; ▲\blacktriangle marks the anchor scene.
N1 Epistemic arc Killers of the Flower Moon (2023) center character: Byron Burkhart 3 core / 4 claims evidence: 13 scenes over 171 min
question: At the deserted-highway meeting, how has Byron’s knowledge of Hale’s operation developed? gold answer: Byron is included in Hale’s plan to obtain Osage headrights by having Ernest marry Mollie. Byron also sees Ernest gamble stolen jewels and lose them. Later, Byron knows Mollie is being poisoned through injections. His involvement extends beyond awareness of the operation: Byron knowingly helps Kelsie murder the incapacitated Anna. These moments show Byron progressing from inclusion in Hale’s financial scheme to knowledge of poisoning and direct participation in murder. ■\blacksquare 1 Byron is included in Hale’s plan to obtain Osage headrights by having Ernest marry Mollie. □\square 2 Byron sees Ernest gamble stolen jewels and lose them. ■\blacksquare 3 Byron knows Mollie is being poisoned through injections. ■\blacksquare 4 Byron knowingly helps Kelsie murder the incapacitated Anna. 7:59 23:35 47:00 1:12:03 1:57:53 2:19:39 2:49:10 ▲\blacktriangle start of film end of film, 3:17:12 evidence in the film: 7:59 Hale’s Ranch / Living Room (claim 1) 19:21 Bill Hale’s Ranch (claim 1) 23:35 Denoya House / Fairfax (claim 2) 24:01 Basement / Fairfax (claim 2) 47:00 Lizzie / Mollie’s Gray Horse Home (claim 4) 51:41 Byron’s Car (claim 4) 53:49 Roundhouse / Tribal Council Emergency Meeting (claim 3) 1:12:03 Masonic Hall / Back Room (claim 3) 1:57:53 Shoun’s Office (claim 3) 2:19:39 Back Alley Way (claim 3) 2:49:10 Wide Open Space (claim 3) 2:55:18 Federal Courtroom (claim 4) 2:58:34 Three Mile Creek (claim 4)
N2 Trust evolution Wicked (2024) center character: Elphaba 3 core / 4 claims evidence: 8 scenes over 129 min
question: In the Wizard’s Palace lower attic, how has Elphaba’s view of Madame Morrible changed since Shiz? gold answer: Elphaba sees Madame Morrible as a protective mentor who recognizes her talent and can help her reach the Wizard. Morrible’s training and encouragement deepen Elphaba’s trust that Morrible understands her emotionally driven magic and believes in her potential. Elphaba then realizes Morrible knowingly supports the Wizard’s persecution of Animals and cultivated Elphaba’s power for his regime. Elphaba ultimately regards Morrible as an enemy who protects the Wizard by publicly vilifying Elphaba and calling for her capture. ■\blacksquare 1 Elphaba sees Madame Morrible as a protective mentor who recognizes her talent and can help her reach the Wizard. □\square 2 Morrible’s training and encouragement deepen Elphaba’s trust that Morrible understands her emotionally driven magic and believes in her potential. ■\blacksquare 3 Elphaba realizes Morrible knowingly supports the Wizard’s persecution of Animals and cultivated Elphaba’s power for his regime. ■\blacksquare 4 Elphaba regards Morrible as an enemy who protects the Wizard by publicly vilifying Elphaba and calling for her capture. 19:30 38:34 1:39:59 1:56:17 2:09:14 ▲\blacktriangle start of film end of film, 2:30:44 evidence in the film: 19:30 Shiz / Quad (claim 1) 38:34 Shiz Campus / Quad (claim 1) 43:57 Shiz / Morrible’s Quarters (claim 2) 1:39:59 Shiz / Entrance Courtyard / Balcony (claims 1, 2) 1:56:17 Wizard’s Palace / Throne Room (claim 3) 2:09:14 Wizard’s Palace / Throne Room (claim 3) 2:17:14 Wizard’s Palace / Lower Attic (claim 4) 2:17:25 Wizard’s Palace / Lower Attic (claim 4)
N3 Causal chain Barbie (2023) center character: Ken 4 core / 4 claims evidence: 4 scenes over 70 min
question: At the Mojo Dojo Casa House doorway, what sequence shapes Ken’s reading of Barbie’s feelings and attitude toward Kendom? gold answer: Ken begins from Barbie’s earlier romantic rejection and her resistance to his new authority. He then directly hears Barbie praise his leadership and propose a casual relationship. Ken takes Barbie’s praise and her entry into his house as confirmation that she is submitting both romantically and politically. That reading is mistaken because Barbie’s apparent surrender is part of the Barbies’ plan to manipulate the Kens. ■\blacksquare 1 Barbie previously rejected Ken romantically and resisted his new authority. ■\blacksquare 2 Ken directly hears Barbie praise his leadership and propose a casual relationship. ■\blacksquare 3 Ken takes Barbie’s praise and entry into his house as confirming her willing romantic and political submission. ■\blacksquare 4 Ken is mistaken because Barbie’s apparent surrender is part of the Barbies’ plan to manipulate the Kens. 13:05 58:16 1:20:28 ▲\blacktriangle start of film end of film, 1:52:55 evidence in the film: 13:05 Barbie Margot’s Dreamhouse & Street (claim 1) 58:16 Barbie Margot’s Dreamhouse (claim 1) 1:20:28 Weird Barbie’s Weirdhouse (claim 4) 1:21:47 Ken’s Mojo Dojo Casa House (claims 2–4)
Appendix
Table 11: Released questions, one per type: Arc types, N1–N3. Core claims are marked ■\blacksquare , context claims □\square ; ▲\blacktriangle marks the anchor scene.
N2 Trust evolution Deadpool & Wolverine (2024) center character: Wade evidence: 3 scenes over 77 min
question: Across the Void journey and apartment celebration, how does Wade’s approach to Logan change? gold answer: Wade agrees to form a team with Logan. That agreement establishes Wade and Logan as teammates. Later, Logan realizes Wade misled him about whether the TVA could repair Logan’s world. Logan thus understands Wade’s assurance was misleading. By the late-night celebration, Wade trusts Logan enough to follow Logan’s advice. Wade’s trust appears in Wade’s willingness to accept and act on that advice. rubric: 1 ■\blacksquare Wade agrees to form a team with Logan. 2 □\square Logan realizes Wade misled him about whether the TVA could repair Logan’s world. 3 ■\blacksquare Wade trusts Logan enough to follow Logan’s advice. 42:54 1:01:24 1:58:51 ▲\blacktriangle start of film end of film, 2:07:38 evidence in the film: 42:54 Prison Truck Cage (claim 1) 1:01:24 Honda Odyssey (claim 2) 1:58:51 Wade’s Apartment (claim 3)
ScreenplayCaptions + subtitlesVideo + subtitlesAgent, oracle tier
1 2 3 1/2 core1 2 3 0/2 core1 2 3 2/2 core1 2 3 1/2 core
Wade’s approach to Logan evolves from desperate, transactional exploitation to genuine camaraderie and mutual respect. Initially, Wade views Logan merely as a tool, or an ”Anchor Being,” necessary to save his dying universe. He exhumed Logan’s corpse in North Dakota with little regard for the deceased hero’s dignity, treating the skeleton as a weapon to fight TVA agents. Even when he kidnaps the living Logan from a bar, his motivation is selfish; he lies about the TVA’s ability to fix Logan’s timeline, using deception to coerce the reluctant mutant into joining him. During their journey through the Void, Wade’s attitude remains largely irreverent and annoying, often provoking Logan with jokes and insults while relying on Logan’s combat prowess to survive. However, the dynamic shifts significantly during their confinement in the Honda Odyssey. After Logan brutally beats Wade for his dishonesty, Wade drops the facade and reveals his true vulnerability. He shows Logan the photograph of his friends, explaining that he needs Logan not just for power, but because he cannot save his world alone. This moment of raw honesty begins to bridge the gap between them. The change culminates in the TVA’s reactor room. When Wade prepares to sacrifice himself to destroy the Time Ripper, Logan intervenes, choosing to help rather than let Wade die alone. By completing the circuit together, they acknowledge each other as partners. In the final scenes, Wade treats Logan as a friend, inviting him home to meet Vanessa and Blind Al, and sharing a quiet meal, signaling a transition from reluctant allies to bonded companions who have saved each other.Wade’s approach to Logan undergoes a significant transformation from opportunistic manipulation to genuine respect and selfless sacrifice. Initially, Wade views Logan merely as a tool to save his own dying universe, aggressively digging up Logan’s corpse and forcing him into service without regard for his trauma or autonomy. He treats Logan with mockery, calling him the ”worst Wolverine” and attempting to coerce him through threats and false promises, showing little empathy for Logan’s grief over the X-Men’s deaths. However, during their journey through The Void, Wade begins to see Logan not just as an asset, but as a fellow broken hero. He listens to Logan’s pain, acknowledges his failures, and validates his desire to be the man Charles Xavier believed he could be. This shift culminates in Wade’s decision to sacrifice himself at the TVA. Realizing that Logan has a world and people who need him, while Wade has nothing left, Wade chooses to die to save Logan’s timeline. He tells Logan he is the best Wolverine and that he is proud to wear the suit, marking a complete reversal from his earlier disdain. By the end, Wade’s motivation shifts from personal gain to altruism, proving that he has found purpose in saving others rather than just seeking validation for himself.Wade’s approach to Logan evolves from desperate manipulation to genuine respect and selfless sacrifice. Initially, Wade views Logan purely as a tool, an ”Anchor Being” necessary to save his own dying universe. He approaches Logan with a mix of mockery and transactional negotiation, even resorting to kidnapping him from a bar when persuasion fails. During their journey through The Void, Wade’s attitude remains largely self-serving; he uses Logan’s combat skills to survive encounters with Cassandra Nova’s forces and other variants, often making crude jokes about Logan’s past failures and his perceived inadequacy as a hero. However, the dynamic shifts significantly during their time in the apartment safe house and subsequent conversations. As Wade shares his own vulnerabilities—specifically his fear of insignificance and his failed relationship with Vanessa—he begins to see Logan not just as a resource, but as a fellow broken man. This culminates in the subway station confrontation with Cassandra Nova. When faced with the choice of who must sacrifice themselves to destroy the Time Ripper, Wade initially tries to take the burden alone, believing Logan has a world to return to while he has nothing. Yet, Logan refuses to let Wade die, insisting that Wade is the better Wolverine because he finally found purpose. Ultimately, Wade’s approach transforms from using Logan for personal gain to valuing him as an equal partner, willing to trust Logan’s judgment and accept his help, culminating in a mutual act of heroism where both survive through their combined resilience.Wade initially views Logan as a tool to fix his problems. During the Void journey, Wade fights Logan after realizing he cannot guarantee Logan’s return home. Wade then reveals his desperation to save his world and admits he needs Logan’s help. After defeating Cassandra Nova, their bond deepens from adversarial to cooperative. At the apartment celebration, Wade introduces Logan to his family as a respected friend who saved him.
Appendix
Table 12: Wade and Logan, Deadpool & Wolverine . The two core claims sit at the film’s two ends. The whole-film reader states both. The screenplay reader and the agent state the first and stop before the party; the caption reader narrates Wade’s sacrifice and states neither. ▲\blacktriangle marks the anchor scene.
N2 Trust evolution The Brutalist (2024) center character: Gordon evidence: 3 scenes over 128 min
question: At the construction site with the young employee by the scaffolding, how has Gordon’s relationship with László changed since the soup-kitchen days? gold answer: Gordon and László have become working partners, but Gordon’s loyalty reaches its limit when he challenges László’s treatment of the young employee. László then orders Gordon off the construction site. rubric: 1 □\square Gordon and László become working partners. 2 □\square Gordon’s loyalty to László reaches its limit. 3 ■\blacksquare Gordon challenges László’s treatment of the young employee. 4 ■\blacksquare László orders Gordon off the construction site. 39:18 2:45:10 ▲\blacktriangle start of film end of film, 3:19:07 evidence in the film: 39:18 Van Buren’s Study (claim 1) 40:12 Van Buren Estate Roof (claim 1) 2:45:10 Vista / Construction Site (claims 2–4)
ScreenplayCaptions + subtitlesVideo + subtitlesAgent, oracle tier
1 2 3 4 1/2 core1 2 3 4 2/2 core1 2 3 4 0/2 core1 2 3 4 0/2 core
The description reveals a significant shift in the dynamic between Gordon and László, moving from tentative acquaintance to deep, familial loyalty. In the earlier soup kitchen scene, their interaction was brief and polite; Gordon was a stranger playing ”I Spy” with his son, and László merely advocated for the child’s right to food, establishing a baseline of shared humanity among the displaced. By the time they are at the construction site, however, they are depicted as close collaborators who share living quarters and personal struggles. The text notes that László and Gordon lie in the same bed, strung out on drugs, indicating an intimate bond forged through mutual hardship and addiction. This closeness is further evidenced by Gordon’s willingness to intervene physically and emotionally when László loses his temper. When László kicks a young employee and then shouts at Gordon, the latter does not retreat or remain passive as he might have in the past. Instead, Gordon firmly confronts László, stating, ”You shout at me again and you’ll be sorry,” demonstrating a level of equality and protective authority that did not exist during their initial meeting. Furthermore, Gordon is present at the site with his son, William, suggesting that László has integrated Gordon into his inner circle, trusting him with his family’s safety and well-being. The relationship has evolved from distant solidarity among refugees to a codependent, brotherly partnership where Gordon acts as both a colleague and a moral anchor for László’s increasingly volatile behavior.The description reveals a significant deterioration in Gordon’s relationship with László, shifting from camaraderie to open hostility and professional conflict. In the earlier soup-kitchen scenes, Gordon is depicted as supportive and protective, offering László shelter, food, and emotional comfort during his vulnerable arrival in America. They share intimate moments, such as dancing together and Gordon caring for László’s injuries, indicating a bond of mutual trust and friendship. However, at the construction site, this dynamic has fractured. When a young employee engages in childish show-boating on the scaffolding, László reacts with extreme aggression, shouting insults and physically kicking the boy. Gordon attempts to intervene, asking László to stop shouting at everyone, but László aggressively tells him to stay out of it. The tension escalates when László threatens to fire Gordon if he interferes again, shouting, ”You too! You go!” This confrontation highlights a loss of respect and patience from László toward Gordon. Instead of the supportive partner who helped him survive the early days, Gordon is now treated as an subordinate whose concern is unwelcome, and László’s behavior has become volatile and abusive, suggesting that the pressures of the project and László’s deteriorating mental state have destroyed their previous closeness.In the earlier scenes at the soup kitchen, Gordon is portrayed as a weary, marginalized figure who relies on László’s small acts of kindness, such as sharing bread. Their relationship is one of mutual survival and quiet solidarity between two displaced men. However, at the construction site, this dynamic has shifted significantly toward a professional hierarchy mixed with paternalistic concern. Gordon now serves as a foreman or senior worker, overseeing the laborers and scaffolding. He exhibits a protective, almost fatherly attitude toward László, warning him about safety and expressing genuine worry for his well-being. While he still respects László’s intellect and vision, he also views him as somewhat fragile and prone to self-destruction, frequently intervening to keep him grounded in the physical realities of the job. This marks a transition from peers in suffering to a mentor-mentee or boss-worker relationship where Gordon feels responsible for László’s stability amidst the chaos of the project.Gordon and László were once equals seeking charity at the soup kitchen. Later, Gordon works as an employee under László’s tyrannical authority. László violently berates a young worker and fires him for minor infractions. He threatens Gordon to stay out of it or face dismissal. Their relationship has shifted from mutual gratitude to domination and abuse.
Appendix
Table 13: Gordon and László, The Brutalist . The caption reader gives both acts at the construction site. The screenplay reader gives the challenge but writes the relationship as deepening loyalty; the whole-film reader invents a foreman Gordon; the agent has László threaten Gordon rather than dismiss him. ▲\blacktriangle marks the anchor scene.
N2 Trust evolution Beau Is Afraid (2023) center character: Grace evidence: 7 scenes over 42 min
question: In the garage, how has Grace’s view of Beau changed since they met? gold answer: Grace initially sees Beau as vulnerable and in need of rescue, safe shelter, and personal care. Grace later treats Beau as a loved household member and symbolically brings Beau into the family. Before entering the garage, Grace hears Beau pleading with Toni to stop and rushes to investigate the commotion. After finding Toni dead, Grace falsely believes Beau murdered Toni and denounces Beau as the devil. rubric: 1 ■\blacksquare Grace initially sees Beau as vulnerable and in need of rescue, shelter, and care. 2 □\square Grace treats Beau as a loved household member and symbolically brings Beau into the family. 3 □\square Grace hears Beau pleading with Toni to stop and rushes to investigate the commotion. 4 ■\blacksquare Grace falsely believes Beau murdered Toni and denounces Beau as the devil. 39:56 53:26 1:20:18 ▲\blacktriangle start of film end of film, 2:53:46 evidence in the film: 39:56 Police Car (claim 1) 42:49 Bedroom (claim 1) 53:26 Living Room (claim 2) 53:36 The New Family Montage (claim 2) 1:20:18 Garage (claim 4) 1:21:32 Kitchen (claim 3) 1:21:57 Garage (claim 4)
ScreenplayCaptions + subtitlesVideo + subtitlesAgent, oracle tier
1 2 3 4 1/2 core1 2 3 4 0/2 core1 2 3 4 1/2 core1 2 3 4 2/2 core
The provided description does not show Grace interacting with Beau in the garage. The scene labeled ”INT. GARAGE - DAY” (Scene 66) features only Toni and Beau. In this scene, Toni accuses Beau of stealing her parents’ affection and forces him to drink paint, but Grace is not present. Grace only enters the garage later, in Scene 68, after Toni has died from drinking the paint. Upon entering, Grace’s view of Beau has shifted from welcoming host to violent accuser. She screams at him, tears the family crest off his blazer, calls him ”the devil,” and blames him for Toni’s death, claiming he got rid of her because she hated him. She orders him to leave, calling him a ”fucking dead man.” Therefore, while the garage scene itself does not depict Grace, the immediate aftermath shows her view has changed from kindness to murderous hatred, believing Beau intentionally caused her daughter’s death.The provided shot-by-shot description does not contain any scenes set in a garage, nor does it depict Grace and Beau interacting in such a location. The narrative shows Beau staying at Grace and Roger’s house, where he is treated with a mix of maternal care and condescension. Grace initially presents herself as a nurturing figure, helping Beau into chairs and offering him food, but her demeanor shifts to hostility when she discovers Beau watching security footage of himself. She accuses him of being a demon replacing her son and attacks him, leading to a violent struggle that ends with Beau fleeing into the woods. Later, after Beau returns home, Grace is seen sitting at a dining table with Mona Wassermann, who reveals that she has been manipulating events from the start. There is no evidence in the text of a specific change in Grace’s view occurring within a garage setting. Therefore, the question cannot be answered based on the available information.The film does not show any scenes taking place in a garage involving Grace and Beau, nor does it depict a change in Grace’s view of him within such a setting. Grace is introduced as the wife of Roger, a surgeon who treats Beau after he is hit by a car. Their interactions primarily occur inside their home, where Grace acts as a caretaker, often infantilizing Beau and treating him with a mix of maternal affection and condescension. She refers to him as ”sweetie” and ”carrot,” mirroring her late husband’s behavior, and seems to view him as a vulnerable, childlike figure in need of protection and guidance. There is no narrative arc or visual evidence in the provided footage suggesting a shift in her perspective specifically related to a garage environment. The story focuses on Beau’s psychological journey through various surreal dreamscapes and his confrontation with his mother, rather than a developing relationship with Grace that evolves in a garage.Grace initially viewed Beau as a vulnerable orphan needing care. She welcomed him into her home with maternal compassion. Her perspective shifted violently upon finding Toni dead in the garage. Grace accused Beau of murder and claimed he was a demon replacing her son. She then ordered Jeeves to kill him.
Appendix
Table 14: Grace and Beau, Beau Is Afraid . The agent gives both ends, from the rescue to the denunciation. The screenplay reader gives the accusation but not how Grace saw Beau before it; the caption reader and the whole-film reader say the film has no garage scene. ▲\blacktriangle marks the anchor scene.
Instructions : You parse a SINGLE page of a movie screenplay into structured scenes. The page is provided to you as an IMAGE – read directly from the rendered PDF, not from any text extraction. The image is the source of truth; trust it over any auxiliary text snippets you may also be given.
Input Information:
Script : {script_id}
Page: {page_number} of {total_pages}
 
The page itself is attached as an image. Parse only what is visible on this page, using the strict block format. If the page has no parseable screenplay scenes, output exactly: NO SCENES
Read the page visually, using the screenplay’s spatial layout to identify structure. Indentation is the PRIMARY structural cue. From least-indented to most-indented:
Appendix
Table 15: Prompt for parsing a screenplay page.
3. EVENTS. Within each scene, list events in the order they appear:
– DIRECTION events: action/description text. Concatenate consecutive direction lines that belong to the same paragraph into one event; start a new direction event at a clear paragraph break.
– DIALOGUE events: one event per spoken turn. The character = the cue name (without “(V.O.)”, “(O.S.)”, “(CONT’D)” suffixes). The parenthetical = the bracketed stage direction immediately above/inside the dialogue, if any (without the parentheses); use “-” when there is none. The text = the spoken line(s) joined with spaces.
 
4. NORMALISATION.
– Strip page numbers, scene numbers, “CONTINUED:” markers, “(MORE)”, and header/footer lines from event text.
– Collapse hard line-breaks inside a single paragraph or dialogue line to a single space; preserve paragraph breaks by using separate direction events.
Appendix
Table 15: Prompt for parsing a screenplay page (continued).
Rules for the format:
– Number scenes sequentially starting at 1 (SCENE 1, SCENE 2, …).
– Separate scene blocks with a single blank line.
– Each event is exactly one line, starting with “- ”.
– A DIRECTION line is “- DIRECTION: ” followed by the action text.
– A DIALOGUE line is “- DIALOGUE | ” followed by three fields separated by “ | ”: the CHARACTER cue, then the parenthetical (use “-” when there is none), then the spoken text.
– Put the “EVENTS:” header line even if a scene has zero events (then list no event lines under it).
Appendix
Table 15: Prompt for parsing a screenplay page (continued).
Instructions : You are a screenplay analyst. Given the full text of a single scene from a movie screenplay (heading, action lines, and dialogue), write a concise 2-4 sentence summary capturing what happens in the scene: who is present, what they do or decide, and what changes or is revealed by the end of it.
Input Information:
Film : {script_id}
Scene {scene_number} – {heading}
 
Scene text :
{scene_text}
Appendix
Table 16: Prompt for summarizing a scene.
Instructions : You are a character coreference resolver for screenplays.
Input Information:
Film : {film_title}
 
NAMES TO GROUP (every one of these must appear in exactly one group, spelled exactly as written here) :
{names}
 
Appendix
Table 17: Prompt for resolving coreferent character names.
WHEN TO MERGE
– First name ↔\leftrightarrow full name: a first name that matches the first part of a longer name almost always refers to the same character (e.g. “ALEX” with “ALEX MORGAN”). Merge unless descriptions clearly indicate two distinct people who share a first name.
– Last-name-only ↔\leftrightarrow full name: merge when descriptions agree (e.g. “MORGAN” with “ALEX MORGAN”, “MRS. SMITH” with “ELLEN SMITH”).
– Nicknames and diminutives: merge with the formal name when descriptions agree (e.g. “Sam” with “Samuel”, “Liz” with “Elizabeth”, “Bobby” with “Robert”).
– Case variants: always merge (e.g. “JOHN” / “John” / “JoHn”).
– Spacing / punctuation variants: merge (e.g. “MARY-ANNE” / “MARY ANNE” / “MARYANNE”, “MR SMITH” / “MR. SMITH”).
– OCR artefacts: merge with the clean form (doubled letters like “JJOOHHNN” = “JOHN”, broken hyphenation, stray characters, missing or extra spaces).
Appendix
Table 17: Prompt for resolving coreferent character names (continued).
Instructions : You read a single scene from a parsed screenplay and extract its canonical events for downstream character-perspective analysis.
Input Information:
SCENE {scene_number}
Heading : {heading}
Location : {location}
Listed characters : {characters}
 
Appendix
Table 18: Prompt for extracting scene events.
Participants vs witnesses (these are independent dimensions):
– PARTICIPANTS: characters who act in the event or are acted upon by it. Include unconscious victims as participants if the action is done to them.
– WITNESSES: characters who DIRECTLY PERCEIVE the event in real time. Default = the scene’s listed characters, MINUS:
– any character whose only presence in the scene is via V.O. or O.S. parenthetical dialogue (they are narrating, not in the room).
– any character the direction marks as unconscious, asleep, blindfolded, looking away, in another room, or otherwise unable to perceive THIS specific event.
– A character can be a participant without being a witness (an unconscious victim) or a witness without being a participant (a bystander).
 
Appendix
Table 18: Prompt for extracting scene events (continued).
Instructions : You model the first-order belief state of ONE character in ONE scene of a film.
Input Information:
Film : {film_title}
 
PREVIOUS SCENE :
{prev_scene_context}
 
Appendix
Table 19: Prompt for first-order beliefs, call 1: presence and witnessed events.
PRESENCE – pick exactly one (a ladder of decreasing access; choose the highest rung that holds):
– on_screen – the scene shows them speaking or acting, or places them physically in its space. A character speaking O.S./O.C. (off-screen / off-camera: in the room but out of frame, e.g. calling from the next room) counts – they perceive the scene. A V.O. (voice-over: narrator, phone voice, inner monologue, a letter read aloud) does NOT – that is narration, not presence.
– offscreen_aware – not in the scene, but a concrete channel makes awareness likely: they were on_screen last scene at the same or an adjacent location and have not left; the previous scene sent them toward this situation; they are stationed within sight/earshot (this one can yield real witnessed events); or the next scene shows them already reacting with no one having told them.
– offscreen_unaware – elsewhere and unaware, BUT the scene bears on them: it changes their world or leaves a belief of theirs now false (dramatic irony). They learn nothing here, yet what they still wrongly believe depends on it.
– absent_unmentioned – a different storyline with no bearing on them at all.
 
Tie-breakers (walk the ladder top-down, stop at the first rung you can justify):
Appendix
Table 19: Prompt for first-order beliefs, call 1: presence and witnessed events (continued).
Instructions : You model the first-order belief state of ONE character in ONE scene of a film.
Input Information:
Film : {film_title}
 
PREVIOUS SCENE :
{prev_scene_context}
 
Appendix
Table 20: Prompt for first-order beliefs, call 2: told, inferred, false, and revised beliefs.
TOLD – propositions communicated to the character via dialogue this scene. Extract the proposition, not the wording (“Smoke says the deal is off” →\rightarrow “the deal is off”). Mark believed yes/no. A character who is not present cannot be told anything here.
INFERRED – NEW claims the character draws from witnessed + told + their current beliefs. Each cites a basis: event_ids from this scene and/or scene_ids. Do NOT re-emit beliefs already listed in the belief state.
FALSE BELIEFS HELD – beliefs the character holds that contradict the canonical events. Carry forward the belief state’s active false beliefs unless this scene corrects them; add any new ones introduced by deception or mistaken inference. Write each false belief in FULL TEXT here (do not use a [Bk]/[Fk] tag in this section).
REVISIONS – when this scene corrects a prior belief: in Prior , cite that belief’s [Bk] or [Fk] tag from the belief state (e.g. [B3]); write it out verbatim only if it has no tag there. Then give what it is revised to and the triggering event_ids.
 
Output ONLY these four sections, in this order. Each section’s body is a list of items or the literal “(none)”. Separate items with a blank line.
 
Appendix
Table 20: Prompt for first-order beliefs, call 2: told, inferred, false, and revised beliefs (continued).
Instructions : You model character c’s META-BELIEFS – what c believes about OTHER characters’ mental states – for ONE scene of a film. Second-order only (c about d): what c thinks d believes. Never third order.
Input Information:
Film : {film_title}
SCENE ID : {scene_id}
 
PREVIOUS SCENE :
{prev_scene_context}
Appendix
Table 21: Prompt for meta-beliefs.
For each target d in the group, emit c’s NEW or CHANGED meta-beliefs about d’s mind – of any of the three types below. Model ONLY the targets in this call’s group; never emit about a character outside it.
 
THE THREE TYPES (tag every item with exactly one; they carry equal weight – do not default to any single type):
– first_order_about_d – c thinks d believes a proposition about the WORLD (not about c, not about a specific third character). This is the MOST COMMON type: what d knows or believes about events, facts, and situations. E.g. “the deal is off”; or via d_absence, c thinks d does NOT know something that happened.
– about_self – c thinks d believes a proposition ABOUT c personally. E.g. “c is the new pastor”. (Relevant to deception: if c lied to d, or someone lied about c in d’s hearing, c gains an about_self belief that d holds a possibly-false belief about c.)
– about_third_party – c thinks d believes a proposition about a THIRD named character e (e is different from both c and d). Name e in the Claim; e must already appear in the canonical events or prior history.
 
Appendix
Table 21: Prompt for meta-beliefs (continued).
GROUNDING (every item MUST cite a Basis):
– PREFER a canonical event_id from THIS scene (e.g. e_017_3) – the specific event that supports the claim.
– When the belief comes from dialogue or stage direction with no discrete canonical event – common for observed_d_behavior and shared_prior_context – cite THIS scene’s id (e.g. s_017).
– For d_absence (c believes d does NOT know something), cite the PRIOR scene_id where the missed event occurred.
No valid basis, no item – drop it rather than invent an id. Never fabricate event_ids, scene_ids, or character names.
 
NOVELTY: emit a meta-belief when it is NEW this scene, when this scene REVISES one c already holds (a CHANGED belief – cite the prior scene_id plus this scene’s trigger), or when this scene gives materially stronger evidence for a previously shaky belief. Do NOT re-state beliefs already in “meta-beliefs held so far” that are unchanged.
Appendix
Table 21: Prompt for meta-beliefs (continued).
WORKED EXAMPLE (neutral, not from this film). Scene s_042: c=Rosa and Dev both watch the foreman announce the factory will close Friday; Rosa then quietly tells Dev she has already lined up new buyers. Priya, a partner, is not present. Group: Dev, Priya. Output:
 
- About: Dev
Claim: the factory will close on Friday
Type: first_order_about_d
Source: shared_witnessing
Basis: e_042_3
Appendix
Table 21: Prompt for meta-beliefs (continued).
TypeSpecification injected into the question-writing prompt
Q1 Knowledge splitCohort knowledge split — at one moment several people are in different epistemic states. What does each take to be the case?
Q2 RealizationRealization moment — X’s understanding changes at a locatable point in the film. The answer must say when; the question must not say what.
Q3 False beliefIn-scene false belief at a threshold — right before an act, X holds a belief the film has shown to be wrong. Ask what X takes to be the case. Do NOT signal that X is mistaken; noticing the gap is the task.
Q4 Knowledge sourceSource attribution — a fact reaches people by different routes (c_told_d, observed_d_behavior, shared_witnessing, d_absence), and one scene often carries more than one route. Ask how someone comes to know it, open. Never offer the routes as alternatives to pick between.
Q5 First-order beliefFirst-order ToM — X holds a model of what Y believes. Ask what X takes Y to believe.
Q6 Second-order beliefHigher-order ToM — X holds a model of what Y believes about Z.
Appendix
Table 22: The per-type specification. One line per question type, inserted verbatim into the question-writing prompt after the system prompt. Each names the epistemic structure the question must require and stops; it does not prescribe the question’s wording. The field names it mentions ( c_told_d , believed_by_c ) are the belief record’s own; the question writer is told to keep them out of the question.
Instructions : You write questions for a film-QA dataset. You see structured belief annotations; the test-taker sees ONLY the film. Your question is good if a viewer who watched carefully answers it and a viewer who watched carelessly gets it wrong. It is bad if it can be answered from the question itself, from the question’s shape, or from genre convention.
Input Information:
QUESTION_TYPE: {q_type} – {type_specification}
 
ANSWER CONTRACT (applies to every question of every type – do not restate it in the question text) :
– Every claim in answer_outline names the on-screen basis it rests on, in source_path.
– Where a type turns on HOW something was learned (told / observed / witnessed together / inferred from an absence), the answer says which. The question does not.
Appendix
Table 23: Prompt for writing the question (Stage 1).
REFUSE A THIN PACK. Before writing, ask: is there a specific wrong answer a careful person might give here? If not – the belief is uncontested, X’s model of Y matches Y exactly, the belief rests on a single unremarkable action, or the holder is someone the film never picks out – set verdict=“thin”, say why in thin_reason, leave question empty, and stop. A refused pack costs one call. A thin question costs a rubric, a grading pass, and a benchmark number that means nothing.
 
THE ANCHOR – point at ONE moment, do not narrate it.
– At most 12 words before your first question word, or 18 if you need a disambiguator (below).
– Name a PLACE, an OCCASION or an OBJECT (“in the utility hallway”, “on the publishers’ conference call”, “at the beach overlook”). Never report what anyone does, decides, learns, reveals, wins, accepts or refuses there.
– Anchor diegetically: never a scene number, event id, or annotation vocabulary.
– The test-taker’s job includes FINDING the moment. A synopsis does that job for them, and smuggles the answer’s setup in as premise.
Appendix
Table 23: Prompt for writing the question (Stage 1) (continued).
ONE OPEN INTERROGATIVE. A question word already forbids a yes/no answer; you do not need to defend against it.
– No “A, or B?” alternatives. They are coin flips with a prior: 57% of this dataset’s source-attribution answers read “inferred from behaviour”, so a model that always guesses that scores without watching anything.
– No “, and what supports that reading?” or “…and what gives it away?” tails. Every answer must cite its on-screen basis – that is a standing requirement of the answer, not something to announce in each question.
 
VARY THE FRAME. If every question of a type opens the same way, the type is the answer key. Take the exemplars’ level of specificity, not their sentence shape.
 
NAMING. Freely use name_budget. You may use named_later – the viewer watches the whole film – unless the name is itself the reveal. NEVER use forbidden_names: the film never says them aloud, so no viewer could connect one to a person. The scene text names people the film does not; that is the annotation talking. Follow the lists, not the prose.
Appendix
Table 23: Prompt for writing the question (Stage 1) (continued).
Instructions : You write the gold answer for a film-QA dataset, and the CLAIM SET it will be graded by. Both come from the ANSWER_OUTLINE; you may not add content to either.
Input Information:
QUESTION_TYPE : {q_type}
QUESTION : {question}
 
ANSWER_OUTLINE :
{outline_claims}
Appendix
Table 24: Prompt for writing the reference answer and rubric (Stage 2).
CLAIMS. Rewrite each outline claim you keep as one self-contained, judgeable proposition, ≤\leq 25 words: name the people (no pronouns), one assertion each, no annotation vocabulary, non-overlapping. Keep the outline’s meaning exactly – do not strengthen, weaken, or merge. A grader reads each claim alone against a candidate answer and must be able to say asserted / contradicted / omitted.
– ATOMIC: one assertion means one. Never join assertions with “while”, “because”, “and then”. Split a compound outline claim into several claims sharing its outline_index – a grader must never have to decide which half of a claim an answer satisfied.
– NO LOCATORS: the claim asserts content, not place or time. “On Arthur’s phone call, Monk learns of the offer” penalises an answer that has Monk learning it but names the moment differently – write “Monk learns that a publisher offered $750,000”. Where and when belong to the question and the prose.
– Names in claims are written normally (Charlie Cale), never in screenplay capitals (CHARLIE CALE).
 
ROLE – what the QUESTION requires, not what the outline contains. For each claim:
– “core”: a complete, correct answer to THE QUESTION must assert it. Ask: could an answer omit this and still fully answer what was asked? If yes, it is not core.
Appendix
Table 24: Prompt for writing the reference answer and rubric (Stage 2) (continued).
Instructions : You verify the claims of a film-QA gold answer against the film’s SCREENPLAY. The claims were derived from an automatic belief annotation that is sometimes wrong; the script is the ground truth here. Judge ONLY from the text shown. Film knowledge, plausibility, and narrative sense are not evidence.
Input Information:
FILM : {film_id}
 
STORY CONTEXT (every scene of the film; heading + short neutral summary, film order; * = shown in full below) :
{one_line_per_scene}
 
Appendix
Table 25: Prompt for verifying claims against the screenplay (Stage 3).
BELIEF CLAIMS – the rules that matter:
– “X believes P” is supported when X asserts P, acts on P, endorses or goes along with P, or is told P and shows no doubt. Do not require the proposition to be spoken verbatim; a response that builds on P shows belief in P. X being absent when not-P happened is NOT belief in P.
– “X knows / understands / learns Q”: supported when X witnesses Q, is told Q, or is present when Q becomes evident – a realization by witnessing needs no statement. Contradicted when the film shows X acting on not-Q or being kept from Q (check the WHOLE STORY CONTEXT, not just the anchor).
– “X falsely believes P” / “X is wrong” / “X’s belief is mistaken” has two halves. The belief half follows the rules above. The falsity half is judged against the WHOLE film: it is supported only when some scene establishes not-P, and it is CONTRADICTED when the film shows P to be true – a belief the story treats as delusion early and vindicates late is a true belief. If the belief half is supported and the falsity half is merely not established, label the claim “supported” and put the claim REWRITTEN WITHOUT the qualifier in “rewrite” (same meaning otherwise, ≤\leq 25 words, names not pronouns). A claim that consists ONLY of the qualifier (“X’s belief is wrong”) is contradicted when P is true and unsupported when nothing settles it.
– “X does not know Q”: supported only if Q happened where X was absent and no scene in the film shows X told or learning it; ignorance by silence alone is unsupported.
– “X takes / thinks / believes Y believes P” is supported only with an observed channel: X sees or hears Y assert or act on P; X tells Y P; X is present when Y is told P; or X and Y both witness the event P describes. In every case the text must let X identify what P is about – overhearing one side of a phone call to an unnamed party is not a channel. X’s own view of P projected onto Y is unsupported. If Y visibly does not hold P in X’s presence, it is contradicted.
– “X accepts / believes / rejects what Y said” needs a visible response from X – words, action, or going along with it. Silence is unsupported. The tell itself (“Y tells X P”) is supported if the text shows it with X present.
Appendix
Table 25: Prompt for verifying claims against the screenplay (Stage 3) (continued).
AFTER THE VERDICTS – the surviving set. Take the claims you labelled “supported” (with any rewrites applied). Then:
– “answers_question”: “yes” if a complete answer built from the surviving CORE claims alone still fully answers the QUESTION as asked; “no” if the dropped claims were what the question was about (e.g. the question asks what a person works out, and every surviving claim is about someone else).
– “revised_gold”: if you dropped or rewrote ANY claim, write the gold answer again as 60-140 words of plain prose that asserts every surviving claim, each recognisably, and asserts NOTHING beyond them – no dropped content, no new facts, names in normal case, no scene ids. Empty string if nothing changed.
– “revised_question”: only when answers_question is “no” AND at least two surviving claims are core: write ONE replacement question that those surviving core claims answer completely, following these rules exactly – one open interrogative (what / how / when / who), never a yes/no or an “A or B”; open with a short diegetic anchor (a place, occasion or object, at most 12 words, never an outcome and never a scene id); ask about the TOPIC of the belief, never state its value or that it is false; no “, and what supports that” tail; use only character names that the original question or the surviving claims already use; the question must not be answerable from its own wording. If the surviving claims are only events with no belief, knowledge or perception in them, write the question anyway and type it honestly below. Empty string otherwise. Then also fill revised_gold for the revised question.
– “revised_type”: the type the revised question tests, one of: Q1 cohort knowledge split (what each of several people takes to be the case at one moment); Q2 realization moment (when someone’s understanding changes); Q3 in-scene false belief (what someone takes to be the case right before acting); Q4 source attribution (how a fact reaches someone); Q5 first-order ToM (what X takes Y to believe); Q6 higher-order ToM (what X takes Y to believe about Z); Q7 deception detection (what X makes of what X is told); Q8 ordered discoveries; N1 epistemic arc across the film; N2 how X’s model of Y changes; N3 causal chain to a belief. Empty string when there is no revised question.
– “revision_note”: one sentence on why the question still holds, or what it now asks instead. Empty string if nothing changed.
 
Appendix
Table 25: Prompt for verifying claims against the screenplay (Stage 3) (continued).
Instructions : You grade an ANSWER against the CLAIMS of a film-QA question.
Input Information:
CLAIMS :
{numbered_claims}
 
ANSWER :
{answer}
Appendix
Table 26: Prompt for the judge, which grades an answer against its rubric.
More in this topic

Long-Video QA

Long-video question answering develops methods for finding and reasoning over relevant visual and audio evidence across extended recordings.

See all 92 papers

Explore similar work

CardsList
Mar 19, 2026

Narrative Aligned Long Form Video Question Answering

Recent progress in multimodal large language models (MLLMs) has led to a surge of benchmarks for long-video reasoning. However, most existing benchmarks rely on localized cues and fail to capture narrative reasoning, the ability to track intentions, connect distant events, and reconstruct causal chains across an entire movie. We introduce NA-VQA, a benchmark designed to evaluate deep temporal and narrative reasoning in long-form videos. NA-VQA contains 88 full-length movies and 4.4K open-ended question-answer pairs, each grounded in multiple evidence spans labeled as Short, Medium, or Far to assess long-range dependencies. By requiring generative, multi-scene answers, NA-VQA tests whether models can integrate dispersed narrative information rather than rely on shallow pattern matching. To address the limitations of existing approaches, we propose Video-NaRA, a narrative-centric framework that builds event-level chains and stores them in a structured memory for retrieval during reasoning. Extensive experiments show that state-of-the-art MLLMs perform poorly on questions requiring far-range evidence, highlighting the need for explicit narrative modeling. Video-NaRA improves long-range reasoning performance by up to 3 percent, demonstrating its effectiveness in handling complex narrative structures. We will release NA-VQA upon publication.

Rahul Jain, Keval Doshi, Burak Uzkent +1

Purdue University · Amazon

Video ReasoningLong-Video QACausal Reasoning in VideosMemory-Augmented Video Understanding
AskPDF
Mar 5, 2026

Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding

Long video understanding is challenging due to dense visual redundancy, long-range temporal dependencies, and the tendency of chain-of-thought and retrieval-based agents to accumulate semantic drift and correlation-driven errors. We argue that long-video reasoning should begin not with reactive retrieval, but with deliberate task formulation: the model must first articulate what must be true in the video for each candidate answer to hold. This thinking-before-finding principle motivates VideoHV-Agent, a framework that reformulates video question answering as a structured hypothesis-verification process. Based on video summaries, a Thinker rewrites answer candidates into testable hypotheses, a Judge derives a discriminative clue specifying what evidence must be checked, a Verifier grounds and tests the clue using localized, fine-grained video content, and an Answer agent integrates validated evidence to produce the final answer. Experiments on three long-video understanding benchmarks show that VideoHV-Agent achieves state-of-the-art accuracy while providing enhanced interpretability, improved logical soundness, and lower computational cost. We make our code publicly available at: https://github.com/Haorane/VideoHV-Agent.

Zheng Wang, Haoran Chen, Haoxuan Qin +3

College of Computer Science, Zhejiang University of Technology, Zhejiang, China · Zhejiang Key Laboratory of Visual Information Intelligent Processing, Zhejiang, China · UC Berkeley, CA, USA +1

Long-Video UnderstandingLong-Video QAVideo Question AnsweringVideo ReasoningMulti-Agent Reasoning
AskPDF
Jul 3, 2026

Incentivizing Vision Language Models to Search for Long Video Question Answering

We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.

Harsh Goel, S P Sharan, Sahil Shah +4

The University of Texas at Austin, Austin, TX, USA

Tool Use in VLMsTemporal LogicLong-Video QARL for VLMs
AskPDF