As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

Zhongrui Gui; Junyu Xie; Tengda Han; Weidi Xie; Andrew Zisserman

cs.CVOct 27, 2026

As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

Authors: Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman

Organizations: Visual Geometry Group, Department of Engineering Science, University of Oxford, UK · School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, China

Abstract

Films introduce characters incrementally as they progress. We study the problem of identifying principal characters and assigning their names directly from the movie itself, without using external cast lists, actor photographs, or pre-built character banks. We introduce the character naming task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name. This task is challenging because name resolution often depends on cinematic cues such as shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze, and temporal continuity. We provide a dataset and annotations for evaluating character naming and first character appearances, develop a video-and-dialogue model for the task, and compare it with proprietary multimodal large language models. We further introduce a progressive framework for building a character bank as the film unfolds. Together, these contributions support online, human-like movie understanding in which character identities are inferred from the film itself.

Figures & tables

Figure 1: Character naming task. We associate character names to their facial appearances in movies, mirroring how viewers infer who is who while watching. Characters may be introduced through diverse cues, including self-introduction, direct address, and third-party reference. We show three representative examples from Four Weddings and a Funeral (1994). Correctly linking a character to their name is central to movie understanding, enabling coherent character tracking across scenes as the story unfolds.

Figure 2: Example annotations from the MCN dataset. The MCN dataset provides manually annotated timestamps for each character’s first visual occurrence and first naming, along with ground-truth name-to-face correspondences for the Character Naming task.

Task	Split	# Movies	Duration (h)	# Characters	# Questions
CN	Train	$377$	$108.13$	$1853$	$22210/21150$
CN	Test	$30$	$6.51$	$226$	$1582/1399$
FO	Test	$20$	$39.98$	$287$	$574$

Table 1: Statistics of the Character Naming (CN) and First Occurrence (FO) annotations. Each question corresponds to a name-to-track mapping query or a first-occurrence timestamp annotation.

Figure 3: Streaming pipeline for character bank construction. As new shots arrive, we extract observations (character tracks, subtitles, and mentioned names) and reason jointly over them to link face tracks to names. Each step incrementally updates the bank through three operations: naming an existing cluster ( Stevie , first cluster), enriching a cluster with additional face exemplars (second cluster), and instantiating a new entry for a previously unseen character (fourth cluster). The sample is drawn from The Ides of March (2011).

Model	Propriet.	Fine-Tuned	Overall Acc.	Known Acc.
Random Baseline	$\times$	$\times$	$29.90$	$33.74$
Gemini-3-Flash-Preview	✓	$\times$	$79.57$	$86.62$
Gemini-3.1-Pro-Preview	✓	$\times$	$\mathbf{82.47}$	$\mathbf{89.69}$
Qwen3VL-4B-Thinking	$\times$	$\times$	$52.15$	$56.68$
Qwen3VL-8B-Instruct	$\times$	$\times$	$53.60$	$58.83$
Qwen3.5-9B	$\times$	$\times$	$61.57$	$67.69$

Table 2: Character naming accuracy on the MCN test set. We report overall accuracy (including none labels) and known accuracy (restricted to queries with a valid face track). Our fine-tuned Qwen3VL-4B-Thinking closes most of the gap to proprietary frontier models despite being substantially smaller in terms of model size.

Figure 4: Qualitative results for Character Naming. Examples are from Four Weddings and a Funeral and Jerry Maguire . In each case, our default “Qwen3VL-4B-Thinking (SFT + RL)” model reasons jointly over subtitles and video frames to link a queried name to the correct face — in the first, by following a verbal introduction to a later podium shot; in the second, by resolving direct address across a shot–reverse-shot exchange.

Model	Propriet.	Fine-Tuned	Overall Acc.	Known Acc.
Random Baseline	$\times$	$\times$	$18.42$	$20.93$
Gemini-3-Flash-Preview	✓	$\times$	$78.62$	$\mathbf{81.34}$
Gemini-3.1-Pro-Preview	✓	$\times$	$\mathbf{80.50}$	$78.36$
Qwen3VL-8B-Instruct	$\times$	$\times$	$45.28$	$45.52$
Qwen3.5-9B	$\times$	$\times$	$50.94$	$54.48$
Qwen3VL-4B-Thinking (SFT + RL)	$\times$	✓	$\mathbf{64.78}$	$\mathbf{71.64}$

Table 3: Character naming accuracy on the MCN test subset from unseen movies. We report the accuracy results on a test subset derived from unseen movies (released after the models were trained) by all evaluated models to isolate the effects of knowledge leakage. Overall accuracy includes none labels, and known accuracy is restricted to queries with a valid face track.

Method	Streaming	mTIoU v	mTIoU n	PCR@5	PCR@10	PCR@15
Gemini-3-Flash-Preview	$\times$	$51.81$	$30.92$	$35.00$	$32.04$	$29.84$
Gemini-3.1-Pro-Preview	$\times$	$49.05$	$30.10$	$44.00$	$37.15$	$34.93$
Ours	✓	$\mathbf{84.90}$	$\mathbf{54.40}$	$\mathbf{63.00}$	$\mathbf{54.54}$	$\mathbf{48.19}$

Table 4: Results on the MCN First Occurrence dataset. We report mean Temporal IoU (mTIoU) against two reference intervals, first visual occurrence and first naming, and Principal Character Recall (PCR@

K

) on the final character bank for

K\in\{5,10,15\}

. Our method operates in a streaming setting, whereas proprietary baselines are given full-movie access.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SFT	RL	Augment.	Overall Acc.	Known Acc.
Qwen3VL-8B-Instruct	✓	$\times$	$\times$	$64.92$	$71.77$
Qwen3VL-4B-Thinking	✓	$\times$	$\times$	$62.39$	$67.98$
Qwen3VL-4B-Thinking	✓	$\times$	✓	$64.16$	$70.41$
Qwen3VL-4B-Thinking	✓	✓	$\times$	$71.37$	$79.56$
Qwen3VL-4B-Thinking	✓	✓	✓	$\mathbf{72.76}$	$\mathbf{81.06}$

Appendix

Table 5: Ablation study on Character Naming. We compare our two-stage (SFT + RL) fine-tuning of Qwen3VL-4B-Thinking against direct SFT on a larger Qwen3VL-8B-Instruct model, and further ablate the contributions of the RL stage and color-permutation augmentation.

Variant	mTIoU v	mTIoU n	PCR@5	PCR@10	PCR@15
Ours (Full)	$84.90$	$\mathbf{54.40}$	$\mathbf{63.00}$	$\mathbf{54.54}$	$\mathbf{48.19}$
w/o Face-Track Filtering	$83.30$	$52.27$	$60.00$	$53.01$	$47.47$
w/o Query Name Filtering	$\mathbf{85.13}$	$52.16$	$57.00$	$49.46$	$44.99$
w/o Inter-Entry Name Disambiguation	$84.90$	$52.40$	$62.00$	$53.00$	$47.48$

Appendix

Table 6: Ablation of error-correction mechanisms. We start from our full model and disable one mechanism at a time, measuring the impact on mTIoUs and PCR@Ks.

Method	Streaming	mTIoU v	mTIoU n	PCR@5	PCR@10	PCR@15
Gemini-3-Flash-Preview	$\times$	$51.81$	$30.92$	$35.0$	$32.0$	$29.8$
Gemini-3.1-Pro-Preview	$\times$	$49.05$	$30.10$	$44.0$	$37.2$	$34.9$
MIL [ 27 ]	$\times$	$82.85$	$33.38$	$34.0$	$19.7$	$16.7$
Ours	✓	$\mathbf{84.90}$	$\mathbf{54.40}$	$\mathbf{63.0}$	$\mathbf{54.5}$	$\mathbf{48.2}$

Appendix

Table 7: Results of the MIL baseline on the MCN First Occurrence dataset. We report an additional baseline for streaming character bank construction.

Figure 5: Examples from the MCN dataset. Six character-naming annotations illustrating the range of problem types covered: direct address with shot–reverse-shot structure, direct address with eyeline match, direct address over the telephone, on-screen text, and OCR-based reasoning. Clips are drawn from Moneyball , Gangs of New York , Sleepless in Seattle , The Help , Four Weddings and a Funeral , and The Ides of March .

Figure 6: Qualitative results for character naming. We associate character names with their facial appearances in movies, mirroring how audiences infer who is who while watching. Characters may be introduced through diverse cues, including direct address by other characters (top), visual reasoning with common knowledge (middle), and third-party reference (bottom). In each case, the model grounds the name to the correct colored box by combining the spoken cue with visual evidence such as gaze, gesture, and scene role. Correctly linking character identities is central to movie understanding, enabling coherent character tracking across scenes. The examples are taken from Sleepless in Seattle , Four Weddings and a Funeral and Meet the Parents .

Figure 7: Comparison of reasoning efficiency on character naming. We compare our model against Qwen3.5-9B on the same scene. Our model produces a concise, well-grounded answer that directly links the spoken cue (“Congratulations, Walter”) to visual evidence (gaze and handshake direction), correctly identifying the orange box as Walter. In contrast, the baseline generates a lengthy chain of thought with repeated re-examinations and self-verifications, yet arrives at an incorrect mapping (green). This illustrates that longer reasoning does not guarantee better grounding: our model achieves more accurate character naming with substantially fewer tokens.

Figure 8: Examples of failure cases for both our model and Gemini-3.1-Pro-Preview. We show three examples in which Gemini-3.1-Pro-Preview, the strongest proprietary baseline, fails. These cases suggest that it struggles to capture subtle actions that are important for character naming. In the last example, Gemini is given the original audio rather than the erroneous subtitles for inference. The examples are taken from Four Weddings and a Funeral and Jerry Maguire .

Figure 9: Examples of failure cases for either our model or Gemini-3.1-Pro-Preview. We show three examples in which Gemini-3.1-Pro-Preview or our model fails under common failure modes. These cases suggest that our model is better at leveraging fine-grained visual cues, speaker activity, and cross-turn dialogue context to resolve character names, while remaining errors often arise from subtle actions, ambiguous speaker grounding, or noisy subtitles. The examples are taken from Crazy, Stupid, Love. , Star Wars: Episode IV - A New Hope , and The Big Lebowski .

Figure 10: Example of the constructed character bank for Juno . We show the character bank constructed by our streaming pipeline. Each row (apart from the last two) corresponds to one character entry and includes representative face tracks, the predicted canonical name with confidence, and discovered aliases. The example illustrates that the bank captures major characters with clean visual clusters and resolves name variations, such as mapping Mama Bear to Juno , Bren to Brenda , and McGovern to Mac . Some naming errors remain, typically due to noisy or ambiguous character-name associations.

Figure 11: Example of the constructed character bank for Meet the Parents . We show another example of a character bank constructed by our streaming pipeline. Our method correctly identifies the most prominent characters, but errors remain for the bottom three entries. These characters often appear in complex multi-person scenes, making reliable name binding more difficult. In addition, transcription errors can introduce noisy name variants; for example, Fonker and Falker should both be resolved to Focker for Greg Focker .

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 167 papers

Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video
Jul 13, 2026Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar +1Visual Object Tracking Video-Language Model Evaluation
Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering
Oct 7, 2026Anas Filali Razzouki, Killian Steunou, Khalil Guetari +3Video Captioning Video QA
Télécom SudParis, Institut Polytechnique de Paris, France · Moments Lab Research, France
Character Identity is not Speaker Identity: KyaraBench and KyaraEmbed for Character Verification
Oct 5, 2026Joonyong Park, Jerry LiAudio Representation Learning Representation Learning
Spellbrush, CA, USA

Jul 13, 2026J/K move · Enter open · S save

Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar, Roghayeh Leila Barmaki

Visual Object Tracking Video-Language Model Evaluation Video QA Long-Video Understanding Shortcut Learning Multimodal Model Evaluation+1

Open paper AskPDF arXiv

Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini~~2.5~~Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.

Abstract

Video-Language Models

Explore similar work

Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering

Character Identity is not Speaker Identity: KyaraBench and KyaraEmbed for Character Verification