StoryMovie: A Dataset for Semantic Alignment of Visual Stories with Movie Scripts and Subtitles

Daniel Oliveira; David Martins de Matos

cs.CVFeb 25, 2026

StoryMovie: A Dataset for Semantic Alignment of Visual Stories with Movie Scripts and Subtitles

Authors: Daniel Oliveira, David Martins de Matos

Organizations: INESC-ID Lisboa, R. Alves Redol 9, 1000-029 Lisboa, Portugal · Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais, 1049-001 Lisboa, Portugal

Abstract

Visual storytelling models that correctly ground entities in images may still hallucinate semantic relationships, generating incorrect dialogue attribution, character interactions, or emotional states. We introduce StoryMovie, a dataset of 1,757 stories aligned with movie scripts and subtitles through LCS matching. Our alignment pipeline synchronizes screenplay dialogue with subtitle timestamps, enabling dialogue attribution by linking character names from scripts to temporal positions from subtitles. Using this aligned content, we generate stories that maintain visual grounding tags while incorporating authentic character names, dialogue, and relationship dynamics. We fine-tune Qwen Storyteller3 on this dataset, building on prior work in visual grounding and entity re-identification. Evaluation using DeepSeek V3 as judge shows that Storyteller3 achieves an 89.9% win rate against base Qwen2.5-VL 7B on subtitle alignment. Compared to Storyteller, trained without script grounding, Storyteller3 achieves 48.5% versus 38.0%, confirming that semantic alignment progressively improves dialogue attribution beyond visual grounding alone.

Figures & tables

Figure 1: Example story from the StoryMovie dataset showing entity grounding with script-aligned dialogue. Character names from the screenplay (e.g., Mr. Johnny, Loretta) are linked to visual entities, and dialogue is attributed based on script-subtitle alignment. Loretta is re-identified across frames, appearing as char4 in images 1 and 3 and as char2 in images 2, 4, and 5.

Reference Type	Ours	Qwen2.5-VL 7B	Ties
Subtitles	89.9% $\pm$ 1.4	3.5% $\pm$ 0.8	6.5% $\pm$ 0.4
Description	63.4% $\pm$ 0.9	4.1% $\pm$ 0.9	32.5% $\pm$ 1.2
Synopsis	87.6% $\pm$ 1.1	6.8% $\pm$ 0.7	5.7% $\pm$ 0.5

Table 1: Semantic alignment evaluation: Ours (Qwen Storyteller3) vs. Qwen2.5-VL 7B (base model). Values represent win rates from pairwise preference evaluation. Results are mean

\pm

standard deviation across three independent evaluation runs.

Reference Type	Ours	Baseline	Ties
Subtitles	48.5% $\pm$ 1.4	38.0% $\pm$ 1.6	13.5% $\pm$ 0.5
Description	35.5% $\pm$ 1.7	15.2% $\pm$ 1.3	49.2% $\pm$ 1.0
Synopsis	42.7% $\pm$ 1.5	28.5% $\pm$ 1.4	28.8% $\pm$ 0.8

Table 2: Semantic alignment evaluation: Ours (Qwen Storyteller3) vs. Baseline (Qwen Storyteller, trained without script alignment). Values represent win rates from pairwise preference evaluation. Results are mean

\pm

standard deviation across three independent evaluation runs.

Reference Type	Baseline	Qwen2.5-VL 7B	Ties
Subtitles	86.4% $\pm$ 0.9	12.2% $\pm$ 1.2	1.4% $\pm$ 0.3
Description	72.4% $\pm$ 1.2	23.6% $\pm$ 1.8	4.0% $\pm$ 0.7
Synopsis	86.0% $\pm$ 1.2	12.2% $\pm$ 0.8	1.9% $\pm$ 0.5

Table 3: Reference alignment evaluation: Baseline (Qwen Storyteller) vs. Qwen2.5-VL 7B from prior work. Values represent win rates from pairwise preference evaluation. Results are mean

\pm

standard deviation across three independent evaluation runs.

Model	Overall	Emot.	Action	Relat.
Ours	93.9	89.5	97.4	94.7
Baseline	63.2	65.8	68.4	55.3

Table 4: Factual accuracy evaluation via script-derived Q&A. Accuracy (%) on 38 test stories (114 questions total, 3 per story).

Visual Storytelling

See all 23 papers

STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories
Jan 13, 2026Qiuyu Tian, Zequn Liu, Yiding Li +9KG Construction LLM Evaluation
Southeast University, Nanjing, China · Beijing Zhongguancun Academy, Beijing, China · ZhuiWen Technology Co., Ltd., Beijing, China +1
StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description
Jul 13, 2026Seung Hyun Hahm, Minh T. Dinh, SouYoung JinMemory-Augmented VLMs Video-Language Models
Dartmouth College, USA
OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
Apr 13, 2026Junfu Pu, Yuxin Chen, Teng Wang +1Long-Video Understanding Multimodal Large Language Models
ARC Lab, Tencent

Jan 13, 2026J/K move · Enter open · S save

STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories

Qiuyu Tian, Zequn Liu, Yiding Li, Fengyi Chen+8

Southeast University, Nanjing, China · Beijing Zhongguancun Academy, Beijing, China · ZhuiWen Technology Co., Ltd., Beijing, China+1

KG Construction LLM Evaluation Long-Context QA

Open paper AskPDF arXiv

Movie screenplays are rich long-form narratives that interleave complex character relationships, temporally ordered events, and dialogue-driven interactions. While prior benchmarks target individual subtasks such as question answering or dialogue generation, they rarely evaluate whether models can construct a coherent story world and use it consistently across multiple forms of reasoning and generation. We introduce STAGE (Screenplay Text, Agents, Graphs and Evaluation), a unified benchmark for narrative understanding over full-length movie screenplays. STAGE defines four tasks: knowledge graph construction, scene-level event summarization, long-context screenplay question answering, and in-script character role-playing, all grounded in a shared narrative world representation. The benchmark provides cleaned scripts, curated knowledge graphs, and event- and character-centric annotations for 150 films across English and Chinese, enabling holistic evaluation of models' abilities to build world representations, abstract and verify narrative events, reason over long narratives, and generate character-consistent responses.

Abstract

Visual Storytelling

Explore similar work

STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories

StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video