OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

Junfu Pu; Yuxin Chen; Teng Wang; Ying Shan

cs.CVApr 13, 2026

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

Authors: Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan

Organizations: ARC Lab, Tencent

Abstract

Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.

Figures & tables

Figure 1: Overview of our Video-to-Script (V2S) framework. Given a long-form cinematic video, our pipeline performs temporally grounded scene-event parsing and generates a structured script with multimodal fields (dialogue, action, expression, and audio cues).

Figure 2: Overview of the memory-augmented progressive annotation pipeline. The character profile manager injects historical profiles to guide plot reasoning and dynamically updates character memory. The generated plot description and raw audio-visuals are then fed into a Gemini-based annotator to produce a fine-grained video script.

Figure 3: Overview of the proposed architecture. Instruction, video, and audio are encoded into multimodal tokens and fused in the LLM via AV-DeepStack across multiple layers. The model first performs multimodal plot and character-relationship reasoning and then generates structured script outputs, including location, environment, events, character, action, expression, and dialogue for each event.

Panel of Figure 4 — Figure 4: Comparison of two strategies for extending OmniScript to long videos. Left (Strategy 1): Direct context extension, trained with long-video annotations (including memory-refine labels) and cross-video composition to create pseudo videos and pseudo scripts. Right (Strategy 2): Two-stage inference: first train a plot-segmentation model to predict each segment’s timestamps, plot, character list, and relations, then feed each clip with its plot/character information into OmniScript for segment-level generation, and finally merge all segment outputs.

Model	Param./B	Omni	Char.	Dia.	Act.	Exp.	Aud.	Overall	[email protected]
Proprietary Models
Gemini-3-flash [ 12 ]		$\checkmark$	28.8	50.3	28.2	25.5	11.2	28.8	44.3
Gemini-3-pro [ 13 ]		$\checkmark$	39.8	68.8	37.4	35.4	13.3	38.9	64.4
Gemini-2.5-flash [ 10 ]		$\checkmark$	40.1	75.5	42.8	36.5	22.8	43.6	74.3
Gemini-2.5-pro [ 11 ]		$\checkmark$	41.7	75.0	41.9	39.0	17.0	42.9	73.4
Seed-1.8 [ 3 ]		$\times$	40.9	54.4	35.1	29.6	12.4	34.5	50.7

Table 1: Comparison of Event-level metrics for SOTA models on 5-minute videos. Omni indicates whether a model supports audio input (

\checkmark

: yes,

\times

: no). In model names, -T denotes Thinking mode. For MoE models, parameters are reported as total parameters/activated parameters.

{\dagger}

indicates that a 5-minute video input to the model is first divided into 1-minute segments, which are then sequentially fed into the model. The outputs of all segments are then concatenated to form the final output.

Model	Param./B	Omni	Loc.	Type	Env.	Time	Mood	Overall	[email protected]
Proprietary Models
Gemini-3-flash [ 12 ]		$\checkmark$	54.6	59.8	42.7	54.9	50.4	52.5	70.3
Gemini-3-pro [ 13 ]		$\checkmark$	58.8	63.1	46.9	61.6	54.8	57.0	75.3
Gemini-2.5-flash [ 10 ]		$\checkmark$	52.8	57.1	45.7	56.1	50.3	52.3	69.6
Gemini-2.5-pro [ 11 ]		$\checkmark$	56.6	62.4	50.8	60.1	54.6	56.9	74.1
Seed-1.8 [ 3 ]		$\times$	57.9	58.6	47.7	58.7	52.8	55.1	74.0

Table 2: Comparison of Scene-level metrics for SOTA models on 5-minute videos. Omni indicates whether a model supports audio input (

\checkmark

: yes,

\times

: no).

{\dagger}

Panel of Figure 5 — Figure 5: Performance comparison across video durations on multiple metric dimensions.

Panel of Figure 6 — Figure 6: Additional long-video evaluation results. (a) Event-level, scene-level, and overall F1 scores across durations. (b) Event-level fine-grained F1 scores on character, action, and dialogue. (c) Event-level fine-grained Recall scores on character, action, and dialogue.

Model	CoT	Reward	Char.	Dia.	Act.	Exp.	Aud.	Overall	[email protected]
SFT	$\times$	–	35.6	68.2	30.5	31.2	11.12	35.3	66.6
SFT	$\checkmark$	–	37.8	71.0	33.5	31.2	11.5	37.0	68.9
SFT+RL	$\times$	Segmented	37.1	70.9	32.8	32.5	11.7	37.0	69.0
SFT+RL	$\checkmark$	Global	39.2	69.0	32.4	31.8	12.3	37.0	68.7
SFT+RL	$\checkmark$	Segmented	39.2	72.2	33.7	31.9	11.6	37.7	69.3

Table 3: Ablation study for training strategy.

Model	SV	Char.	Dialog	Action	Exp.	Audio	Overall	[email protected]
Qwen3VL-235B-A22B	$\checkmark$	38.1	58.6	33.0	29.1	6.0	33.0	62.0
Qwen3VL-235B-A22B	$\times$	26.2	7.7	29.8	23.5	6.0	18.6	45.1
Gemini-3-pro	$\checkmark$	39.8	68.8	37.4	35.4	13.3	38.9	64.4
Gemini-3-pro	$\times$	40.4	60.9	34.7	33.6	13.2	36.6	60.3
Ours-8B	$\checkmark$	39.2	72.2	33.7	31.9	11.6	37.7	69.3
Ours-8B	$\times$	34.1	63.8	31.8	30.6	11.7	34.4	67.0

Table 4: Comparison of event-level performance with and without masking video subtitles. SV denotes Subtitle Visible (

\checkmark

: visible,

\times

: masked).

Model	Omni	Char.	Dia.	Act.	Exp.	Aud.	Overall	[email protected]
Qwen3-VL-8B-SFT	$\times$	34.0	52.0	30.3	28.5	10.5	31.1	68.2
Ours-8B-SFT	$\checkmark$	35.6	68.2	30.5	31.2	11.1	35.3	66.6

Table 5: Comparison of event-level performance with and without audio as input.

Figure 7: Qualitative Visualization of the Character Profile Manager output ( The Croods ). The generated profile consists of two core components. Left: A comprehensive character relationship graph detailing the social topology (e.g., siblings, parent-child, and companions). Right: Fine-grained appearance descriptions for each extracted identity. This structured representation serves as a persistent semantic memory, ensuring identity consistency and logically sound interactions during long-context script generation.

Figure 8: Qualitative visualization of OmniScript outputs. OmniScript organizes each video into scene-level metadata and timestamped events with aligned characters, actions, dialogue, expressions, and audio cues, producing an interpretable and temporally coherent script representation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 22: Statistics of training set for our OmniScript model.

Figure 23: Statistics of 5-minute video clips in our benchmark.

Long-Video Understanding

Long-video understanding enables models to retrieve, remember, and reason about events across extended visual and audio-visual sequences.

See all 150 papers

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
Feb 9, 2026Linli Yao, Yuancheng Wei, Yaojie Zhang +12Audio-Visual Understanding Video Captioning
School of Computer Science, Peking University · South China University of Technology · University of Electronic Science and Technology of China +3
Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding
Apr 13, 2026Tencent Hunyuan TeamVideo Captioning Temporal Video Understanding
Narrative Aligned Long Form Video Question Answering
Mar 19, 2026Rahul Jain, Keval Doshi, Burak Uzkent +1Video Reasoning Long-Video QA
Purdue University · Amazon

Feb 9, 2026J/K move · Enter open · S save

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Linli Yao, Yuancheng Wei, Yaojie Zhang, Lei Li+11

School of Computer Science, Peking University · South China University of Technology · University of Electronic Science and Technology of China+3

Audio-Visual Understanding Video Captioning Dense Video Captioning Temporal Video Understanding

Open paper AskPDF arXiv

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are available at https://github.com/yaolinli/TimeChat-Captioner.

Abstract

Long-Video Understanding

Explore similar work

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding

Narrative Aligned Long Form Video Question Answering