Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation

Chen Zhao; Lehan Shen; Qilong Huang; Hongyu Li; Zhuoliang Kang; Xiaoming Wei; Jian Yang; Ying Tai

cs.CVOct 11, 2026

Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation

Authors: Chen Zhao, Lehan Shen, Qilong Huang, Hongyu Li, Zhuoliang Kang, Xiaoming Wei, Jian Yang, Ying Tai

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · Meituan, China

Abstract

Coherent multi-shot audio-visual video generation requires maintaining consistent identities while depicting evolving states and coordinating sounds across shot boundaries. Although dense captions describe individual shots in detail, they rarely specify which entities persist, how their states change, or which entities and actions produce sounds. To address this limitation, we propose Structured Stateful Relational Scripts (SSRS), a textual supervision representation that separates persistent entity and voice identities from shot-specific visual and auditory states. SSRS explicitly encodes sound-source bindings and temporal relations, while a global audio-visual event sequence connects local actions and sounds, including events that continue across shot boundaries. We further introduce MSAV-Script-100K, a large-scale dataset of curated multi-shot audio-visual videos paired with detailed SSRS annotations, enabling models to learn identity persistence, state transitions, and event continuity from structured supervision. Experiments demonstrate improvements in cross-shot consistency, adherence to specified state changes, and audio-visual alignment, highlighting the effectiveness of our SSRS and MSAV-Script-100K for coherent multi-shot audio-visual generation. Our proposed dataset is available at MSAV-Script-100K.

Figures & tables

Figure 1: Overview of the MSAV-Script-100K construction pipeline. Videos collected from YouTube and existing datasets are segmented into multi-shot clips, followed by perceptual and semantic screening and tri-dimensional assessment of visual quality, audio quality, and audio-visual synchronization, yielding a dataset that preserves consecutive shots and continuous source audio.

Figure 2: Quality assessment of MSAV-Script-100K against raw data. The final dataset exhibits higher visual aesthetics, perceptual quality, and motion smoothness (a–c), stronger audio-visual synchronization (d), and improved audio enjoyment, usefulness, and production quality (e, f, h). Audio production complexity (g) characterizes differences in audio composition.

Figure 3: From content acquisition to structured supervision. (a) Multi-level expert annotations train a multi-shot audio-visual captioner via SFT and DPO. (b) Dense captions for MSAV-Script-100K undergo source-constrained SSRS conversion and quality control.

Figure 4: SSRS distinguishes persistent identities from evolving states. Global context and shared registries (a) are separated from shot-level presentation, states, and relations (b) . The global event sequence (c) links events across cuts: the same utterance continues despite changing speaker visibility, highlighting that shot boundaries need not coincide with event boundaries .

	Global					Cross-shot				Intra-shot
Method	Narr $\uparrow$	Lip $\uparrow$	Attr $\uparrow$	Sync $\uparrow$	VQ $\uparrow$	C-L $\uparrow$	VC $\uparrow$	Mus $\uparrow$	Spk $\uparrow$	I-L $\uparrow$	Cam $\uparrow$	PQ $\uparrow$	OCR $\uparrow$	WER-S $\uparrow$
Ovi	0.3288	0.0605	0.2864	0.8290	0.5434	0.4333	0.6824	0.4978	0.3094	0.9136	0.5316	0.5733	0.2222	0.0065
JavisDiT++	0.4932	0.0084	0.0668	0.8073	0.6499	0.6731	0.6747	0.5055	0.4651	0.6288	0.6245	0.4981	0.1379	0.0016
JavisGPT	0.2170	0.0022	0.0101	0.7780	0.3813	0.6400	0.7038	0.4992	0.0659	0.8333	0.4281	0.5575	0.0053	0.0028
ShotStream+HF	0.6844	0.1384	0.2762	0.7863	0.5943	0.8704	0.8720	0.4722	0.1860	0.9151	0.6377	0.6641	0.1734	0.0033
LTX-2.5	0.7883	0.1242	0.2500	0.7727	0.7269	0.8152	0.8664	0.4673	0.1418	0.8902	0.7529	0.6416	0.2324	0.7226

Table 1: Comparison on the 150-prompt short-video T2AV subset of MSAVBench. All scores are normalized to

[0,1]

, where higher values indicate better performance. Both Dense SFT and SSRS SFT are fine-tuned from the pretrained LTX-2.3 model. C-L and I-L denote C-Layout and I-Layout, respectively, while WER-S is computed as

\mathrm{WER\text{-}S}=\operatorname{clip}(1-e,0,1)

, where

e

denotes the applicable transcription error rate (WER or CER). Best and second-best results are highlighted in bold and underlined italics , respectively.

Category	$T\uparrow$	$P\uparrow$	$T\uparrow$	$P\uparrow$	$\Delta T$	$\Delta P$
	Dense SFT		SSRS SFT		$\Delta$
Visual-state evolution (VS)	20.00	62.50	60.00	42.50	+40.00	-20.00
Cross-shot audio continuity (AC)	0.00	59.38	50.00	62.50	+50.00	+3.12
Non-speech source binding (NS)	5.00	45.00	45.00	60.00	+40.00	+15.00
Cross-shot event order (EO)	0.00	50.00	40.00	55.00	+40.00	+5.00
On-/off-screen speaker binding (SB)	7.14	25.00	42.86	42.86	+35.71	+17.86

Table 2: Controlled evaluation of state and relational following.

T

denotes target satisfaction and

P

denotes preservation rate. All values are percentages.

\Delta

denotes SSRS SFT minus Dense SFT.

	Cross-shot Organization			Audio-Visual Relations			Control Metrics
Representation	Narr $\uparrow$	Mus $\uparrow$	Spk $\uparrow$	Lip $\uparrow$	Attr $\uparrow$	Sync $\uparrow$	VQ $\uparrow$	Cam $\uparrow$
Full SSRS	0.8467	0.7523	0.4006	0.2468	0.5265	0.8573	0.8376	0.8321
w/o Audio-Visual Relations	0.8622	0.7131	0.4268	0.2374	0.5110	0.8387	0.8137	0.8420
w/o Global Registries and Event	0.8289	0.7168	0.3801	0.2530	0.5003	0.8477	0.8189	0.8218

Table 3: Representation ablations under matched training and evaluation settings. All scores are normalized to

[0,1]

, where higher values indicate better performance.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: An example from MSAV-Script-100K with SSRS annotations. Sampled video frames and the audio waveform are shown together with shared entity and voice registries, global audio-visual context, shot-local states and source relations, and a cross-shot event sequence. The highlighted panel shows a shot-level annotation excerpt.

Panel of Figure 6 — Figure 6: Thematic taxonomy (left) of MSAV-Script-100K and word cloud (right) of SSRS, illustrating the dataset’s topic hierarchy and frequently occurring words in its captions.

Group	Statistic	Value
Scale	Number of selected clips	100,000
	Total duration in hours	237.426
Sources	Clip counts by source	Table 6
Temporal structure	Mean clip duration (s)	8.547
	Median clip duration (s)	7.966
	Clip duration range (s)	6.083–12.040

Appendix

Table 4: Dataset statistics for the selected 100,000 clips. Audio tags and shot counts are automatic estimates. Audio categories may overlap.

Table 11

Figure 7: Quality comparison between MSAV-Script-100K and FineVideo.

Metric $m$	Retention condition	Numerical threshold
Sharpness	$m\geq P_{5}(m)$	10.278426
Image entropy	$m\geq P_{5}(m)$	5.359885
Flat-region proportion	$m\leq P_{95}(m)$	0.906250
Texture score	$m\leq P_{95}(m)$	1.128293
MUSIQ image quality	$m\geq P_{5}(m)$	0.391477
RAFT motion magnitude	$P_{5}(m)\leq m\leq P_{95}(m)$	$[0.004140,0.046509]$

Appendix

Table 7: Visual screening criteria. Numerical thresholds are rounded for display; filtering uses the stored full-precision values. MUSIQ scores are divided by 100, and RAFT magnitudes are normalized by the evaluation-frame diagonal.

Axis	Criterion	Evaluator
Visual	Visual aesthetics	LAION Aesthetic Predictor
Visual	Motion smoothness	AMT
Audio	Audio aesthetics and production quality	AudioBox
Audio	Speech quality	DNSMOS and NISQA
Audio-visual	Temporal synchronization	Synchformer zero-offset probability

Appendix

Table 8: Five criteria used for quality assessment. Each criterion contributes one fifth of the overall score after within-source percentile conversion and criterion-level aggregation.

Figure 8: Ordered event progression. SSRS better follows the intended fire-lighting sequence, while competing methods exhibit repeated, premature, or weakly ordered states.

Figure 9: Multi-stage state evolution. SSRS more clearly aligns successive planting states with the requested shot sequence, whereas competing methods frequently repeat or collapse stages.

Figure 10: Dialogue-driven shot organization. SSRS preserves recurring identities while adapting shot composition to changing speaker and interaction states.

Figure 11: Role-consistent cross-shot event progression. SSRS preserves the leader identity while following the requested point–run–grasp sequence across shots.

Figure 12: Fine-grained object-state evolution. SSRS maintains the toolbox–cart relation while explicitly realizing the requested lowering, strap-tightening, and secured final state.

Figure 13: Sound-source binding. SSRS realizes the requested interaction with the designated blue bell, whereas Dense SFT misses the strike and exhibits object drift across shots.

Figure 14: Cross-shot object-state persistence. SSRS preserves the backpack–bench relation across shot transitions, whereas Dense SFT incorrectly transfers the backpack to a traveler.

Figure 15: Non-speech sound-source binding. SSRS realizes the requested zipper interaction on the designated red bag, whereas Dense SFT leaves both bags largely static. Zipper synchronization is verified using the native generated audio.

Figure 16: Speaker identity binding. SSRS preserves the target repairer across shots and associates the requested speech with repairer B, whereas Dense SFT exhibits identity drift. Speaker activity is verified using the native audio.

Figure 17: Cross-shot audio continuity. SSRS maintains a single guitar event across changing visual views, illustrating that auditory events can persist across shot boundaries, whereas Dense SFT fails to realize the requested multi-shot progression.

Hyperparameter	Configuration
Backbone	Qwen3-Omni
GPU Usage	$8\times$ A100 80 GB
Batch Size per GPU	1
Gradient Accumulation	2
Effective Batch Size	16
Training Time	68 hours

Appendix

Table 9: Training hyperparameters of the Multi-Shot Audio-Visual Captioner.

Hyperparameter	Configuration
Backbone	LTX-2.3 22B
GPU Usage	$2\times 8$ H800 80 GB
Batch Size per GPU	1
Gradient Accumulation	1
Effective Batch Size	16
Training Resolution	$768\times 512$

Appendix

Table 10: Training hyperparameters of the LTX-2.3 audio-visual DiT.

Global metrics
Metric	Dimension	Measurement	Normalization
Narr	Narrative coherence	Binary visual judgments of requested events and their narrative order.	$B(\mathcal{Q}_{\mathrm{narr}})$ .
Lip	Lip–speech synchronization	SyncNet synchronization confidence $c$ and signed frame offset $o$ for matched speaking tracks.	$0.7C(c/6)+0.3C(1-\|o\|/8)$ ; aggregated across tracks using visual-duration weights.
Attr	Speaker attribution	Match diarized audio speakers to active visual speaker tracks.	$\sum_{k}d_{k}\mathbb{I}[m_{k}]/\sum_{k}d_{k}$ , where $d_{k}$ is speech duration and $m_{k}$ indicates a successful assignment.
Sync	Audiovisual synchronization	Synchformer estimates signed delays $\delta_{\mathrm{first}}$ and $\delta_{\mathrm{last}}$ for the first and last feature blocks, in seconds.	$C(1-d/2)$ , where $d=(\|\delta_{\mathrm{first}}\|+\|\delta_{\mathrm{last}}\|)/2$ .
VQ	Visual realization	Answer a shared set of visual multiple-choice questions constructed from the evaluation prompt.	Number of correct answers divided by the total number of questions.

Appendix

Table 11: Definitions and computation of the 14 metrics in our MSAVBench-derived evaluation. All normalized scores lie in

[0,1]

; higher is better.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
Jun 28, 2026Kaiqi Liu, Yunyao Mao, Ziqi Cai +8Audio-Video Generation Audio-Visual Alignment
Peking University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences +1
UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating
Jun 19, 2026Jiehui Huang, Yuechen Zhang, Bin Xia +7Audio-Video Generation Memory-Augmented Video Generation
The Hong Kong University of Science and Technology · Kling Team, Kuaishou Technology · The Chinese University of Hong Kong +2
Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding
Apr 13, 2026Tencent Hunyuan TeamVideo Captioning Temporal Video Understanding

Jun 28, 2026J/K move · Enter open · S save

MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control

Kaiqi Liu, Yunyao Mao, Ziqi Cai, Zheng Geng+7

Peking University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences+1

Audio-Video Generation Audio-Visual Alignment Multi-Shot Video Generation Controllable Video Generation

Open paper AskPDF arXiv

While recent generative models produce high-fidelity videos, they struggle with the complex narrative control required for coherent multi-shot audio-visual generation. Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting. In this paper, we propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control. To resolve temporal misalignment, we propose boundary-aware attention, which leverages hierarchical captions and boundary-aware token routing to render audio-visual elements within their respective temporal boundaries. To improve the controllability for multi-subject scenarios, we propose ID-aware propagation, utilizing identity embeddings and an identity-aware mask to bind specific identities to consistent visual appearances and vocal timbres. To provide comprehensive audio-visual narratives, we present a multi-agent scripting pipeline to transform free-form user inputs into hierarchical captions. Furthermore, we construct MAVINSet, a multi-shot audio-visual dataset for robust training and evaluation. Extensive experiments demonstrate that MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking workflows.

Abstract

Audio-Video Generation

Explore similar work

MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control

UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding