Coherent multi-shot audio-visual video generation requires maintaining consistent identities while depicting evolving states and coordinating sounds across shot boundaries. Although dense captions describe individual shots in detail, they rarely specify which entities persist, how their states change, or which entities and actions produce sounds. To address this limitation, we propose Structured Stateful Relational Scripts (SSRS), a textual supervision representation that separates persistent entity and voice identities from shot-specific visual and auditory states. SSRS explicitly encodes sound-source bindings and temporal relations, while a global audio-visual event sequence connects local actions and sounds, including events that continue across shot boundaries. We further introduce MSAV-Script-100K, a large-scale dataset of curated multi-shot audio-visual videos paired with detailed SSRS annotations, enabling models to learn identity persistence, state transitions, and event continuity from structured supervision. Experiments demonstrate improvements in cross-shot consistency, adherence to specified state changes, and audio-visual alignment, highlighting the effectiveness of our SSRS and MSAV-Script-100K for coherent multi-shot audio-visual generation. Our proposed dataset is available at MSAV-Script-100K.
Figures & tables
Figure 1: Overview of the MSAV-Script-100K construction pipeline. Videos collected from YouTube and existing datasets are segmented into multi-shot clips, followed by perceptual and semantic screening and tri-dimensional assessment of visual quality, audio quality, and audio-visual synchronization, yielding a dataset that preserves consecutive shots and continuous source audio.
Figure 2: Quality assessment of MSAV-Script-100K against raw data. The final dataset exhibits higher visual aesthetics, perceptual quality, and motion smoothness (a–c), stronger audio-visual synchronization (d), and improved audio enjoyment, usefulness, and production quality (e, f, h). Audio production complexity (g) characterizes differences in audio composition.
Figure 3: From content acquisition to structured supervision. (a) Multi-level expert annotations train a multi-shot audio-visual captioner via SFT and DPO. (b) Dense captions for MSAV-Script-100K undergo source-constrained SSRS conversion and quality control.
Figure 4: SSRS distinguishes persistent identities from evolving states. Global context and shared registries (a) are separated from shot-level presentation, states, and relations (b) . The global event sequence (c) links events across cuts: the same utterance continues despite changing speaker visibility, highlighting that shot boundaries need not coincide with event boundaries .
Global
Cross-shot
Intra-shot
Method
Narr ↑
Lip ↑
Attr ↑
Sync ↑
VQ ↑
C-L ↑
VC ↑
Mus ↑
Spk ↑
I-L ↑
Cam ↑
PQ ↑
OCR ↑
WER-S ↑
Ovi
0.3288
0.0605
0.2864
0.8290
0.5434
0.4333
0.6824
0.4978
0.3094
0.9136
0.5316
0.5733
0.2222
0.0065
JavisDiT++
0.4932
0.0084
0.0668
0.8073
0.6499
0.6731
0.6747
0.5055
0.4651
0.6288
0.6245
0.4981
0.1379
0.0016
JavisGPT
0.2170
0.0022
0.0101
0.7780
0.3813
0.6400
0.7038
0.4992
0.0659
0.8333
0.4281
0.5575
0.0053
0.0028
ShotStream+HF
0.6844
0.1384
0.2762
0.7863
0.5943
0.8704
0.8720
0.4722
0.1860
0.9151
0.6377
0.6641
0.1734
0.0033
LTX-2.5
0.7883
0.1242
0.2500
0.7727
0.7269
0.8152
0.8664
0.4673
0.1418
0.8902
0.7529
0.6416
0.2324
0.7226
Table 1: Comparison on the 150-prompt short-video T2AV subset of MSAVBench. All scores are normalized to [0,1] , where higher values indicate better performance. Both Dense SFT and SSRS SFT are fine-tuned from the pretrained LTX-2.3 model. C-L and I-L denote C-Layout and I-Layout, respectively, while WER-S is computed as WER-S=clip(1−e,0,1) , where e denotes the applicable transcription error rate (WER or CER). Best and second-best results are highlighted in bold and underlined italics , respectively.
Dense SFT
SSRS SFT
Δ
Category
T↑
P↑
T↑
P↑
ΔT
ΔP
Visual-state evolution (VS)
20.00
62.50
60.00
42.50
+40.00
-20.00
Cross-shot audio continuity (AC)
0.00
59.38
50.00
62.50
+50.00
+3.12
Non-speech source binding (NS)
5.00
45.00
45.00
60.00
+40.00
+15.00
Cross-shot event order (EO)
0.00
50.00
40.00
55.00
+40.00
+5.00
On-/off-screen speaker binding (SB)
7.14
25.00
42.86
42.86
+35.71
+17.86
Table 2: Controlled evaluation of state and relational following. T denotes target satisfaction and P denotes preservation rate. All values are percentages. Δ denotes SSRS SFT minus Dense SFT.
Cross-shot Organization
Audio-Visual Relations
Control Metrics
Representation
Narr ↑
Mus ↑
Spk ↑
Lip ↑
Attr ↑
Sync ↑
VQ ↑
Cam ↑
Full SSRS
0.8467
0.7523
0.4006
0.2468
0.5265
0.8573
0.8376
0.8321
w/o Audio-Visual Relations
0.8622
0.7131
0.4268
0.2374
0.5110
0.8387
0.8137
0.8420
w/o Global Registries and Event
0.8289
0.7168
0.3801
0.2530
0.5003
0.8477
0.8189
0.8218
Table 3: Representation ablations under matched training and evaluation settings. All scores are normalized to [0,1] , where higher values indicate better performance.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: An example from MSAV-Script-100K with SSRS annotations. Sampled video frames and the audio waveform are shown together with shared entity and voice registries, global audio-visual context, shot-local states and source relations, and a cross-shot event sequence. The highlighted panel shows a shot-level annotation excerpt.
Figure 6: Thematic taxonomy (left) of MSAV-Script-100K and word cloud (right) of SSRS, illustrating the dataset’s topic hierarchy and frequently occurring words in its captions.
Group
Statistic
Value
Scale
Number of selected clips
100,000
Total duration in hours
237.426
Sources
Clip counts by source
Table 6
Temporal structure
Mean clip duration (s)
8.547
Median clip duration (s)
7.966
Clip duration range (s)
6.083–12.040
Appendix
Table 4: Dataset statistics for the selected 100,000 clips. Audio tags and shot counts are automatic estimates. Audio categories may overlap.
Table 11
Figure 7: Quality comparison between MSAV-Script-100K and FineVideo.
Metric m
Retention condition
Numerical threshold
Sharpness
m≥P5(m)
10.278426
Image entropy
m≥P5(m)
5.359885
Flat-region proportion
m≤P95(m)
0.906250
Texture score
m≤P95(m)
1.128293
MUSIQ image quality
m≥P5(m)
0.391477
RAFT motion magnitude
P5(m)≤m≤P95(m)
[0.004140,0.046509]
Appendix
Table 7: Visual screening criteria. Numerical thresholds are rounded for display; filtering uses the stored full-precision values. MUSIQ scores are divided by 100, and RAFT magnitudes are normalized by the evaluation-frame diagonal.
Axis
Criterion
Evaluator
Visual
Visual aesthetics
LAION Aesthetic Predictor
Visual
Motion smoothness
AMT
Audio
Audio aesthetics and production quality
AudioBox
Audio
Speech quality
DNSMOS and NISQA
Audio-visual
Temporal synchronization
Synchformer zero-offset probability
Appendix
Table 8: Five criteria used for quality assessment. Each criterion contributes one fifth of the overall score after within-source percentile conversion and criterion-level aggregation.
Figure 8: Ordered event progression. SSRS better follows the intended fire-lighting sequence, while competing methods exhibit repeated, premature, or weakly ordered states.
Figure 9: Multi-stage state evolution. SSRS more clearly aligns successive planting states with the requested shot sequence, whereas competing methods frequently repeat or collapse stages.
Figure 10: Dialogue-driven shot organization. SSRS preserves recurring identities while adapting shot composition to changing speaker and interaction states.
Figure 11: Role-consistent cross-shot event progression. SSRS preserves the leader identity while following the requested point–run–grasp sequence across shots.
Figure 12: Fine-grained object-state evolution. SSRS maintains the toolbox–cart relation while explicitly realizing the requested lowering, strap-tightening, and secured final state.
Figure 13: Sound-source binding. SSRS realizes the requested interaction with the designated blue bell, whereas Dense SFT misses the strike and exhibits object drift across shots.
Figure 14: Cross-shot object-state persistence. SSRS preserves the backpack–bench relation across shot transitions, whereas Dense SFT incorrectly transfers the backpack to a traveler.
Figure 15: Non-speech sound-source binding. SSRS realizes the requested zipper interaction on the designated red bag, whereas Dense SFT leaves both bags largely static. Zipper synchronization is verified using the native generated audio.
Figure 16: Speaker identity binding. SSRS preserves the target repairer across shots and associates the requested speech with repairer B, whereas Dense SFT exhibits identity drift. Speaker activity is verified using the native audio.
Figure 17: Cross-shot audio continuity. SSRS maintains a single guitar event across changing visual views, illustrating that auditory events can persist across shot boundaries, whereas Dense SFT fails to realize the requested multi-shot progression.
Hyperparameter
Configuration
Backbone
Qwen3-Omni
GPU Usage
8× A100 80 GB
Batch Size per GPU
1
Gradient Accumulation
2
Effective Batch Size
16
Training Time
68 hours
Appendix
Table 9: Training hyperparameters of the Multi-Shot Audio-Visual Captioner.
Hyperparameter
Configuration
Backbone
LTX-2.3 22B
GPU Usage
2×8 H800 80 GB
Batch Size per GPU
1
Gradient Accumulation
1
Effective Batch Size
16
Training Resolution
768×512
Appendix
Table 10: Training hyperparameters of the LTX-2.3 audio-visual DiT.
Metric
Dimension
Measurement
Normalization
Global metrics
Narr
Narrative coherence
Binary visual judgments of requested events and their narrative order.
B(Qnarr) .
Lip
Lip–speech synchronization
SyncNet synchronization confidence c and signed frame offset o for matched speaking tracks.
0.7C(c/6)+0.3C(1−∣o∣/8) ; aggregated across tracks using visual-duration weights.
Attr
Speaker attribution
Match diarized audio speakers to active visual speaker tracks.
∑kdkI[mk]/∑kdk , where dk is speech duration and mk indicates a successful assignment.
Sync
Audiovisual synchronization
Synchformer estimates signed delays δfirst and δlast for the first and last feature blocks, in seconds.
C(1−d/2) , where d=(∣δfirst∣+∣δlast∣)/2 .
VQ
Visual realization
Answer a shared set of visual multiple-choice questions constructed from the evaluation prompt.
Number of correct answers divided by the total number of questions.
Appendix
Table 11: Definitions and computation of the 14 metrics in our MSAVBench-derived evaluation. All normalized scores lie in [0,1] ; higher is better.
While recent generative models produce high-fidelity videos, they struggle with the complex narrative control required for coherent multi-shot audio-visual generation. Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting. In this paper, we propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control. To resolve temporal misalignment, we propose boundary-aware attention, which leverages hierarchical captions and boundary-aware token routing to render audio-visual elements within their respective temporal boundaries. To improve the controllability for multi-subject scenarios, we propose ID-aware propagation, utilizing identity embeddings and an identity-aware mask to bind specific identities to consistent visual appearances and vocal timbres. To provide comprehensive audio-visual narratives, we present a multi-agent scripting pipeline to transform free-form user inputs into hierarchical captions. Furthermore, we construct MAVINSet, a multi-shot audio-visual dataset for robust training and evaluation. Extensive experiments demonstrate that MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking workflows.
Kaiqi Liu, Yunyao Mao, Ziqi Cai +8
Peking University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences +1
Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of 200 multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.
Jiehui Huang, Yuechen Zhang, Bin Xia +7
The Hong Kong University of Science and Technology · Kling Team, Kuaishou Technology · The Chinese University of Hong Kong +2
Advances in Multimodal Large Language Models (MLLMs) are transforming video captioning from a descriptive endpoint into a semantic interface for both video understanding and generation. However, the dominant paradigm still casts videos as monolithic narrative paragraphs that entangle visual, auditory, and identity information. This dense coupling not only compromises representational fidelity but also limits scalability, since even local edits can trigger global rewrites. To address this structural bottleneck, we propose Multi-Stream Scene Script (MTSS), a novel paradigm that replaces monolithic text with factorized and explicitly grounded scene descriptions. MTSS is built on two core principles: Stream Factorization, which decouples a video into complementary streams (Reference, Shot, Event, and Global), and Relational Grounding, which reconnects these isolated streams through explicit identity and temporal links to maintain holistic video consistency. Extensive experiments demonstrate that MTSS consistently enhances video understanding across various models, achieving an average reduction of 25% in the total error rate on Video-SALMONN-2 and an average performance gain of 67% on the Daily-Omni reasoning benchmark. It also narrows the performance gap between smaller and larger MLLMs, indicating a substantially more learnable caption interface. Finally, even without architectural adaptation, replacing monolithic prompts with MTSS in multi-shot video generation yields substantial human-rated improvements: a 45% boost in cross-shot identity consistency, a 56% boost in audio-visual alignment, and a 71% boost in temporal controllability.