In cinematic storytelling, sound and visual transitions do not always coincide in lockstep: dialogue frequently bridges across cuts to reveal reactions or introduces new scenes via J- and L-cuts, while deliberate silences pace the scene. Existing audio-visual generative frameworks, however, lack explicit mechanisms to coordinate independent visual and acoustic timing over extended horizons. As narrative duration and shot complexity scale, these systems struggle to maintain long-range dialogue pacing, often hallucinating speech during intended silences or erroneously driving mouth movements on silent listeners. To enable fine-grained cinematic control, we present DECOUPAGE, a screenplay-driven framework for long-form, multi-shot, and multi-character joint audio-video generation. Rather than coupling modality boundaries by default, DECOUPAGE uncouples generation into independently bounded visual and acoustic tracks governed by persistent character identities. DECOUPAGE builds on a joint audio-video diffusion model with dual-schedule temporal cross-attention, incorporating a directorial Lip-sync mechanism that selectively decouples speech conditioning from visual facial dynamics during off-screen dialogue. On multi-shot conversational benchmarks, DECOUPAGE achieves precise control over shot transitions, audio timing, and visibility-aware lip synchronization, directing 15- to 60-second screenplay-driven audio-video generation with multimodal coherence.
Figures & tables
Figure 1: Screenplay-directed audio-video generation. Découpage directs visual and audio timing tracks across multi-shot scenes. Four 15-second examples demonstrate precise control over speech bridging across cuts (J-/L-cuts) and silent listener reactions, with all shot transitions and synthesized audio strictly following the screenplay timeline.
Figure 2: Joint audio-video data pipeline. Adjacent shots form audiovisual sequences. Faster-Whisper establishes a fixed speech timeline and filters out empty or non-English transcripts. Gemini 3.0 Pro then produces visual and audio captions with persistent character identities. Finally, adjacent speech events assigned to the same character are grouped into character-level audio intervals.
Figure 3: Overview of Découpage . (a) Dual-track temporal cross-attention guides joint audio-video generation along independent visual and acoustic schedules, allowing dialogue to continue across shot boundaries. (b) The directorial lip-sync mechanism blends speech and no-speech A2V residuals based on speaker visibility to suppress unintended listener articulation.
Video Quality
Audio Quality
Voice Identity
AV Align.
Model
Cut F1 ↑
Cnt MAE ↓
Imaging ↑
WER ↓
Cont. ↑
Leak ↓
PQ ↑
Consist. ↑
Margin ↑
Sync-C ↑
15-second Evaluation Dataset
LTX-2.3 Base
0.26
1.23
0.68
35.69
52.68
18.69
6.18
0.42
0.12
2.52
MiniMax-H3
0.28
0.48
0.70
10.23
41.95
27.28
7.26
0.50
0.14
4.88
MOVA (I2VA)
0.18
7.38
0.56
38.22
39.33
28.47
6.26
0.36
0.01
3.24
Ours
0.99
0.05
0.68
4.59
93.01
1.12
6.91
0.53
0.29
3.59
Table 1: Quantitative results on the 15-second and 60-second benchmarks. Metrics WER, Cont., and Leak are reported in percentage (%). Bold font indicates the best performance within each duration group.
Figure 4: Qualitative comparison on a five-shot screenplay. Columns present shot prompts and the target dialogue schedule; rows show keyframes and synthesized audio waveforms across models. Blue/orange highlights mark each character’s speech windows, and gray denotes silence. Découpage adheres to the multi-shot structure across transitions, respects the Shot-4 listener, and correctly synthesizes the Shot-5 slience and lighter flick, achieving superior audio-visual alignment.
Video Quality
Audio Quality
AV Alignment
Efficiency
Configuration
Cut F1 ↑
Imaging ↑
WER ↓
Cont. ↑
Leak ↓
Sim. ↑
Bias (L) ↓
Sync-C (S) ↑
Time (s/step) ↓
Dual-schedule temporal CA
w/o Temporal CA
0.34
0.65
6.57
39.34
25.98
0.39
2.31
3.16
12.81
Directorial Lip-Sync Mech.
w/o Mechanism
0.99
0.66
4.38
90.89
0.74
0.55
1.78
3.53
12.81
w/ Mechanism ( Ours )
0.99
0.66
4.16
92.83
0.77
0.56
0.33
3.42
13.01
Table 2: Ablation study on core components. Evaluated on a fixed subset of the 15-second conversational benchmark. Best values within each block are in bold . Notably, the lip-sync mechanism is entirely training-free: both configurations in the second block operate on the exact same checkpoint. Runtime denotes seconds per denoising step measured on a single NVIDIA H20 GPU.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative example of a 60-second generation. Eight representative frames show a two-character exchange unfolding across multiple shots in a trumpet shop. Beneath each row, the generated audio waveform is aligned with the corresponding shot sequence. The colored strips indicate the screenplay-specified speaking role (C1 or C2), while gray segments denote intervals with no requested speech. The example visualizes coordinated shot transitions, character turns, and speech–silence timing over a full-minute sequence.
Figure 6: Matched comparison of the Directorial Lip-Sync Mechanism near the end of Shot 2. The first row shows results without the mechanism, and the second row shows results with it, at 5.90, 6.15, and 6.40 seconds. Yellow dashed boxes highlight the mouth region.
Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.
While recent generative models produce high-fidelity videos, they struggle with the complex narrative control required for coherent multi-shot audio-visual generation. Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting. In this paper, we propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control. To resolve temporal misalignment, we propose boundary-aware attention, which leverages hierarchical captions and boundary-aware token routing to render audio-visual elements within their respective temporal boundaries. To improve the controllability for multi-subject scenarios, we propose ID-aware propagation, utilizing identity embeddings and an identity-aware mask to bind specific identities to consistent visual appearances and vocal timbres. To provide comprehensive audio-visual narratives, we present a multi-agent scripting pipeline to transform free-form user inputs into hierarchical captions. Furthermore, we construct MAVINSet, a multi-shot audio-visual dataset for robust training and evaluation. Extensive experiments demonstrate that MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking workflows.
Kaiqi Liu, Yunyao Mao, Ziqi Cai +8
Peking University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences +1
Recent diffusion and autoregressive models have substantially improved text-to-video generation, yet producing coherent long-form story videos with consistent characters and aligned audio remains challenging. Existing methods often suffer from narrative drift, unstable character identity, weak cross-scene continuity, and audio-visual mismatch over extended sequences. We propose DramaAgent, a hierarchical, agentic, and model-agnostic framework for long-form text-to-video-and-audio generation. Rather than improving the underlying video backbone itself, DramaAgent introduces an upper-level control layer that decomposes generation into story planning, persistent character conditioning, scene-wise synthesis, and reflection-guided targeted repair. The framework maintains reusable story and character states across scenes, diagnoses failures such as identity drift, missing scene semantics, temporal discontinuity, and cross-modal mismatch, and repairs problematic clips in a stage-specific manner. Experiments across multiple video generation backbones show that DramaAgent improves long-horizon coherence, character consistency, narrative fidelity, and scene-level audio-visual consistency over direct generation and strong baselines. These results suggest that hierarchical agentic control is a practical direction for controllable long-form audiovisual generation. Code: https://github.com/AIGeeksGroup/DramaAgent. Website: https://aigeeksgroup.github.io/DramaAgent.