Découpage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation
Organizations: The Hong Kong University of Science and Technology · Ant Group · Zhejiang University
Abstract
In cinematic storytelling, sound and visual transitions do not always coincide in lockstep: dialogue frequently bridges across cuts to reveal reactions or introduces new scenes via J- and L-cuts, while deliberate silences pace the scene. Existing audio-visual generative frameworks, however, lack explicit mechanisms to coordinate independent visual and acoustic timing over extended horizons. As narrative duration and shot complexity scale, these systems struggle to maintain long-range dialogue pacing, often hallucinating speech during intended silences or erroneously driving mouth movements on silent listeners. To enable fine-grained cinematic control, we present DECOUPAGE, a screenplay-driven framework for long-form, multi-shot, and multi-character joint audio-video generation. Rather than coupling modality boundaries by default, DECOUPAGE uncouples generation into independently bounded visual and acoustic tracks governed by persistent character identities. DECOUPAGE builds on a joint audio-video diffusion model with dual-schedule temporal cross-attention, incorporating a directorial Lip-sync mechanism that selectively decouples speech conditioning from visual facial dynamics during off-screen dialogue. On multi-shot conversational benchmarks, DECOUPAGE achieves precise control over shot transitions, audio timing, and visibility-aware lip synchronization, directing 15- to 60-second screenplay-driven audio-video generation with multimodal coherence.
Figures & tables
| Video Quality | Audio Quality | Voice Identity | AV Align. | |||||||
| Model | Cut F1 | Cnt MAE | Imaging | WER | Cont. | Leak | PQ | Consist. | Margin | Sync-C |
| 15-second Evaluation Dataset | ||||||||||
| LTX-2.3 Base | 0.26 | 1.23 | 0.68 | 35.69 | 52.68 | 18.69 | 6.18 | 0.42 | 0.12 | 2.52 |
| MiniMax-H3 | 0.28 | 0.48 | 0.70 | 10.23 | 41.95 | 27.28 | 7.26 | 0.50 | 0.14 | 4.88 |
| MOVA (I2VA) | 0.18 | 7.38 | 0.56 | 38.22 | 39.33 | 28.47 | 6.26 | 0.36 | 0.01 | 3.24 |
| Ours | 0.99 | 0.05 | 0.68 | 4.59 | 93.01 | 1.12 | 6.91 | 0.53 | 0.29 | 3.59 |
| Video Quality | Audio Quality | AV Alignment | Efficiency | ||||||
| Configuration | Cut F1 | Imaging | WER | Cont. | Leak | Sim. | Bias (L) | Sync-C (S) | Time (s/step) |
| Dual-schedule temporal CA | |||||||||
| w/o Temporal CA | 0.34 | 0.65 | 6.57 | 39.34 | 25.98 | 0.39 | 2.31 | 3.16 | 12.81 |
| Directorial Lip-Sync Mech. | |||||||||
| w/o Mechanism | 0.99 | 0.66 | 4.38 | 90.89 | 0.74 | 0.55 | 1.78 | 3.53 | 12.81 |
| w/ Mechanism ( Ours ) | 0.99 | 0.66 | 4.16 | 92.83 | 0.77 | 0.56 | 0.33 | 3.42 | 13.01 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Audio-Video Generation
Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.
See all 166 papers