Organizations: The Hong Kong University of Science and Technology · KlingAI Research · The Chinese University of Hong Kong · Northwestern Polytechnical University · Dalian University of Technology
Recent advances in text-to-video generation have enabled high-fidelity single-shot synthesis, yet practical storytelling creation requires multi-shot videos with both strong inter-shot consistency and fine-grained controllability. Existing multi-shot video generation approaches achieve strong consistency but still fall short in controllability. In this paper, we propose Mise-en-Video (MeV), a highly controllable multi-shot video generation framework that allows video creators to conduct staging the multi-shot in the 3D space and then generate the corresponding multi-shot video. To achieve this, our framework combines a hierarchical attention mechanism for multi-level narrative consistency with 3D-aware layout guidance for precise, per-shot spatial control. To support training and evaluation, we develop standardized annotation pipelines and construct a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes. Extensive experiments demonstrate that MeV consistently outperforms strong baselines in video quality, inter-shot consistency, and subject-level controllability.
Figures & tables
Figure 2: Overview of MeV. Creators first design the 3D scene and provide a hierarchical caption. We project the 3D layout into an NOCS-based encoding scheme and use it as guidance for ControlDiT. Meanwhile, we decompose the caption into global-, shot-, and subject-level descriptions, and fuse them via a Hierarchical Context Attention mechanism to enhance narrative consistency and subject coherence across shots.
Figure 3: Visualization of the Hierarchical Context Attention Mask, where each visual token attends only to its corresponding text tokens via cross-attention.
Figure 4: Multi-shot data annotation pipeline. First, we collect videos from the Internet and segment them into shots using TransNetV2 [ 34 ] to form multi-shot samples. Second, we construct 3D layouts by detecting and tracking subjects (Detectron2 [ 44 ] + SAM2 [ 33 ] ), recovering human meshes (CameraHMR [ 31 ] ), reconstructing scene depth and aligning meshes to a unified coordinate system (Pi3 [ 42 ] ), then encoding 3D boxes in NOCS and projecting them to 2D layout sequences. Third, we generate hierarchical captions (global/shot/subject) with Gemini, including consistent subject IDs across shots.
Method
Inter-shot Cons.
Intra-shot Cons.
Text Align. ↑
Trans. Control ↑
Aesthetic Quality ↑
Narrative Coherence ↑
Semantic ↑
Subject ↑
Background ↑
Subject ↑
Background ↑
HoloCine (14B)
0.6423
0.5955
0.5385
0.8101
0.7099
0.1986
0.9826
0.5598
0.8839
EchoShot
0.5829
0.4492
0.4070
0.6717
0.6251
0.1987
0.9321
0.5414
0.8393
MultiShotMaster
0.5973
0.4751
0.5220
0.7150
0.5962
0.1914
0.9563
0.5452
0.8810
Ours
0.6175
0.5499
0.5762
0.7630
0.6844
0.1991
0.9812
0.5459
0.8869
Table 1: Quantitative results for multi-shot video generation. Among the 1B models, the best results are highlighted in boldface , while the second best are underlined . Here, Sub., Bg., Cons., and Align. denote Subject, Background, Consistency, and Alignment, respectively.
Method
Mean Diff (rad)
OS Yaw ↑
mAOE (rad) ↓
Yaw ↓
Pitch ↓
Roll ↓
MultiShotMaster
0.3188
0.5933
0.3160
0.9222
0.4094
EchoShot
0.3690
0.4304
0.4241
0.8963
0.4079
Ours
0.1497
0.2641
0.1909
0.9641
0.2015
Table 2: Quantitative Comparison of Orientation Estimation Results
Method
Mean Diff (rad)
OS Yaw ↑
mAOE (rad) ↓
Yaw ↓
Pitch ↓
Roll ↓
w.o. 3D Layout
0.4983
0.2470
0.5130
0.8575
0.4194
Base
0.1497
0.2641
0.1909
0.9641
0.2015
Table 3: Ablation Study on 3D Layout Guidance
Method
Inter-shot Cons.
Text Align. ↑
Trans. Control ↑
Narrative Coherence ↑
Semantic ↑
Sub. ↑
Bg. ↑
w.o. Sub
0.6098
0.5468
0.5239
0.2001
0.9858
0.8452
w.o. Mask
0.3560
0.7781
0.6014
0.1530
0.9461
0.8304
Base
0.6180
0.5905
0.6166
0.1969
0.9771
0.8631
Table 4: Ablation Study on Hierarchical Context Attention
Figure 5: Qualitative comparison. We show frames from each shot generated by all compared methods. Our method not only strictly follows the prompt and maintains strong visual coherence, but also produces natural transitions between shots. In addition, our method closely follows the guidance provided by the 3D layout, especially in Shots 3 and 5, where it better realizes the subject’s layout and motion.
Figure 6: Qualitative demonstration of controlling the subject dynamics using our orientation maps.
Figure 7: MeV enables versatile control over the generated multi-shot videos. (Row 1 → 2) We change the textual descriptions while keeping the layout unchanged. Note we change from Subject B with “white knitted beanie” to Subject D “red coat”. (Row 2 → 3) We keep the text unchanged and swap the positions of Subject C and Subject D. (Row 3 → 4) We delete Subject A. (Row 4 → 5) We let Subject A turn to the right in another shot. (Note that different rows are generated separately. Even though different rows use the same identity text description, the identity consistency is not ensured.)
Figure 8: More qualitative results of MeV on multi-shot video generation.
Multi-shot video generation creates sequences of distinct shots that follow a shared story while preserving subjects, settings, and temporal continuity.