Text-driven video generation has democratized film creation, but camera control in cinematic multi-shot scenarios remains a significant block. Implicit textual prompts lack precision, while explicit trajectory conditioning imposes prohibitive manual overhead and often triggers execution failures in current models. To overcome this bottleneck, we propose a data-centric paradigm shift, positing that aligned (Caption, Trajectory, Video) triplets form an inherent joint distribution that can connect automated plotting and precise execution. Guided by this insight, we present ShotVerse, a ``Plan-then-Control'' framework that decouples generation into two collaborative agents: a VLM (Vision-Language Model)-based Planner that leverages spatial priors to obtain cinematic, globally aligned trajectories from text, and a Controller that renders these trajectories into multi-shot video content via a camera adapter. Central to our approach is the construction of a data foundation: we design an automated multi-shot camera calibration pipeline aligns disjoint single-shot trajectories into a unified global coordinate system. This facilitates the curation of ShotVerse-Bench, a high-fidelity cinematic dataset with a three-track evaluation protocol that serves as the bedrock for our framework. Extensive experiments demonstrate that ShotVerse effectively bridges the gap between unreliable textual control and labor-intensive manual plotting, achieving superior cinematic aesthetics and generating multi-shot videos that are both camera-accurate and cross-shot consistent.
Figures & tables
Figure 1. Cinematic, Camera-Controlled, Multi-Shot Video Creation via our ShotVerse Framework. Examples demonstrate high-fidelity and great camera-controlled generation across diverse genres. The inset 3D plots visualize the plotted explicit trajectories.
Figure 2. Method Overview. (i) Dataset Curation . We construct the ShotVerse-Bench by aligning multi-shot trajectories into a unified global coordinate system via camera calibration, paired with hierarchical global and per-shot captions. (ii) Trajectory Plotting : The Planner utilizes a VLM to process the hierarchical prompt interleaved with learnable trajectory query tokens. These inputs are encoded into context-aware embeddings and transformed into explicit camera poses via a Trajectory Decoder and a Pose De-Tokenizer. (iii) Trajectory Injection : The Controller synthesizes high-fidelity videos using a holistic DiT backbone. It precisely follows the trajectories via a Camera Adapter and a 4D Rotary Positional Embedding strategy.
Dataset
Caption Annotation
Traj Type
Domain
Statistics
Traj
Scene
Intent
#Vocab
#Sample
#Frame
MVImgNet ( 60 )
×
×
×
–
Object/Scene-Centric
Captured
22K
6.5M
RealEstate10k ( 68 )
×
×
×
–
Object/Scene-Centric
Youtube
79K
11M
DL3DV-10K ( 37 )
×
×
×
–
Object/Scene-Centric
Captured
10K
51M
CCD ( 29 )
✓
×
×
48
Tracking
Synthetic
25K
4.5M
E.T. ( 10 )
✓
×
×
1790
Tracking
Film
115K
11M
Table 1. Comparisons of Camera Trajectory Datasets. ShotVerse-Bench is the first large-scale dataset that provides multi-shot cinematic camera trajectories together with rich, multi-level caption annotations.
Method
Dataset
DataDoP ( 62 )
ShotVerse-Bench
F1-Score ↑
CLaTr-CLIP ↑
F1-Score ↑
CLaTr-CLIP ↑
CCD ( 29 )
Pre-Trained
0.315
4.247
0.323
7.032
E.T. ( 10 )
Pre-Trained
0.319
0.000
0.289
2.462
Director3D ( 35 )
Pre-Trained
0.126
0.000
0.162
1.237
GenDoP ( 62 )
Pre-Trained
0.399
32.408
0.326
23.089
GenDoP ( 62 )
ShotVerse-Bench
0.268
24.132
0.343
33.875
Table 2. Track A: Quantitative Results of Text-Trajectory Alignment.
Method
Trans. Error ↓
Rotation Error ↓
CAS ↑
MotionCtrl ( 54 )
0.0900
2.56
0.329
CameraCtrl ( 22 )
0.0571
1.28
0.343
ReCamMaster ( 3 )
0.0589
1.12
0.408
ShotVerse (Ours)
0.0163
0.73
0.500
Table 3. Track B: Quantitative Evaluation of Camera Control. All methods receive ground-truth trajectories.
Method
Sem. Consist. (Global) ↑
Sem. Consist. (Shot) ↑
Aesthetic Quality ↑
Shot Trans. Accuracy ↑
FVD ↓
HoloCine ( 42 )
0.297
0.254
4.981
0.645
407.54
MultiShotMaster ( 51 )
0.279
0.247
5.210
0.927
440.78
Sora2 ( 43 )
0.297
-
5.344
-
372.13
VEO3 ( 19 )
0.282
-
5.441
-
941.50
Kling3.0 ( 31 )
0.288
-
5.167
-
719.44
Seedance2.0 ( 17 )
0.285
-
5.381
-
605.17
Table 4. Track C: Quantitative Evaluation of Multi-Shot Quality. Without shot-splitting, shot metrics cannot be calculated for some baselines.
Method
Motion Type Appropriateness ↑
Motion Duration Appropriateness ↑
Subject Emphasis & Saliency ↑
Cinematic Pacing ↑
HoloCine ( 42 )
4.324
4.281
3.997
3.208
VEO3 ( 19 )
4.402
4.189
4.252
3.288
VLM-Based
Sora2 ( 43 )
4.371
4.258
3.892
3.236
Kling3.0 ( 31 )
4.302
4.153
3.872
3.108
Seedance2.0 ( 17 )
4.402
4.279
4.328
3.279
ShotVerse (Ours)
4.447
4.304
4.426
3.384
Table 5. Track C: Quantitative Evaluation of Cinematic Quality.
w/o VLM encoder
w/ VLM decoder
w/o Query Tokens
Ours
F1-Score ↑
0.343
0.248
0.251
0.422
CLaTr-CLIP ↑
33.875
15.078
18.796
35.016
Table 6. Quantitative Evaluation of Ablation Study (Planner).
Method
Trans. Error ↓
Rotation Error ↓
Shot Trans. Acc. ↑
Sem. Consist. (Global) ↑
Sem. Consist. (Shot) ↑
Aesthetic Quality ↑
w/o Cam. Enc. (HoloCine)
0.0609
1.27
0.645
0.297
0.254
4.981
w/ Low&High Noise Enc.
0.0189
0.74
0.930
0.296
0.250
5.321
w/ 3D RoPE
0.0323
1.04
0.429
0.290
0.251
5.413
w/ Synthetic Data
0.0509
1.35
0.705
0.292
0.253
4.833
w/o Camera Calibration
0.0165
0.79
0.931
0.296
0.251
5.136
ShotVerse (Ours)
0.0163
0.73
0.933
0.299
0.255
5.465
Table 7. Quantitative Evaluation of Ablation Study (Controller).
Figure 3. Comparisons with the State-of-the-Art Baseline Methods. Early camera-controlled text-driven generation models ( e.g. , CameraCtrl, MotionCtrl) struggle to handle complex cinematic camera trajectories. ReCamMaster executes the trajectory but drifts away from the subject in Shot 1. HoloCine, MultiShotMaster, Sora2, VEO3, and Kling3.0, and Seedance2.0 fail to execute the complex “orbit” command, remaining nearly static. These failures demonstrate that for text-driven models, scaling up caption density is insufficient to achieve precise control without explicit geometric guidance.
Recent advances in text-to-video generation have enabled high-fidelity single-shot synthesis, yet practical storytelling creation requires multi-shot videos with both strong inter-shot consistency and fine-grained controllability. Existing multi-shot video generation approaches achieve strong consistency but still fall short in controllability. In this paper, we propose Mise-en-Video (MeV), a highly controllable multi-shot video generation framework that allows video creators to conduct staging the multi-shot in the 3D space and then generate the corresponding multi-shot video. To achieve this, our framework combines a hierarchical attention mechanism for multi-level narrative consistency with 3D-aware layout guidance for precise, per-shot spatial control. To support training and evaluation, we develop standardized annotation pipelines and construct a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes. Extensive experiments demonstrate that MeV consistently outperforms strong baselines in video quality, inter-shot consistency, and subject-level controllability.
Yingshuang Zou, Xiaoyu Shi, Yawen Luo +8
The Hong Kong University of Science and Technology · KlingAI Research · The Chinese University of Hong Kong +2
Current video generation models achieve impressive results in single-shot generation, yet remain limited in cinematic video generation, where coherent narratives and effective multi-shot composition require explicit shot planning. To address this challenge, we propose ShotPlan, a framework for explicit multi-shot cinematic video generation built upon a video diffusion foundation model. Our method introduces learnable planning tokens that capture shot-level transition cues and can be seamlessly integrated with the original video generation tokens to control transition timestamps. Unlike standard video generation tokens, the proposed planning tokens are equipped with Fractional Temporal Rotary Position Embedding (FRoPE), enabling shot transitions to be modeled at the frame level. Experiments demonstrate that ShotPlan significantly outperforms existing cinematic video generation methods, offering more flexible shot management and stronger inter-shot consistency.
Su Guo, Guangce Liu, Haosen Yang +7
Institute of Artificial Intelligence (TeleAI), China Telecom · Harbin Institute of Technology
Camera motion control is essential for directing viewpoint changes in generative systems. However, existing methods typically condition the generation process on a single specific modality, such as explicit pose trajectories or reference videos, limiting their ability to support heterogeneous user inputs. To address this limitation, we present TriMotion, a modality-agnostic framework for camera-controlled video generation that maps video, pose, and text inputs, describing the same camera trajectory into a shared motion embedding space. Learning such a space requires synchronized supervision across modalities. Therefore, we build the Motion Triplet Dataset by extending a Multi-Cam Video Dataset with geometry-grounded motion descriptions derived from camera extrinsics. We further introduce a latent motion consistency objective that leverages the motion embedding space to encourage the generated video to follow the target camera trajectory directly in latent space, avoiding the cost of pixel-space decoding. Extensive experiments show that TriMotion generates high-quality videos that accurately follow the target camera trajectories across all three modalities. Beyond standard generation, the shared motion embedding space also enables flexible applications such as sequential motion composition and cross-modal motion interpolation.
Seunghyun Shin, Jifei Song, Wooseok Jeon +2
GIST · Huawei Noah’s Ark Lab · Yonsei University +1