Organizations: Fudan University · The Hong Kong University of Science and Technology · Tongyi Lab, Alibaba Group · Zhejiang University · MMLab, The University of Hong Kong · Nanyang Technological University · Show Lab, National University of Singapore
While large-scale diffusion models have revolutionized video synthesis, achieving precise control over both multi-subject identity and multi-granularity motion remains a significant challenge. Recent attempts to bridge this gap often suffer from limited motion granularity, control ambiguity, and identity degradation, leading to suboptimal performance on identity preservation and motion control. In this work, we present DreamVideo-Omni, a unified framework enabling harmonious multi-subject customization with omni-motion control via a progressive two-stage training paradigm. In the first stage, we integrate comprehensive control signals for joint training, encompassing subject appearances, global motion, local dynamics, and camera movements. To ensure robust and precise controllability, we introduce a condition-aware 3D rotary positional embedding to coordinate heterogeneous inputs and a hierarchical motion injection strategy to enhance global motion guidance. Furthermore, to resolve multi-subject ambiguity, we introduce group and role embeddings to explicitly anchor motion signals to specific identities, effectively disentangling complex scenes into independent controllable instances. In the second stage, to mitigate identity degradation, we design a latent identity reward feedback learning paradigm by training a latent identity reward model upon a pretrained video diffusion backbone. This provides motion-aware identity rewards in the latent space, prioritizing identity preservation aligned with human preferences. Supported by our curated large-scale dataset and the comprehensive DreamOmni Bench for multi-subject and omni-motion control evaluation, DreamVideo-Omni demonstrates superior performance in generating high-quality videos with precise controllability.
Figures & tables
Fig. 1: Zero-shot multi-subject customization and omni-motion control achieved by DreamVideo-Omni. Our method enables seamless multi-subject customization, precise motion and camera control, and simultaneous single/multi-subject customization with omni-motion control.
Fig. 2: Overview of DreamVideo-Omni. In Stage 1, the framework introduces an all-in-one video DiT that incorporates reference images, bboxes, and trajectories for multi-subject customization and omni-motion control. Stage 2 further enhances identity fidelity via the proposed latent identity reward feedback learning mechanism, which utilizes a latent identity reward model to directly evaluate intermediate latents, completely bypassing the expensive VAE decoder for faster training.
Fig. 3: Pipeline of dataset construction .
No. of Videos
Reference Images
Multi- Subject
All-Frame Mask
All-Frame Box
All-Frame Trajectory
WebVid-10M [ 97 ]
∼ 10M
✗
✗
✗
✗
✗
UCF-101 [ 98 ]
13,320
✗
✗
✗
✗
✗
DAVIS [ 99 ]
50
✗
✗
✓
✓
✗
GOT-10k [ 100 ]
9,695
✗
✗
✗
✓
✗
VideoBooth [ 10 ]
48,724
✓
✗
✗
✗
✗
DreamVideo-2 [ 17 ]
230,160
✓
✗
✓
✓
✗
TABLE I: Comparison with datasets for video customization and controllable generation. Our dataset uniquely supports multi-subject customization with comprehensive motion control annotations.
Fig. 4: Visualization of a test sample from DreamOmni Bench. Our benchmark supports fine-grained evaluation through comprehensive annotations, including multiple reference images for each subject, detailed captions, and precise spatial-temporal ground truths such as bounding boxes, motion trajectories, and subject masks.
Fig. 5: Qualitative comparison of joint subject customization and motion control. Previous methods struggle to balance identity preservation with accurate motion control. In contrast, our method delivers high-fidelity subject customization that strictly follows complex spatial trajectories.
Method
R-CLIP ↑
R-DINO ↑
Face-S ↑
mIoU ↑
EPE ↓
CLIP-T ↑
DreamVideo-2 [ 17 ]
0.731
0.429
0.157
0.212
24.05
0.297
DreamVideo-Omni (Ours)
0.739
0.499
0.301
0.558
9.31
0.308
TABLE II: Quantitative comparison on DreamOmni Bench.
Method
Subject Mode
Face Mode
CLIP-T ↑
R-DINO ↑
EPE ↓
CLIP-T ↑
Face-S ↑
EPE ↓
Tora + Flux.1 [ 79 ]
0.254
0.587
19.72
0.265
0.363
17.41
Video Alchemist [ 49 ]
0.268
0.626
-
0.272
0.411
-
Tora2 [ 20 ]
0.273
0.615
17.43
0.274
0.419
13.52
DreamVideo-Omni (Ours)
0.273
0.628
11.21
0.273
0.417
8.50
TABLE III: Quantitative comparison on MSRVTT-Personalization Bench. We follow the experimental settings and evaluation protocols of Tora2 [ 20 ] and Video Alchemist [ 49 ] , reporting results from their original papers.
Fig. 6: Qualitative comparison of subject customization . DreamVideo-Omni generates videos with accurate subject appearance and enhanced motion dynamics, aligning with provided prompts.
Method
R-CLIP ↑
R-DINO ↑
Face-S ↑
CLIP-T ↑
Single-Subject Mode
VACE [ 103 ]
0.732
0.480
0.174
0.293
Phantom [ 12 ]
0.738
0.485
0.299
0.296
DreamVideo-Omni
0.739
0.499
0.301
0.308
Multi-Subject Mode
VACE [ 103 ]
0.719
0.497
0.275
0.293
TABLE IV: Quantitative comparison of subject customization on DreamOmni Bench. We report results separately for single-subject and multi-subject scenarios. All models have 1.3B parameters.
Method
mIoU ↑
EPE ↓
CLIP-T ↑
Single-Subject Mode
Tora [ 79 ] (T2V, 1.1B)
0.163
31.74
0.307
Wan-Move [ 16 ] (I2V, 14B)
0.507
14.43
0.305
DreamVideo-Omni (T2V, 1.3B)
0.558
9.31
0.308
Multi-Subject Mode
Tora [ 79 ] (T2V, 1.1B)
0.162
32.84
0.306
TABLE V: Quantitative comparison of motion control capability on DreamOmni Bench. We report results separately for single-subject and multi-subject scenarios.
Fig. 7: Qualitative comparison of motion control . Our DreamVideo-Omni achieves precise motion trajectory control.
Setting
Method
Subject Fidelity
Motion Consistency
Text Alignment
Overall Quality
Joint ID & Motion
DreamVideo-2 [ 17 ]
22.4%
18.3%
21.5%
10.8%
Ours
77.6%
81.7%
78.5%
89.2%
Pure Subject Customization
VACE [ 103 ]
16.3%
-
15.6%
19.5%
Phantom [ 12 ]
19.5%
-
16.8%
20.2%
Ours
64.2%
-
67.6%
60.3%
Pure Motion Control
Tora [ 79 ]
-
9.5%
16.5%
13.4%
TABLE VI: Human Evaluation. We report the percentage of user votes for each method across different settings.
Fig. 8: Emergent generative capabilities of DreamVideo-Omni. Despite being built on a text-to-video (T2V) base model, our framework naturally enables zero-shot Image-to-Video (I2V) generation and first-frame-conditioned trajectory control without task-specific fine-tuning.
Method
R-CLIP ↑
R-DINO ↑
Face-S ↑
mIoU ↑
EPE ↓
CLIP-T ↑
Single-Subject Mode
w/o Cond-Aware 3D RoPE
0.625
0.139
0.039
0.274
30.22
0.216
w/o Group & Role Emb.
0.738
0.486
0.254
0.524
26.24
0.309
w/o Hierarchical BBox Injection
0.733
0.508
0.257
0.400
31.84
0.307
Ours (Stage1)
0.733
0.483
0.251
0.556
10.53
0.306
w/o LIReFL (Stage2 SFT only)
0.735
0.487
0.266
0.561
10.01
0.307
TABLE VII: Quantitative ablation studies on each component in DreamVideo-Omni. We report results separately for single-subject and multi-subject scenarios to analyze the impact of each module.
Fig. 9: Ablation study of each component in DreamVideo-Omni .
Method
[0, 0.2]
(0.2, 0.4]
(0.4, 0.6]
(0.6, 0.8]
(0.8, 1.0]
Avg
Ours (Default)
0.702
0.722
0.709
0.724
0.743
0.720
Optimization Objective
w/ BT Loss [ 105 ]
0.491
0.657
0.681
0.706
0.743
0.656
Image Injection Strategy
w/ Ref. Image as KV
0.451
0.555
0.415
0.445
0.408
0.455
Parameter Tuning Scope
TABLE VIII: Ablation studies on the latent identity reward model. The columns denote five intervals of normalized diffusion timesteps t∈[0,1] . The first row represents our default setting (BCE Loss, Ref. Image as Q, Frozen Text & Patch Embed.), while subsequent rows illustrate the impact of altering specific components.
Range of tm
R-CLIP ↑
R-DINO ↑
Face-S ↑
mIoU ↑
EPE ↓
CLIP-T ↑
Single-Subject Mode
Last 3 timesteps
0.737
0.494
0.293
0.543
9.98
0.307
All timesteps (Ours)
0.739
0.499
0.301
0.558
9.31
0.308
Multi-Subject Mode
Last 3 timesteps
0.717
0.518
0.324
0.573
6.30
0.307
All timesteps (Ours)
0.720
0.524
0.329
0.570
6.08
0.306
TABLE IX: Ablation study on the range of timestep tm in LIReFL.
λ2
R-CLIP ↑
R-DINO ↑
Face-S ↑
mIoU ↑
EPE ↓
CLIP-T ↑
Single-Subject Mode
0.01
0.737
0.505
0.279
0.560
9.85
0.307
0.10 (Ours)
0.739
0.499
0.301
0.558
9.31
0.308
0.25
0.735
0.492
0.272
0.555
9.65
0.307
0.50
0.718
0.482
0.223
0.541
9.75
0.306
1.00
0.674
0.350
0.120
0.350
25.00
0.280
TABLE X: Ablation study on loss weight λ2 of LIReFL.
Recent video diffusion models achieve strong visual quality and temporal coherence, but still lack coordinated control over scene layout, customized subject identity, and camera/subject motion. We study controllable video generation from three prompts: a first-frame scene image, 3D-aware multi-view subject references, and a motion-driving signal. This setting is challenging because background regions are often trackable under camera motion, while foreground subjects can rotate, self-occlude, and reveal new regions that require identity-consistent appearance. We introduce Tri-Prompting, a video diffusion framework that combines multi-view subject conditioning with dual-conditioned motion control. Tri-Prompting uses XYZ tracking points for visible background motion and low-resolution RGB proxies for foreground subject pose, allowing the model to recover fine appearance from multi-view references while retaining flexibility for plausible subject-scene interactions. An inference-time ControlNet scale schedule further balances motion controllability and visual realism. We evaluate Tri-Prompting against specialized baselines: DaS for motion reconstruction and Phantom for subject-driven generation. Tri-Prompting achieves competitive or improved reconstruction quality, stronger multi-view identity preservation, and better 3D consistency, while enabling 3D-aware subject insertion and in-scene manipulation.
Multi-subject video generation faces two key challenges: uncontrollable fidelity strength and potential semantic drift. We address these by analyzing the internal mechanisms of Diffusion Transformers (DiTs). We found that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects. Building on this insight, we propose Dual-phase Intrinsic Attention Leveraging (DIAL), a framework that uses these internal signals for both training and inference. In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining. In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning (RL). This RL procedure effectively anchors the model's attention to reference subjects and mitigates semantic drift. Extensive experiments show that DIAL significantly outperforms baseline models on the OpenS2V-Eval benchmark, consistently improving identity consistency and enabling controllable fidelity strength.
Multi subject video editing modifies designated subjects while preserving non target content, but faces cross subject attribute leakage, and occlusion ambiguity. Existing approaches rely on masks and struggle to distinguish overlapping subjects or ensure consistent generation. To address these limitations, we propose MDN-Control, a training free framework jointly controlling target localization, occlusion geometry, and appearance initialization. Specifically, mask-guided localization provides consistent target localization, while depth-aware occlusion control resolves ambiguous boundaries between overlapping subjects. We further introduce noise latent prompting, which retrieves Gaussian initializations from a noise library for prompt relevant priors. Experiments on MSVBench show that MDN-Control achieves the lowest CM-Err and the highest Q-Edit, while maintaining competitive text alignment and temporal consistency, demonstrating the effectiveness of combining spatial, geometric, and latent priors for multi subject video editing.
Jiayi Yu, Xi Ye, Lina Wang +1
School of Cyber Science and Engineering, Wuhan University