DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning

Yujie Wei; Xinyu Liu; Shiwei Zhang; Hangjie Yuan; Jinbo Xing; Zhekai Chen; Xiang Wang; Haonan Qiu

cs.CVMar 12, 2026

DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning

Authors: Yujie Wei, Xinyu Liu, Shiwei Zhang, Hangjie Yuan, Jinbo Xing, Zhekai Chen, Xiang Wang, Haonan Qiu,

Organizations: Fudan University · The Hong Kong University of Science and Technology · Tongyi Lab, Alibaba Group · Zhejiang University ·

Abstract

While large-scale diffusion models have revolutionized video synthesis, achieving precise control over both multi-subject identity and multi-granularity motion remains a significant challenge. Recent attempts to bridge this gap often suffer from limited motion granularity, control ambiguity, and identity degradation, leading to suboptimal performance on identity preservation and motion control. In this work, we present DreamVideo-Omni, a unified framework enabling harmonious multi-subject customization with omni-motion control via a progressive two-stage training paradigm. In the first stage, we integrate comprehensive control signals for joint training, encompassing subject appearances, global motion, local dynamics, and camera movements. To ensure robust and precise controllability, we introduce a condition-aware 3D rotary positional embedding to coordinate heterogeneous inputs and a hierarchical motion injection strategy to enhance global motion guidance. Furthermore, to resolve multi-subject ambiguity, we introduce group and role embeddings to explicitly anchor motion signals to specific identities, effectively disentangling complex scenes into independent controllable instances. In the second stage, to mitigate identity degradation, we design a latent identity reward feedback learning paradigm by training a latent identity reward model upon a pretrained video diffusion backbone. This provides motion-aware identity rewards in the latent space, prioritizing identity preservation aligned with human preferences. Supported by our curated large-scale dataset and the comprehensive DreamOmni Bench for multi-subject and omni-motion control evaluation, DreamVideo-Omni demonstrates superior performance in generating high-quality videos with precise controllability.

Figures & tables

Fig. 1: Zero-shot multi-subject customization and omni-motion control achieved by DreamVideo-Omni. Our method enables seamless multi-subject customization, precise motion and camera control, and simultaneous single/multi-subject customization with omni-motion control.

Fig. 2: Overview of DreamVideo-Omni. In Stage 1, the framework introduces an all-in-one video DiT that incorporates reference images, bboxes, and trajectories for multi-subject customization and omni-motion control. Stage 2 further enhances identity fidelity via the proposed latent identity reward feedback learning mechanism, which utilizes a latent identity reward model to directly evaluate intermediate latents, completely bypassing the expensive VAE decoder for faster training.

Fig. 3: Pipeline of dataset construction .

	No. of Videos	Reference Images	Multi- Subject	All-Frame Mask	All-Frame Box	All-Frame Trajectory
WebVid-10M [ 97 ]	$\sim$ 10M	✗	✗	✗	✗	✗
UCF-101 [ 98 ]	13,320	✗	✗	✗	✗	✗
DAVIS [ 99 ]	50	✗	✗	✓	✓	✗
GOT-10k [ 100 ]	9,695	✗	✗	✗	✓	✗
VideoBooth [ 10 ]	48,724	✓	✗	✗	✗	✗
DreamVideo-2 [ 17 ]	230,160	✓	✗	✓	✓	✗

TABLE I: Comparison with datasets for video customization and controllable generation. Our dataset uniquely supports multi-subject customization with comprehensive motion control annotations.

Fig. 4: Visualization of a test sample from DreamOmni Bench. Our benchmark supports fine-grained evaluation through comprehensive annotations, including multiple reference images for each subject, detailed captions, and precise spatial-temporal ground truths such as bounding boxes, motion trajectories, and subject masks.

Fig. 5: Qualitative comparison of joint subject customization and motion control. Previous methods struggle to balance identity preservation with accurate motion control. In contrast, our method delivers high-fidelity subject customization that strictly follows complex spatial trajectories.

Method	R-CLIP $\uparrow$	R-DINO $\uparrow$	Face-S $\uparrow$	mIoU $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$
DreamVideo-2 [ 17 ]	0.731	0.429	0.157	0.212	24.05	0.297
DreamVideo-Omni (Ours)	0.739	0.499	0.301	0.558	9.31	0.308

TABLE II: Quantitative comparison on DreamOmni Bench.

Method	Subject Mode			Face Mode
Method	CLIP-T $\uparrow$	R-DINO $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$	Face-S $\uparrow$	EPE $\downarrow$
Tora + Flux.1 [ 79 ]	0.254	0.587	19.72	0.265	0.363	17.41
Video Alchemist [ 49 ]	0.268	0.626	-	0.272	0.411	-
Tora2 [ 20 ]	0.273	0.615	17.43	0.274	0.419	13.52
DreamVideo-Omni (Ours)	0.273	0.628	11.21	0.273	0.417	8.50

TABLE III: Quantitative comparison on MSRVTT-Personalization Bench. We follow the experimental settings and evaluation protocols of Tora2 [ 20 ] and Video Alchemist [ 49 ] , reporting results from their original papers.

Fig. 6: Qualitative comparison of subject customization . DreamVideo-Omni generates videos with accurate subject appearance and enhanced motion dynamics, aligning with provided prompts.

Method	R-CLIP $\uparrow$	R-DINO $\uparrow$	Face-S $\uparrow$	CLIP-T $\uparrow$
Single-Subject Mode
VACE [ 103 ]	0.732	0.480	0.174	0.293
Phantom [ 12 ]	0.738	0.485	0.299	0.296
DreamVideo-Omni	0.739	0.499	0.301	0.308
Multi-Subject Mode
VACE [ 103 ]	0.719	0.497	0.275	0.293

TABLE IV: Quantitative comparison of subject customization on DreamOmni Bench. We report results separately for single-subject and multi-subject scenarios. All models have 1.3B parameters.

Method	mIoU $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$
Single-Subject Mode
Tora [ 79 ] (T2V, 1.1B)	0.163	31.74	0.307
Wan-Move [ 16 ] (I2V, 14B)	0.507	14.43	0.305
DreamVideo-Omni (T2V, 1.3B)	0.558	9.31	0.308
Multi-Subject Mode
Tora [ 79 ] (T2V, 1.1B)	0.162	32.84	0.306

TABLE V: Quantitative comparison of motion control capability on DreamOmni Bench. We report results separately for single-subject and multi-subject scenarios.

Fig. 7: Qualitative comparison of motion control . Our DreamVideo-Omni achieves precise motion trajectory control.

Setting	Method	Subject Fidelity	Motion Consistency	Text Alignment	Overall Quality
Joint ID & Motion	DreamVideo-2 [ 17 ]	22.4%	18.3%	21.5%	10.8%
Joint ID & Motion	Ours	77.6%	81.7%	78.5%	89.2%
Pure Subject Customization	VACE [ 103 ]	16.3%	-	15.6%	19.5%
	Phantom [ 12 ]	19.5%	-	16.8%	20.2%
	Ours	64.2%	-	67.6%	60.3%
Pure Motion Control	Tora [ 79 ]	-	9.5%	16.5%	13.4%

TABLE VI: Human Evaluation. We report the percentage of user votes for each method across different settings.

Fig. 8: Emergent generative capabilities of DreamVideo-Omni. Despite being built on a text-to-video (T2V) base model, our framework naturally enables zero-shot Image-to-Video (I2V) generation and first-frame-conditioned trajectory control without task-specific fine-tuning.

Single-Subject Mode
Method	R-CLIP $\uparrow$	R-DINO $\uparrow$	Face-S $\uparrow$	mIoU $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$
w/o Cond-Aware 3D RoPE	0.625	0.139	0.039	0.274	30.22	0.216
w/o Group & Role Emb.	0.738	0.486	0.254	0.524	26.24	0.309
w/o Hierarchical BBox Injection	0.733	0.508	0.257	0.400	31.84	0.307
Ours (Stage1)	0.733	0.483	0.251	0.556	10.53	0.306
w/o LIReFL (Stage2 SFT only)	0.735	0.487	0.266	0.561	10.01	0.307

TABLE VII: Quantitative ablation studies on each component in DreamVideo-Omni. We report results separately for single-subject and multi-subject scenarios to analyze the impact of each module.

Fig. 9: Ablation study of each component in DreamVideo-Omni .

Optimization Objective
Method	[0, 0.2]	(0.2, 0.4]	(0.4, 0.6]	(0.6, 0.8]	(0.8, 1.0]	Avg
Ours (Default)	0.702	0.722	0.709	0.724	0.743	0.720
w/ BT Loss [ 105 ]	0.491	0.657	0.681	0.706	0.743	0.656
Image Injection Strategy
w/ Ref. Image as KV	0.451	0.555	0.415	0.445	0.408	0.455
Parameter Tuning Scope

TABLE VIII: Ablation studies on the latent identity reward model. The columns denote five intervals of normalized diffusion timesteps

t\in[0,1]

. The first row represents our default setting (BCE Loss, Ref. Image as Q, Frozen Text & Patch Embed.), while subsequent rows illustrate the impact of altering specific components.

Range of $t_{m}$	R-CLIP $\uparrow$	R-DINO $\uparrow$	Face-S $\uparrow$	mIoU $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$
Single-Subject Mode
Last 3 timesteps	0.737	0.494	0.293	0.543	9.98	0.307
All timesteps (Ours)	0.739	0.499	0.301	0.558	9.31	0.308
Multi-Subject Mode
Last 3 timesteps	0.717	0.518	0.324	0.573	6.30	0.307
All timesteps (Ours)	0.720	0.524	0.329	0.570	6.08	0.306

TABLE IX: Ablation study on the range of timestep

t_{m}

in LIReFL.

$\lambda_{2}$	R-CLIP $\uparrow$	R-DINO $\uparrow$	Face-S $\uparrow$	mIoU $\uparrow$	EPE $\downarrow$	CLIP-T $\uparrow$
Single-Subject Mode
0.01	0.737	0.505	0.279	0.560	9.85	0.307
0.10 (Ours)	0.739	0.499	0.301	0.558	9.31	0.308
0.25	0.735	0.492	0.272	0.555	9.65	0.307
0.50	0.718	0.482	0.223	0.541	9.75	0.306
1.00	0.674	0.350	0.120	0.350	25.00	0.280

TABLE X: Ablation study on loss weight

\lambda_{2}

of LIReFL.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion
Mar 16, 2026Zhenghong Zhou, Xiaohang Zhan, Zhiqin Chen +83D-Aware Video Generation Video Generation
Adobe Research · University of Rochester
Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation
Sep 10, 2026Niange Yu, Ye Tian, Biaolong Chen +5Diffusion Transformer Identity-Preserving Video Generation
Alibaba Group · Peking University
MDN-Control: Mask-Depth-Noise Guided Region Control for Multi-Subject Video Editing
Sep 15, 2026Jiayi Yu, Xi Ye, Lina Wang +1Diffusion Model Guidance Video Diffusion Models
School of Cyber Science and Engineering, Wuhan University

Mar 16, 2026J/K move · Enter open · S save

Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion

Zhenghong Zhou, Xiaohang Zhan, Zhiqin Chen, Soo Ye Kim+7

Adobe Research · University of Rochester

3D-Aware Video Generation Video Generation Controllable Video Generation Identity-Preserving Video Generation Multi-View Video Generation

Open paper AskPDF arXiv

Recent video diffusion models achieve strong visual quality and temporal coherence, but still lack coordinated control over scene layout, customized subject identity, and camera/subject motion. We study controllable video generation from three prompts: a first-frame scene image, 3D-aware multi-view subject references, and a motion-driving signal. This setting is challenging because background regions are often trackable under camera motion, while foreground subjects can rotate, self-occlude, and reveal new regions that require identity-consistent appearance. We introduce Tri-Prompting, a video diffusion framework that combines multi-view subject conditioning with dual-conditioned motion control. Tri-Prompting uses XYZ tracking points for visible background motion and low-resolution RGB proxies for foreground subject pose, allowing the model to recover fine appearance from multi-view references while retaining flexibility for plausible subject-scene interactions. An inference-time ControlNet scale schedule further balances motion controllability and visual realism. We evaluate Tri-Prompting against specialized baselines: DaS for motion reconstruction and Phantom for subject-driven generation. Tri-Prompting achieves competitive or improved reconstruction quality, stronger multi-view identity preservation, and better 3D consistency, while enabling 3D-aware subject insertion and in-scene manipulation.

Abstract

Controllable Video Generation

Explore similar work

Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

MDN-Control: Mask-Depth-Noise Guided Region Control for Multi-Subject Video Editing