DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data

Wonjoon Jin; Jiyun Won; Janghyeok Han; Qi Dai; Chong Luo; Seung-Hwan Baek; Sunghyun Cho

cs.CVApr 2, 2026

DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data

Authors: Wonjoon Jin, Jiyun Won, Janghyeok Han, Qi Dai, Chong Luo, Seung-Hwan Baek, Sunghyun Cho

Organizations: POSTECH · Microsoft Research Asia

Abstract

Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.

Figures & tables

Figure 2: Overview of our dataset generation pipeline.

Figure 3: Overview of DynaVid. (a) The motion generator first synthesizes motion and then produces video frames conditioned on the generated motion. For camera-controlled video synthesis, Plücker embeddings are provided as additional input. (b) Our framework adopts VACE [ 25 ] to incorporate control signals such as Plücker embeddings or optical flow maps.

Figure 4: Qualitative comparison of dynamic object motion generation. CogVideoX [ 56 ] and Wan2.2-5B [ 48 ] often produce distorted or unrealistic human motions with visual artifacts. HyperMotion [ 52 ] produces unnatural appearances because it relies on the first frame as input. In contrast, our method generates realistic videos with natural and highly dynamic motions.

	Pexels					DynaVid-Human test
	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )
CogVideoX-5B	1519.54	0.5646	0.6613	0.9844	0.9673	2238.68	0.5071	0.5562	0.9779	0.9565
Wan2.2-5B	1172.02	0.5779	0.7235	0.9928	0.9883	1775.99	0.5389	0.6974	0.9904	0.9791
HyperMotion	420.82*	0.5578	0.6972	0.9922	0.9850	391.22*	0.5092	0.6265	0.9939	0.9914
Ours	1126.38	0.5807	0.7342	0.9900	0.9748	1351.94	0.5312	0.7352	0.9931	0.9864

Table 1: Quantitative evaluation of object motion generation on the Pexels [ 37 ] and DynaVid-Human test datasets, representing common and highly dynamic scenes, respectively. Our method achieves comparable or superior results on both datasets compared to other baseline methods. *: HyperMotion [ 52 ] uses a first frame as input, resulting in lower FVD scores due to the appearance similarity.

Figure 5: Qualitative comparison of camera-controlled video synthesis. Red arrows indicate the directions of camera motion. AC3D [ 1 ] fails to follow the extreme 180 ∘ rotation, while GEN3C [ 40 ] produces noticeable artifacts in regions unseen from the initial view (zoomed-in red box). In contrast, our method produces natural-looking videos that faithfully follow the input camera trajectory. For fair comparison, the same camera parameters are used for all methods.

	RE10K test						DynaVid-Camera test
	mRotErr ( $\downarrow$ )	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )	mRotErr ( $\downarrow$ )	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )
AC3D	0.1347	685.05	0.5184	0.6636	0.9918	0.9651	1.1529	782.01	0.4483	0.5407	0.9727	0.9403
GEN3C	0.0809	566.99*	0.4579	0.6079	0.9899	0.9731	1.1852	237.15*	0.3889	0.5659	0.9844	0.9611
Ours	0.1136	664.21	0.5425	0.7224	0.9888	0.9699	0.9289	674.72	0.4501	0.6713	0.9760	0.9487

Table 2: Quantitative evaluation of camera-controlled video synthesis on RE10K [ 61 ] and DynaVid-Camera, representing common and dynamic scenes, respectively. While achieving comparable rotation errors on common scenes, our method significantly outperforms other baseline methods on scenes with rapidly changing viewpoints. *: GEN3C [ 40 ] uses a first frame as input, resulting in lower FVD scores.

	Pexels					DynaVid-Human test
	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )	FVD ( $\downarrow$ )	A-Qual ( $\uparrow$ )	I-Qual ( $\uparrow$ )	M-Smooth ( $\uparrow$ )	T-Flick ( $\uparrow$ )
Ours	1126.38	0.5807	0.7342	0.9900	0.9748	1351.94	0.5312	0.7352	0.9931	0.9864
w/o synthetic motion data (a)	(-49.85)	(-0.0199)	(-0.0156)	(-0.0002)	(-0.0019)	(+527.04)	(+0.0106)	(-0.0403)	(-0.0062)	(-0.0249)
w/o batch mixture (b)	(+759.36)	(-0.0073)	(-0.0070)	(+0.0030)	(+0.0144)	(-122.24)	(-0.0050)	(-0.0006)	(-0.0022)	(-0.0013)
w/o dataset filtering † (c)	(-33.79)	(+0.0011)	(+0.0035)	(+0.0007)	(+0.0009)	(+32.44)	(+0.0055)	(-0.0140)	(+0.0008)	(+0.0012)
w/ synthetic video data (d)	(+104.43)	(+0.0163)	(+0.0124)	(-0.0012)	(+0.0005)	(-653.98)*	(+0.0112)	(-0.0487)	(-0.0000)	(+0.0049)

Table 3: Ablation study of our main components, evaluating video and motion synthesis quality on the Pexels and DynaVid-Human test datasets.

\dagger

: Our model without data filtering achieves comparable quantitative results but produces noticeably degraded visual quality in highly dynamic motion synthesis, as shown in Fig. 6 . *: Our model trained with rendered synthetic videos reports lower FVD scores, because it produces artificial appearances resembling synthetic data, as shown in Fig. 6 .

Figure 6: Visual examples of the ablation study. Our model without dataset filtering produces video frames that are inconsistent with the input motion (top). Our model trained using the synthetic video dataset reproduces the artificial look of rendered videos (bottom).

	Clean	25dB	20dB	15dB	10dB	5dB
M-Err ( $\downarrow$ )	4.287	4.371	4.374	4.323	4.391	4.434
A-Qual ( $\uparrow$ )	0.5312	0.5233	0.5239	0.5148	0.5071	0.4632

Table 4: Error robustness of the motion-guided video generator.

Figure 7: Capability to synthesize non-human dynamic motions.

Figure 8: Limitations. Our method is less effective in generating videos with multiple people performing dynamic motions.

Explore similar work

VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos
Oct 28, 2025Qiucheng Wu, Handong Zhao, Zhixin Shu +3Synthetic Data Augmentation Camera-Controlled Video Generation
UC, Santa Barbara · Adobe Research · MIT-IBM Watson AI Lab
Exploring the Role of Synthetic Data Augmentation in Controllable Human-Centric Video Generation
Apr 23, 2026Yuanchen Fei, Yude Zou, Zejian Kang +3Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Hunan University · Shanghai Jiaotong University · Westlake University +3
Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation
Jan 15, 2026Dong-Yu Chen, Yixin Guo, Shuojin Yang +2Camera-Controlled Video Generation Video Diffusion Models
BNRist, Department of Computer Science and Technology, Tsinghua University Beijing 100084, China

Oct 28, 2025J/K move · Enter open · S save

VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos

Qiucheng Wu, Handong Zhao, Zhixin Shu, Jing Shi+2

UC, Santa Barbara · Adobe Research · MIT-IBM Watson AI Lab

Synthetic Data Augmentation Camera-Controlled Video Generation Video Diffusion Models

Open paper AskPDF arXiv

Although recent video generative models are getting more capable of following external camera controls, imposed by either text descriptions or camera trajectories, they still struggle to generalize to unconventional camera motions, which is crucial in creating truly original and artistic videos. The challenge lies in finding sufficient training videos with the intended uncommon camera motions. To this end, we propose VividCam, a training paradigm that enables diffusion models to learn complex camera motions from synthetic videos, releasing the reliance on collecting realistic training videos. VividCam incorporates multiple disentanglement strategies that isolate camera motion learning from synthetic appearance artifacts, ensuring more robust motion representation and mitigating domain shift. We show that our design synthesizes a wide range of precisely controlled camera motions using surprisingly simple synthetic data. Notably, this synthetic data often consists of basic geometries within a low-poly 3D scene and can be efficiently rendered by engines like Unity. Our video results can be found in https://wuqiuche.github.io/VividCamDemoPage/ .

Abstract

Explore similar work

VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos

Exploring the Role of Synthetic Data Augmentation in Controllable Human-Centric Video Generation

Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Generation