Organizations: Department of Computer Science and Software Engineering, The University of Western Australia, Australia. · Munich Center for Machine Learning (MCML) and Technical University of Munich (TUM), Germany. · School of Information Technology, Murdoch University, Australia. · Department of Electrical, Electronics and Computer Engineering, The University of Western Australia, Australia.
Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.
Figures & tables
Fig. 1: Overview of the text-to-skeleton generation architecture for training. A text prompt is encoded and prepended as a conditioning prefix to the pose token sequence. The autoregressive Transformer predicts each joint token conditioned on all previously generated tokens and the text description.
Fig. 2: Overview of the text-to-skeleton generation architecture for inference.
Fig. 3: Overview of the pose-conditioned video generation architecture. A reference image is encoded via DINO-ALF to produce appearance tokens, while the skeleton sequence is rasterized and encoded by a 3D CNN into spatiotemporally aligned motion tokens. Both conditioning streams are injected into the DiT denoiser to synthesize the output video.
Fig. 4: Patch-feature magnitude maps ( ℓ2 -norm) across DINOv3 layers. Earlier layers exhibit high activation on the subject and texture-rich regions, while later layers show more uniform magnitudes. This motivates using an early layer as the query for adaptive layer fusion.
Fig. 5: Cross-attention maps for CLIP (top) vs. DINO-ALF (bottom) on a backflip sequence. DINO-ALF attends more precisely to the moving subject, while CLIP attention is scattered.
Method
FID ↓
Rp-top1 ↑
Rp-top2 ↑
Rp-top3 ↑
Diversity ↑
MM-Dist ↓
T2M-GPT [ 67 ]
524.61
0.191
0.287
0.473
40.11
49.85
PriorMDM [ 68 ]
585.31
0.216
0.325
0.501
42.58
44.29
MLD [ 69 ]
467.22
0.335
0.503
0.653
41.67
47.66
HumanDreamer [ 45 ]
322.16
0.411
0.598
0.722
45.33
41.53
Ours
255.19
0.487
0.667
0.784
48.33
38.65
TABLE I: Quantitative evaluation for Text-to-Skeleton generation on our 8,000-pair benchmark. ↓ indicates lower is better and ↑ indicates higher is better.
Model
VBench-I2V
Frame-level
Video-level
Subj. Cons ↑
Bg. Cons ↑
Motion Flick. ↑
Motion Smooth. ↑
Dyn.Degree ↑
Aes.Qual ↑
Img.Qual ↑
SSIM ↑
LPIPS ↓
PSNR ↑
FVD ↓
HumanVid [ 38 ]
85.33
86.91
91.01
88.21
72.72
54.61
64.19
0.737
0.213
25.54
637.1
MimicMotion [ 41 ]
86.01
88.47
92.11
89.95
73.05
55.07
65.14
0.744
0.198
26.66
611.5
Hyper-Motion [ 17 ]
86.66
90.23
93.22
90.08
73.65
55.21
65.42
0.767
0.181
27.13
568.8
UniAnimate-DiT [ 37 ]
87.21
91.17
95.78
92.33
74.87
56.32
66.87
0.751
0.189
27.48
527.4
VACE [ 35 ]
88.31
91.11
95.31
94.81
75.03
57.11
67.91
0.771
0.173
27.91
498.7
TABLE II: Comparison of different methods on VBench. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Configuration
FID ↓
Rp-top1 ↑
Diversity ↑
MM-Dist ↓
Tokenization granularity ( K bins)
K=64
312.41
0.401
44.21
43.17
K=128
278.55
0.452
46.88
40.52
K=256 (Baseline)
255.19
0.487
48.33
38.65
K=512
261.33
0.479
47.65
39.21
Decoder depth ( L layers)
TABLE III: Ablation study on the text-to-skeleton architecture. We analyze the effect of tokenization granularity ( K ), decoder depth ( L ), and decoding strategy. ↓ indicates lower is better; ↑ indicates higher is better.
Fig. 6: Qualitative comparison of generated 2D pose sequences for the prompt “A person performs a cartwheel”. Our method produces more coherent and physically plausible trajectories, especially during inverted phases. Failure cases of competing methods are highlighted with red boxes. For better visualization, we crop the scene to show only the human skeleton.
Fig. 7: Qualitative video comparison with pose-guided baselines. Our method best follows the target 2D skeleton while preserving the reference appearance. Competing methods exhibit hand/leg artifacts under complex motions (red boxes), and our DINO-ALF conditioning better retains fine details (e.g., the red tie). More results are on our project page .
Configuration
Subj. ↑
Bg. ↑
M.Smooth ↑
SSIM ↑
LPIPS ↓
FVD ↓
LoRA rank ( r )
r=16
84.22
90.45
92.11
0.712
0.221
512.8
r=32
88.77
91.58
95.88
0.758
0.193
491.5
r=64 (Baseline)
91.31
93.79
97.39
0.795
0.174
471.3
r=128
91.33
93.72
97.28
0.789
0.178
478.9
Motion encoder architecture
TABLE IV: Ablation study on pose-to-video network parameters. We analyze the effect of LoRA rank, motion encoder architecture, and condition dropout rate. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Model
VBench-I2V
Frame-level
Video-level
Subj. Cons ↑
Bg. Cons ↑
Motion Flick. ↑
Motion Smooth. ↑
Dyn.Degree ↑
Aes.Qual ↑
Img.Qual ↑
SSIM ↑
LPIPS ↓
PSNR ↑
FVD ↓
DLL (single-layer)
87.65
90.03
94.11
93.77
74.13
55.11
66.23
0.752
0.181
27.02
502.5
CLIP embeddings
85.44
89.75
93.63
91.21
73.02
54.76
66.08
0.741
0.195
26.98
517.7
No appearance encoder
75.13
85.14
82.22
82.11
65.53
47.11
60.09
0.673
0.245
24.33
593.4
Ours
91.31
93.79
97.50
97.39
78.50
56.37
67.14
0.795
0.174
28.50
471.3
TABLE V: Ablation study on appearance encoding design. We compare DINO-ALF (Ours), DLL (single-layer), CLIP embeddings, and no appearance encoder. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Fig. 8: Ablation study on the pose-to-video stage. Rows from top to bottom: Ref (reference frame and target pose), CLIP (CLIP embeddings), DINO LL (DLL single-layer), W/O cross (no appearance encoder), and Ours. DINO-ALF yields the best appearance preservation under large pose changes, while CLIP-only or no appearance encoder causes hand/clothing artifacts and identity drift (red boxes).
Configuration
Subj. ↑
Bg. ↑
M.Smooth ↑
SSIM ↑
LPIPS ↓
FVD ↓
GT skeletons (upper bound)
91.31
93.79
97.39
0.795
0.174
471.3
No augmentation
85.11
86.57
89.07
0.652
0.229
574.7
Joint jitter only
86.45
87.26
91.39
0.651
0.213
568.1
Joint dropout only
86.57
87.73
91.11
0.683
0.208
541.5
Temporal shift only
87.31
88.05
92.71
0.713
0.193
528.3
Full augmentation (Ours)
89.23
91.17
95.61
0.781
0.183
492.5
TABLE VI: Ablation study on pose augmentation for error robustness. All rows except the first use predicted skeletons at inference; the GT row serves as an upper bound. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Human video generation remains challenging due to the difficulty of jointly modeling human appearance, motion, and camera viewpoint under limited multi-view data. Existing methods often address these factors separately, resulting in limited controllability or reduced visual quality. We revisit this problem from an image-first perspective, where high-quality human appearance is learned via image generation and used as a prior for video synthesis, decoupling appearance modeling from temporal consistency. We propose a pose- and viewpoint-controllable pipeline that combines a pretrained image backbone with SMPL-X-based motion guidance, together with a training-free temporal refinement stage based on a pretrained video diffusion model. Our method produces high-quality, temporally consistent videos under diverse poses and viewpoints. We also release a canonical human dataset and an auxiliary model for compositional human image synthesis. Code and data are publicly available at https://github.com/Taited/ReImagine.
Zhengwentai Sun, Keru Zheng, Chenghong Li +7
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Future Network of Intelligence Institute, CUHK-Shenzhen, China
Text-to-motion (T2M) generation aims to create realistic human movements from text descriptions, with promising applications in animation and robotics. Despite recent progress, current T2M models perform poorly on unseen text descriptions due to the small scale and limited diversity of existing motion datasets. To address this problem, we introduce OpenT2M, a million-level, high-quality, and open-source motion dataset containing over 2800 hours of human motion. Each sequence undergoes rigorous quality control through physical feasibility validation and multi-granularity filtering, with detailed second-wise text annotations. We also develop an automated pipeline for creating long-horizon sequences, enabling complex motion generation. Building upon OpenT2M, we introduce MonoFrill, a pretrained motion model that achieves compelling T2M results without complicated designs or technique tricks as "frills". Its core component is 2D-PRQ, a novel motion tokenizer that captures spatiotemporal dependencies by dividing the human body into biology parts. Experiments show that OpenT2M significantly improves generalization of existing T2M models, while 2D-PRQ achieves superior reconstruction and strong zero-shot performance. We expect OpenT2M and MonoFrill will advance the T2M field by addressing longstanding data quality and benchmarking challenges.
Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.