Organizations: Department of Computer Science and Software Engineering, The University of Western Australia, Australia. · Munich Center for Machine Learning (MCML) and Technical University of Munich (TUM), Germany. · School of Information Technology, Murdoch University, Australia. · Department of Electrical, Electronics and Computer Engineering, The University of Western Australia, Australia.
Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.
Figures & tables
Fig. 1: Overview of the text-to-skeleton generation architecture for training. A text prompt is encoded and prepended as a conditioning prefix to the pose token sequence. The autoregressive Transformer predicts each joint token conditioned on all previously generated tokens and the text description.
Fig. 2: Overview of the text-to-skeleton generation architecture for inference.
Fig. 3: Overview of the pose-conditioned video generation architecture. A reference image is encoded via DINO-ALF to produce appearance tokens, while the skeleton sequence is rasterized and encoded by a 3D CNN into spatiotemporally aligned motion tokens. Both conditioning streams are injected into the DiT denoiser to synthesize the output video.
Fig. 4: Patch-feature magnitude maps ( ℓ2 -norm) across DINOv3 layers. Earlier layers exhibit high activation on the subject and texture-rich regions, while later layers show more uniform magnitudes. This motivates using an early layer as the query for adaptive layer fusion.
Fig. 5: Cross-attention maps for CLIP (top) vs. DINO-ALF (bottom) on a backflip sequence. DINO-ALF attends more precisely to the moving subject, while CLIP attention is scattered.
Method
FID ↓
Rp-top1 ↑
Rp-top2 ↑
Rp-top3 ↑
Diversity ↑
MM-Dist ↓
T2M-GPT [ 67 ]
524.61
0.191
0.287
0.473
40.11
49.85
PriorMDM [ 68 ]
585.31
0.216
0.325
0.501
42.58
44.29
MLD [ 69 ]
467.22
0.335
0.503
0.653
41.67
47.66
HumanDreamer [ 45 ]
322.16
0.411
0.598
0.722
45.33
41.53
Ours
255.19
0.487
0.667
0.784
48.33
38.65
TABLE I: Quantitative evaluation for Text-to-Skeleton generation on our 8,000-pair benchmark. ↓ indicates lower is better and ↑ indicates higher is better.
Model
VBench-I2V
Frame-level
Video-level
Subj. Cons ↑
Bg. Cons ↑
Motion Flick. ↑
Motion Smooth. ↑
Dyn.Degree ↑
Aes.Qual ↑
Img.Qual ↑
SSIM ↑
LPIPS ↓
PSNR ↑
FVD ↓
HumanVid [ 38 ]
85.33
86.91
91.01
88.21
72.72
54.61
64.19
0.737
0.213
25.54
637.1
MimicMotion [ 41 ]
86.01
88.47
92.11
89.95
73.05
55.07
65.14
0.744
0.198
26.66
611.5
Hyper-Motion [ 17 ]
86.66
90.23
93.22
90.08
73.65
55.21
65.42
0.767
0.181
27.13
568.8
UniAnimate-DiT [ 37 ]
87.21
91.17
95.78
92.33
74.87
56.32
66.87
0.751
0.189
27.48
527.4
VACE [ 35 ]
88.31
91.11
95.31
94.81
75.03
57.11
67.91
0.771
0.173
27.91
498.7
TABLE II: Comparison of different methods on VBench. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Configuration
FID ↓
Rp-top1 ↑
Diversity ↑
MM-Dist ↓
Tokenization granularity ( K bins)
K=64
312.41
0.401
44.21
43.17
K=128
278.55
0.452
46.88
40.52
K=256 (Baseline)
255.19
0.487
48.33
38.65
K=512
261.33
0.479
47.65
39.21
Decoder depth ( L layers)
TABLE III: Ablation study on the text-to-skeleton architecture. We analyze the effect of tokenization granularity ( K ), decoder depth ( L ), and decoding strategy. ↓ indicates lower is better; ↑ indicates higher is better.
Fig. 6: Qualitative comparison of generated 2D pose sequences for the prompt “A person performs a cartwheel”. Our method produces more coherent and physically plausible trajectories, especially during inverted phases. Failure cases of competing methods are highlighted with red boxes. For better visualization, we crop the scene to show only the human skeleton.
Fig. 7: Qualitative video comparison with pose-guided baselines. Our method best follows the target 2D skeleton while preserving the reference appearance. Competing methods exhibit hand/leg artifacts under complex motions (red boxes), and our DINO-ALF conditioning better retains fine details (e.g., the red tie). More results are on our project page .
Configuration
Subj. ↑
Bg. ↑
M.Smooth ↑
SSIM ↑
LPIPS ↓
FVD ↓
LoRA rank ( r )
r=16
84.22
90.45
92.11
0.712
0.221
512.8
r=32
88.77
91.58
95.88
0.758
0.193
491.5
r=64 (Baseline)
91.31
93.79
97.39
0.795
0.174
471.3
r=128
91.33
93.72
97.28
0.789
0.178
478.9
Motion encoder architecture
TABLE IV: Ablation study on pose-to-video network parameters. We analyze the effect of LoRA rank, motion encoder architecture, and condition dropout rate. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Model
VBench-I2V
Frame-level
Video-level
Subj. Cons ↑
Bg. Cons ↑
Motion Flick. ↑
Motion Smooth. ↑
Dyn.Degree ↑
Aes.Qual ↑
Img.Qual ↑
SSIM ↑
LPIPS ↓
PSNR ↑
FVD ↓
DLL (single-layer)
87.65
90.03
94.11
93.77
74.13
55.11
66.23
0.752
0.181
27.02
502.5
CLIP embeddings
85.44
89.75
93.63
91.21
73.02
54.76
66.08
0.741
0.195
26.98
517.7
No appearance encoder
75.13
85.14
82.22
82.11
65.53
47.11
60.09
0.673
0.245
24.33
593.4
Ours
91.31
93.79
97.50
97.39
78.50
56.37
67.14
0.795
0.174
28.50
471.3
TABLE V: Ablation study on appearance encoding design. We compare DINO-ALF (Ours), DLL (single-layer), CLIP embeddings, and no appearance encoder. Higher is better for ( ↑ ); lower is better for ( ↓ ).
Fig. 8: Ablation study on the pose-to-video stage. Rows from top to bottom: Ref (reference frame and target pose), CLIP (CLIP embeddings), DINO LL (DLL single-layer), W/O cross (no appearance encoder), and Ours. DINO-ALF yields the best appearance preservation under large pose changes, while CLIP-only or no appearance encoder causes hand/clothing artifacts and identity drift (red boxes).
Configuration
Subj. ↑
Bg. ↑
M.Smooth ↑
SSIM ↑
LPIPS ↓
FVD ↓
GT skeletons (upper bound)
91.31
93.79
97.39
0.795
0.174
471.3
No augmentation
85.11
86.57
89.07
0.652
0.229
574.7
Joint jitter only
86.45
87.26
91.39
0.651
0.213
568.1
Joint dropout only
86.57
87.73
91.11
0.683
0.208
541.5
Temporal shift only
87.31
88.05
92.71
0.713
0.193
528.3
Full augmentation (Ours)
89.23
91.17
95.61
0.781
0.183
492.5
TABLE VI: Ablation study on pose augmentation for error robustness. All rows except the first use predicted skeletons at inference; the GT row serves as an upper bound. Higher is better for ( ↑ ); lower is better for ( ↓ ).