Although existing 3D dance generation methods perform well in controlled scenarios, they often struggle to generalize in the wild. When conditioned on unseen music, existing methods often produce unstructured or physically implausible dance, largely due to limited music-to-dance data and restricted model capacity. This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design. (1) On the data side, we develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos. To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints that enforce physical plausibility while preserving kinematic smoothness and expressiveness, resulting in a diverse, high-quality multimodal 3D dance dataset totaling 100.69 hours. (2) On model design, we propose Choreographic LLaMA (ChoreoLLaMA), a scalable LLaMA-based architecture. To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt. Additionally, we design a slow/fast-cadence Mixture-of-Experts (MoE) module that enables ChoreoLLaMA to smoothly adapt motion rhythms across varying music tempos. Extensive experiments across diverse dance genres show that our approach surpasses existing methods in both qualitative and quantitative evaluations, marking a step toward scalable, real-world 3D dance generation. Code, models, and data will be released.
Figures & tables
Dataset
Acquisition
Joints num
Genres
Representation
Dancers
T(h)
Tˉ (s)
Dance w/. Melody [ 41 ]
MoCap
21
✗
✗
4
3D joints
-
1.6
92.5
Music2Dance [ 57 ]
MoCap
55
✓
✗
2
3D joints
2
0.96
-
EA-MUD [ 39 ]
Pseudo
24
✗
✗
4
3D joints
-
0.35
73.8
PhantomDance [ 17 ]
Animator
24
✗
✗
13
SMPL
100+
9.6
133.3
AIST++ [ 24 ]
Pseudo
17/24
✗
✗
10
SMPL
30
5.2
13.3
MMD [ 4 ]
MoCap
52
✓
✗
4
FBX
-
9.9
-
Table 1: Comparisons of 3D Dance Datasets. denotes whether containing hand (finger) motion. represents facial expression. Tˉ (sec) denotes the average seconds per sequence. Acquisition methods: MoCap (captured with professional motion capture equipment), Pseudo (previous video-based motion estimation), Animator (manually keyframed by animators).
Figure 2: Overview of our motion collection pipeline. Step 1: We estimate whole-body motion from monocular videos, which contain artifacts. Step 2: We refine these motions through motion imitation in a physics simulator to obtain more physically plausible results, but this step often introduces frequent foot jittering. Step 3: We apply our Foot Restoration Diffusion Model (FRDM) to further correct foot motions. The final results show stable root and foot contacts without jittering or penetration artifacts.
Figure 3: (a) The Foot Restoration Diffusion Model (FRDM) can be trained in a self-supervised manner. We sample x0 from ground-truth motions and obtain xt by adding noise. To repair only the artifacts in the root, knees, and feet, we replace these parts in x0 with the corresponding components from xt to obtain xˊt . We then train a foot denoising network fθ.x^0=fθ(xˊt,t) , j0p=Cumsum(j0v),j0p=FK(j0r) . (b) Given the motion x with foot artifacts, we first sampel xT∼N(0,I) , and get xˊt by replace the root, knee and foot reigon of x by those of xt . In the early denoising steps t>tth , where tth is a threshold, we apply geometric guidance to keep the restored motion geometrically consistent with the original input. In the last steps t≤tth , we use foot-contact guidance to explicitly improve foot stability.
Method
Dataset
FSR ↓
Jitter ↓
Penetration ↓
MoCap
FineDance [ 23 ]
6.22%
12.69
0.6954%
GVHMR
-
28.63 %
31.89
0.7864 %
GVHMR+PHC
-
8.87 %
78.60
0.0536 %
GVHMR+PHC+Smooth
-
14.29 %
15.39
0.0561 %
GVHMR+PHC+FRDM (Ours)
InfiniteDance
5.09%
14.33
0.0559%
Table 2: Motion quality of InfiniteDance vs. marker-based MoCap.
Figure 4: (a) Our residual tokenizer maintains multi-layer codebooks. (b) Previous methods input discrete indices (e.g., “[ 48(1) ]”) to LLaMA. (c) We project continuous quantized embeds xq into xe for LLaMA, preserving fine-grained features.
Figure 5: (a) Given a music clip and a target genre, we first use RAG to retrieve the top-k most relevant reference dances. These retrieved dances, together with the given music and genre embeds, are then fed into the Cadence-MoE to produce fused embeds. ChoreoLLaMA then autoregressively predicts dance tokens that are decoded into dance sequences. (b) Each “Expert” is a neural module composed of linear layers, multi-head attention, and a Mamba block. The “RFFT” and “IRFFT” refer to the real-valued Fast Fourier Transform and its inverse, respectively. (c) At the inference phase, the ChoreoLLaMA predicts dance token indices x^idx one by one, we then lookup the quantized embeds x^q in codebooks and project them in to dance embeds x^e .
Method
Motion Quality
Motion Diversity
BAS ↑
Our Wins ↑
FIDk↓
FIDg↓
FSR ↓
Divk↑
Divg↑
Ground Truth
2.55
0.60
5.09%
9.37
7.12
0.2332
34.8±22.6%
Bailando [ 38 ]
117.38
82.37
15.56%
5.46
5.28
0.2137
88.7±8.9%
EDGE [ 44 ]
96.07
63.53
14.15%
4.36
4.97
0.2321
76.4±7.7%
Lodge [ 22 ]
89.52
60.38
6.72%
3.93
5.00
0.2329
68.1±11.5%
ChoreoLLaMA(Ours)
30.54
16.31
5.33%
6.23
5.11
0.2342
-
Table 3: Comparisons on the InfiniteDance dataset. The “BAS” measures the beat alignment degree between the music and dance. “Our Wins” denotes the percentage of pairwise comparisons in which participants preferred our generated results over competing methods.
Method
Test Setting
FID k ↓
Div k ↑
BAS ↑
Lodge [ 22 ]
AIST++ [ 24 ] (cross-dataset)
48.73
4.26
0.2364
ChoreoLLaMA (Ours)
AIST++ [ 24 ] (cross-dataset)
35.45
5.79
0.2378
Lodge [ 22 ]
FineDance [ 23 ] (cross-dataset)
106.85
4.14
0.2317
ChoreoLLaMA (Ours)
FineDance [ 23 ] (cross-dataset)
59.38
5.81
0.2382
Lodge [ 22 ]
Unseen Music (OOD)
119.66
5.13
0.2332
ChoreoLLaMA (Ours)
Unseen Music (OOD)
56.22
5.52
0.2315
Table 4: Generalization experiments. Both models are trained on InfiniteDance and evaluated on cross-dataset and OOD settings.
Components
Metrics
Token Indices
Token Embeds
RAG
MoE
FIDk↓
Divk↑
BAS ↑
✓
✗
✗
✗
79.84
13.09
0.2073
✗
✓
✗
✗
62.87
5.49
0.2269
✗
✓
✓
✗
38.74
6.16
0.2325
✗
✓
✓
✓
33.14
6.11
0.2348
✗
✓
✓
✓
30.54
6.23
0.2342
Table 5: Ablation study of different components. We progressively add each component to ChoreoLLaMA and evaluate on the InfiniteDance test set.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Duration Distribution of InfiniteDance
Classes
Genres
Duration (h)
Ballet
Ballet
8.31
Modern
Modern
3.77
Folk
Mix
0.94
Dai
1.48
Uygurs
2.48
Mongol
1.34
Appendix
Table 6: Coarse classes and fine-grianed genres distribution of the InfiniteDance dataset.
Reference Dances
FID k ↓
FID m ↓
Div k ↑
BAS ↑
Top-10
38.74
13.48
6.92
0.2340
Top-100–110
47.16
17.34
6.56
0.2339
Top-500–510
51.76
18.73
6.58
0.2335
Top-1000–1010
58.97
18.92
6.39
0.2327
No RAG
62.87
147.41
4.68
0.2269
Appendix
Table 7: Ablation on different retrieved reference similarity ranks.
Distance Type
Kinematic (K)
Geometric (G)
Joint Pos. (J)
Retrieved–Retrieved
11.67
7.97
27.39
Generated–Retrieved
23.52
10.11
43.13
Generated–GT (avg.)
297.94
20.87
57.11
Appendix
Table 8: Feature-space distance analysis: generated motions show influence without copying.
Figure 7: A case show the difference between reference dance and generated dance.
Figure 8: Frequency energy distribution of reference dance motions. Most energy is concentrated in the low-frequency band, justifying the choice of nbins=2 .
nbins
FID k ↓
FID m ↓
Div k ↑
Div m ↑
BAS ↑
2
43.19
18.38
9.05
7.03
0.2335
4
118.49
58.47
12.95
12.44
0.2375
Appendix
Table 10: Cadence-MoE frequency band ablation (60% training data subset).
Figure 9: Visualization of 3D motion capture results from monocular video using GVHMR and SMPLest-X. While this step already captures fine-grained motion details, the results still suffer from physically implausible artifacts. Therefore, we do not use these motions directly. Instead, we further refine them with a physical simulation-based correction and our Foot Restoration Diffusion Model (FRDM) to enforce accurate foot–ground contact.
Figure 10: Visualization of InfiniteDance.
Method
FID↓
SSIM↑
PSNR↑
LPIPS↓
L1 ↓
FID-FVD ↓
FID ↓
DisCo [ 47 ]
57.84
0.51
10.66
0.47
2.4e-4
43.17
522.28
Animate Anyone [ 13 ]
45.90
0.54
12.82
0.44
1.5e-4
33.32
505.01
Champ [ 55 ]
42.61
0.57
13.21
0.41
1.3e-4
31.27
447.21
Ours
37.94
0.62
14.60
0.37
9.7e-5
24.34
405.27
Appendix
Table 11: Quantitative comparison with SOTA methods.
Figure 11: Visualization of pose-guided 2D dance generation
Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization. Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.
Kaixing Yang, Xulong Tang, Ziqiao Peng +4
Renmin University of China · Malou Tech Inc · Wuhan University +1
Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.
Ziyue Yang, Kaixing Yang, Xulong Tang
Brown University · Renmin University of China · The University of Texas at Dallas
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .
Ryota Kimura, Sangheon Park, Natalia Polouliakh +1
Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology