Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.
Figures & tables
Figure 1: Generated dance from in-the-wild music.
Figure 2: Overview of TokenDance.
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
GT
/
/
8.19
7.45
0.2374
FACT [ 21 ]
35.35
22.11
5.94
6.18
0.2209
Bailando [ 28 ]
28.16
9.62
7.83
6.34
0.2332
EDGE [ 31 ]
42.16
22.12
3.96
4.61
0.2334
Lodge [ 19 ]
37.09
18.79
5.58
4.85
0.2423
TokenDance
21.55
11.85
8.05
7.12
0.2313
Table 1: Quantitative analysis on the AIST++ dataset.
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
GT
/
/
9.73
7.44
0.2120
FACT [ 21 ]
113.38
97.05
3.36
6.37
0.1831
Bailando [ 28 ]
82.81
28.17
7.74
6.25
0.2029
EDGE [ 31 ]
94.34
50.38
8.13
6.45
0.2116
Lodge [ 19 ]
45.56
34.29
6.75
5.64
0.2397
TokenDance
47.20
31.85
6.62
6.81
0.2385
Table 2: Quantitative analysis on the FineDance dataset.
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
GT
/
/
8.32
7.68
0.2603
FACT [ 21 ]
37.62
26.32
5.63
6.13
0.2162
Bailando [ 28 ]
29.56
22.47
5.92
6.29
0.2253
EDGE [ 31 ]
34.58
23.72
6.13
6.48
0.2334
POPDG [ 24 ]
27.13
21.41
6.52
6.37
0.2403
TokenDance
17.77
15.95
6.25
6.94
0.2326
Table 3: Quantitative analysis on the PopDanceSet dataset.
Method
Latency@1024 ↓
Latency@4096 ↓
FACT [ 21 ]
35.88 s
142.12 s
Bailando [ 28 ]
5.46 s
14.72 s
EDGE [ 31 ]
8.59 s
27.91 s
Lodge [ 19 ]
4.57 s
11.96 s
TokenDance
1.22 s
2.31 s
Table 4: Comparison on computational latency.
Figure 3: Qualitative comparison with SOTAs on in-the-wild test samples by models trained on FineDance..
Figure 4: Qualitative analysis across dance genres on in-the-wild test samples by models trained on FineDance.
Method
DS ↑
DQ ↑
DC ↑
GT
4.52±0.41
4.45±0.38
4.37±0.43
FACT [ 21 ]
2.11±0.62
2.03±0.58
1.98±0.64
Bailando [ 28 ]
3.48±0.51
3.44±0.49
3.32±0.53
EDGE [ 31 ]
3.52±0.47
3.46±0.50
3.41±0.48
Lodge [ 19 ]
3.71±0.44
3.78±0.42
3.69±0.46
TokenDance
4.12±0.39
4.09±0.37
3.96±0.41
Table 5: User study on in-the-wild test samples using models trained on the FineDance dataset.
Method
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
TokenDance
47.20
31.85
6.62
6.81
0.2385
TokenDance + MT
45.95
30.42
6.75
6.89
0.2412
Bailando [ 28 ]
82.81
28.17
7.74
6.25
0.2029
Bailando [ 28 ] + MT
68.34
24.50
8.01
6.52
0.2147
FACT [ 21 ]
113.38
97.05
3.36
6.37
0.1831
FACT [ 21 ] + MT
105.12
89.45
3.68
6.48
0.1893
Table 6: The model-agnostic effect of Music Tokenization (MT) for model architecture on the FineDance dataset.
Method
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
MFCC → MERT [ 22 ]
42.40
45.90
6.62
6.78
0.232
MERT [ 22 ] (w/o. MT)
35.10
48.60
6.42
6.50
0.228
MFCC → Jukebox [ 5 ]
36.80
32.46
6.15
6.53
0.202
Jukebox [ 5 ] (w/o. MT)
49.30
31.97
5.20
5.12
0.223
MFCC → Wav2Vec2.0 [ 2 ]
60.90
32.60
6.32
5.95
0.225
Wav2Vec2.0 [ 2 ] (w/o. MT)
84.31
65.10
5.91
6.21
0.215
Table 7: The model-agnostic effect of Music Tokenization (MT) for music representation on the FineDance dataset.
Method
MAE@S ↓
MAE@A ↓
MAE@F ↓
w/o. Music Decomp.
0.642
0.358
0.544
TokenDance (Full)
0.519
0.269
0.345
Table 8: Ablation study in music reconstruction task on the FineDance dataset. S, A, and F represent the Semantic, Acoustic, and Full settings, respectively.
Method
FID k↓
FID g↓
DIV k↑
DIV g↑
BAS ↑
w/o. Music Decomp.
53.12
36.45
6.10
6.22
0.225
BiMamba → Mamba
46.95
34.20
6.55
6.48
0.231
BiMamba → Transformer
61.37
42.78
5.75
5.89
0.210
TokenDance (Full)
47.20
31.85
6.62
6.81
0.239
Table 9: Ablation study in music-to-dance generation task.
Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization. Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.
Kaixing Yang, Xulong Tang, Ziqiao Peng +4
Renmin University of China · Malou Tech Inc · Wuhan University +1
Dance is a form of human motion characterized by emotional expression and communication, playing a role in various fields such as music, virtual reality, and content creation. Existing methods for dance generation often fail to adequately capture the inherently sequential, rhythmical, and music-synchronized characteristics of dance. In this paper, we propose MambaDance, a new dance generation approach that leverages a Mamba-based diffusion model. Mamba, well-suited to handling long and autoregressive sequences, is integrated into our two-stage diffusion architecture, substituting off-the-shelf Transformer. Additionally, considering the critical role of musical beats in dance choreography, we propose a Gaussian-based beat representation to explicitly guide the decoding of dance sequences. Experiments on AIST++ and FineDance datasets for each sequence length show that our proposed method effectively generates plausible dance movements while reflecting essential characteristics, consistently from short to long dances, compared to the previous methods. Additional qualitative results and demo videos are available at \small{https://vision3d-lab.github.io/mambadance}.
Sangjune Park, Inhyeok Choi, Donghyeon Soon +2
Ulsan National Institute of Science and Technology, South Korea · Daegu Gyeongbuk Institute of Science and Technology, South Korea
Although existing 3D dance generation methods perform well in controlled scenarios, they often struggle to generalize in the wild. When conditioned on unseen music, existing methods often produce unstructured or physically implausible dance, largely due to limited music-to-dance data and restricted model capacity. This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design. (1) On the data side, we develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos. To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints that enforce physical plausibility while preserving kinematic smoothness and expressiveness, resulting in a diverse, high-quality multimodal 3D dance dataset totaling 100.69 hours. (2) On model design, we propose Choreographic LLaMA (ChoreoLLaMA), a scalable LLaMA-based architecture. To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt. Additionally, we design a slow/fast-cadence Mixture-of-Experts (MoE) module that enables ChoreoLLaMA to smoothly adapt motion rhythms across varying music tempos. Extensive experiments across diverse dance genres show that our approach surpasses existing methods in both qualitative and quantitative evaluations, marking a step toward scalable, real-world 3D dance generation. Code, models, and data will be released.
Ronghui Li, Zhongyuan Hu, Li Siyao +6
Tsinghua University · Peng Cheng Laboratory · S-Lab, Nanyang Technological University