TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

Ziyue Yang; Kaixing Yang; Xulong Tang

cs.AIMar 28, 2026

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

Authors: Ziyue Yang, Kaixing Yang, Xulong Tang

Organizations: Brown University · Renmin University of China · The University of Texas at Dallas

Abstract

Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.

Figures & tables

Figure 1: Generated dance from in-the-wild music.

	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
GT	/	/	8.19	7.45	0.2374
FACT [ 21 ]	35.35	22.11	5.94	6.18	0.2209
Bailando [ 28 ]	28.16	9.62	7.83	6.34	0.2332
EDGE [ 31 ]	42.16	22.12	3.96	4.61	0.2334
Lodge [ 19 ]	37.09	18.79	5.58	4.85	0.2423
TokenDance	21.55	11.85	8.05	7.12	0.2313

Table 1: Quantitative analysis on the AIST++ dataset.

	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
GT	/	/	9.73	7.44	0.2120
FACT [ 21 ]	113.38	97.05	3.36	6.37	0.1831
Bailando [ 28 ]	82.81	28.17	7.74	6.25	0.2029
EDGE [ 31 ]	94.34	50.38	8.13	6.45	0.2116
Lodge [ 19 ]	45.56	34.29	6.75	5.64	0.2397
TokenDance	47.20	31.85	6.62	6.81	0.2385

Table 2: Quantitative analysis on the FineDance dataset.

	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
GT	/	/	8.32	7.68	0.2603
FACT [ 21 ]	37.62	26.32	5.63	6.13	0.2162
Bailando [ 28 ]	29.56	22.47	5.92	6.29	0.2253
EDGE [ 31 ]	34.58	23.72	6.13	6.48	0.2334
POPDG [ 24 ]	27.13	21.41	6.52	6.37	0.2403
TokenDance	17.77	15.95	6.25	6.94	0.2326

Table 3: Quantitative analysis on the PopDanceSet dataset.

Method	Latency@1024 $\downarrow$	Latency@4096 $\downarrow$
FACT [ 21 ]	35.88 s	142.12 s
Bailando [ 28 ]	5.46 s	14.72 s
EDGE [ 31 ]	8.59 s	27.91 s
Lodge [ 19 ]	4.57 s	11.96 s
TokenDance	1.22 s	2.31 s

Table 4: Comparison on computational latency.

Figure 3: Qualitative comparison with SOTAs on in-the-wild test samples by models trained on FineDance..

Figure 4: Qualitative analysis across dance genres on in-the-wild test samples by models trained on FineDance.

Method	DS $\uparrow$	DQ $\uparrow$	DC $\uparrow$
GT	$4.52\pm 0.41$	$4.45\pm 0.38$	$4.37\pm 0.43$
FACT [ 21 ]	$2.11\pm 0.62$	$2.03\pm 0.58$	$1.98\pm 0.64$
Bailando [ 28 ]	$3.48\pm 0.51$	$3.44\pm 0.49$	$3.32\pm 0.53$
EDGE [ 31 ]	$3.52\pm 0.47$	$3.46\pm 0.50$	$3.41\pm 0.48$
Lodge [ 19 ]	$3.71\pm 0.44$	$3.78\pm 0.42$	$3.69\pm 0.46$
TokenDance	$\mathbf{4.12\pm 0.39}$	$\mathbf{4.09\pm 0.37}$	$\mathbf{3.96\pm 0.41}$

Table 5: User study on in-the-wild test samples using models trained on the FineDance dataset.

Method	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
TokenDance	47.20	31.85	6.62	6.81	0.2385
TokenDance + MT	45.95	30.42	6.75	6.89	0.2412
Bailando [ 28 ]	82.81	28.17	7.74	6.25	0.2029
Bailando [ 28 ] + MT	68.34	24.50	8.01	6.52	0.2147
FACT [ 21 ]	113.38	97.05	3.36	6.37	0.1831
FACT [ 21 ] + MT	105.12	89.45	3.68	6.48	0.1893

Table 6: The model-agnostic effect of Music Tokenization (MT) for model architecture on the FineDance dataset.

Method	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
MFCC $\rightarrow$ MERT [ 22 ]	42.40	45.90	6.62	6.78	0.232
MERT [ 22 ] (w/o. MT)	35.10	48.60	6.42	6.50	0.228
MFCC $\rightarrow$ Jukebox [ 5 ]	36.80	32.46	6.15	6.53	0.202
Jukebox [ 5 ] (w/o. MT)	49.30	31.97	5.20	5.12	0.223
MFCC $\rightarrow$ Wav2Vec2.0 [ 2 ]	60.90	32.60	6.32	5.95	0.225
Wav2Vec2.0 [ 2 ] (w/o. MT)	84.31	65.10	5.91	6.21	0.215

Table 7: The model-agnostic effect of Music Tokenization (MT) for music representation on the FineDance dataset.

Method	MAE@S $\downarrow$	MAE@A $\downarrow$	MAE@F $\downarrow$
w/o. Music Decomp.	0.642	0.358	0.544
TokenDance (Full)	0.519	0.269	0.345

Table 8: Ablation study in music reconstruction task on the FineDance dataset. S, A, and F represent the Semantic, Acoustic, and Full settings, respectively.

Method	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	DIV ${}_{k}\uparrow$	DIV ${}_{g}\uparrow$	BAS $\uparrow$
w/o. Music Decomp.	53.12	36.45	6.10	6.22	0.225
BiMamba $\rightarrow$ Mamba	46.95	34.20	6.55	6.48	0.231
BiMamba $\rightarrow$ Transformer	61.37	42.78	5.75	5.89	0.210
TokenDance (Full)	47.20	31.85	6.62	6.81	0.239

Table 9: Ablation study in music-to-dance generation task.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Nov 26, 2025Kaixing Yang, Xulong Tang, Ziqiao Peng +4MeanFlow Human Motion Generation
Renmin University of China · Malou Tech Inc · Wuhan University +1
Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model
Mar 9, 2026Sangjune Park, Inhyeok Choi, Donghyeon Soon +2Human Motion Generation Diffusion Models for Motion Generation
Ulsan National Institute of Science and Technology, South Korea · Daegu Gyeongbuk Institute of Science and Technology, South Korea
InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization
Mar 10, 2026Ronghui Li, Zhongyuan Hu, Li Siyao +6Human Motion Generation Diffusion Models for Motion Generation
Tsinghua University · Peng Cheng Laboratory · S-Lab, Nanyang Technological University

Nov 26, 2025J/K move · Enter open · S save

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation

Kaixing Yang, Xulong Tang, Ziqiao Peng, Xiangyue Zhang+3

Renmin University of China · Malou Tech Inc · Wuhan University+1

MeanFlow Human Motion Generation

Open paper AskPDF arXiv

Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization. Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.

Abstract

Human Motion Generation

Explore similar work

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation

Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model

InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization