InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization

Ronghui Li; Zhongyuan Hu; Li Siyao; Youliang Zhang; Haozhe Xie; Mingyuan Zhang; Jie Guo; Xiu Li

cs.CVMar 10, 2026

InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization

Authors: Ronghui Li, Zhongyuan Hu, Li Siyao, Youliang Zhang, Haozhe Xie, Mingyuan Zhang, Jie Guo, Xiu Li,

Organizations: Tsinghua University · Peng Cheng Laboratory · S-Lab, Nanyang Technological University

Abstract

Although existing 3D dance generation methods perform well in controlled scenarios, they often struggle to generalize in the wild. When conditioned on unseen music, existing methods often produce unstructured or physically implausible dance, largely due to limited music-to-dance data and restricted model capacity. This work aims to push the frontier of generalizable 3D dance generation by scaling up both data and model design. (1) On the data side, we develop a fully automated pipeline that reconstructs high-fidelity 3D dance motions from monocular videos. To eliminate the physical artifacts prevalent in existing reconstruction methods, we introduce a Foot Restoration Diffusion Model (FRDM) guided by foot-contact and geometric constraints that enforce physical plausibility while preserving kinematic smoothness and expressiveness, resulting in a diverse, high-quality multimodal 3D dance dataset totaling 100.69 hours. (2) On model design, we propose Choreographic LLaMA (ChoreoLLaMA), a scalable LLaMA-based architecture. To enhance robustness under unfamiliar music conditions, we integrate a retrieval-augmented generation (RAG) module that injects reference dance as a prompt. Additionally, we design a slow/fast-cadence Mixture-of-Experts (MoE) module that enables ChoreoLLaMA to smoothly adapt motion rhythms across varying music tempos. Extensive experiments across diverse dance genres show that our approach surpasses existing methods in both qualitative and quantitative evaluations, marking a step toward scalable, real-world 3D dance generation. Code, models, and data will be released.

Figures & tables

Dataset	Acquisition	Joints num			Genres	Representation	Dancers	T(h)	$\bar{\text{T}}$ (s)
Dance w/. Melody [ 41 ]	MoCap	21	✗	✗	4	3D joints	-	1.6	92.5
Music2Dance [ 57 ]	MoCap	55	✓	✗	2	3D joints	2	0.96	-
EA-MUD [ 39 ]	Pseudo	24	✗	✗	4	3D joints	-	0.35	73.8
PhantomDance [ 17 ]	Animator	24	✗	✗	13	SMPL	100+	9.6	133.3
AIST++ [ 24 ]	Pseudo	17/24	✗	✗	10	SMPL	30	5.2	13.3
MMD [ 4 ]	MoCap	52	✓	✗	4	FBX	-	9.9	-

Table 1: Comparisons of 3D Dance Datasets. denotes whether containing hand (finger) motion. represents facial expression.

\bar{\text{T}}

(sec) denotes the average seconds per sequence. Acquisition methods: MoCap (captured with professional motion capture equipment), Pseudo (previous video-based motion estimation), Animator (manually keyframed by animators).

Figure 2: Overview of our motion collection pipeline. Step 1: We estimate whole-body motion from monocular videos, which contain artifacts. Step 2: We refine these motions through motion imitation in a physics simulator to obtain more physically plausible results, but this step often introduces frequent foot jittering. Step 3: We apply our Foot Restoration Diffusion Model (FRDM) to further correct foot motions. The final results show stable root and foot contacts without jittering or penetration artifacts.

Figure 3: (a) The Foot Restoration Diffusion Model (FRDM) can be trained in a self-supervised manner. We sample $\bm{x}_{0}$ from ground-truth motions and obtain $\bm{x}_{t}$ by adding noise. To repair only the artifacts in the root, knees, and feet, we replace these parts in $\bm{x}_{0}$ with the corresponding components from $\bm{x}_{t}$ to obtain $\acute{\bm{x}}_{t}$ . We then train a foot denoising network $\bm{f_{\theta}}.\hat{\bm{x}}_{0}=\bm{f_{\theta}}(\acute{\bm{x}}_{t},t)$ , $\bm{j}^{p}_{0}=\text{Cumsum}(\bm{j}^{v}_{0}),\bm{j}^{p}_{0}=\text{FK}(\bm{j}^{r}_{0})$ . (b) Given the motion $\bm{x}$ with foot artifacts, we first sampel $\bm{x}_{T}\sim\mathcal{N}(0,I)$ , and get $\acute{\bm{x}}_{t}$ by replace the root, knee and foot reigon of $\bm{x}$ by those of $\bm{x}_{t}$ . In the early denoising steps $t>t_{th}$ , where $t_{th}$ is a threshold, we apply geometric guidance to keep the restored motion geometrically consistent with the original input. In the last steps $t\leq t_{th}$ , we use foot-contact guidance to explicitly improve foot stability.

Method	Dataset	FSR $\downarrow$	Jitter $\downarrow$	Penetration $\downarrow$
MoCap	FineDance [ 23 ]	6.22%	12.69	0.6954%
GVHMR	-	28.63 $\%$	31.89	0.7864 $\%$
GVHMR+PHC	-	8.87 $\%$	78.60	0.0536 $\%$
GVHMR+PHC+Smooth	-	14.29 $\%$	15.39	0.0561 $\%$
GVHMR+PHC+FRDM (Ours)	InfiniteDance	5.09%	14.33	0.0559%

Table 2: Motion quality of InfiniteDance vs. marker-based MoCap.

Figure 4: (a) Our residual tokenizer maintains multi-layer codebooks. (b) Previous methods input discrete indices (e.g., “[ $48_{(1)}$ ]”) to LLaMA. (c) We project continuous quantized embeds $\bm{x}_{q}$ into $\bm{x}_{e}$ for LLaMA, preserving fine-grained features.

Figure 5: (a) Given a music clip and a target genre, we first use RAG to retrieve the top-k most relevant reference dances. These retrieved dances, together with the given music and genre embeds, are then fed into the Cadence-MoE to produce fused embeds. ChoreoLLaMA then autoregressively predicts dance tokens that are decoded into dance sequences. (b) Each “Expert” is a neural module composed of linear layers, multi-head attention, and a Mamba block. The “RFFT” and “IRFFT” refer to the real-valued Fast Fourier Transform and its inverse, respectively. (c) At the inference phase, the ChoreoLLaMA predicts dance token indices $\hat{\bm{x}}_{idx}$ one by one, we then lookup the quantized embeds $\hat{\bm{x}}_{q}$ in codebooks and project them in to dance embeds $\hat{\bm{x}}_{e}$ .

Method	Motion Quality			Motion Diversity		BAS $\uparrow$	Our Wins $\uparrow$
Method	$\mathrm{FID}_{k}\downarrow$	$\mathrm{FID}_{g}\downarrow$	FSR $\downarrow$	$\mathrm{Div}_{k}\uparrow$	$\mathrm{Div}_{g}\uparrow$	BAS $\uparrow$	Our Wins $\uparrow$
Ground Truth	2.55	0.60	5.09%	9.37	7.12	0.2332	$34.8\pm 22.6\%$
Bailando [ 38 ]	117.38	82.37	15.56%	5.46	5.28	0.2137	$88.7\pm 8.9\%$
EDGE [ 44 ]	96.07	63.53	14.15%	4.36	4.97	0.2321	$76.4\pm 7.7\%$
Lodge [ 22 ]	89.52	60.38	6.72%	3.93	5.00	0.2329	$68.1\pm 11.5\%$
ChoreoLLaMA(Ours)	30.54	16.31	5.33%	6.23	5.11	0.2342	-

Table 3: Comparisons on the InfiniteDance dataset. The “BAS” measures the beat alignment degree between the music and dance. “Our Wins” denotes the percentage of pairwise comparisons in which participants preferred our generated results over competing methods.

Method	Test Setting	FID k $\downarrow$	Div k $\uparrow$	BAS $\uparrow$
Lodge [ 22 ]	AIST++ [ 24 ] (cross-dataset)	48.73	4.26	0.2364
ChoreoLLaMA (Ours)	AIST++ [ 24 ] (cross-dataset)	35.45	5.79	0.2378
Lodge [ 22 ]	FineDance [ 23 ] (cross-dataset)	106.85	4.14	0.2317
ChoreoLLaMA (Ours)	FineDance [ 23 ] (cross-dataset)	59.38	5.81	0.2382
Lodge [ 22 ]	Unseen Music (OOD)	119.66	5.13	0.2332
ChoreoLLaMA (Ours)	Unseen Music (OOD)	56.22	5.52	0.2315

Table 4: Generalization experiments. Both models are trained on InfiniteDance and evaluated on cross-dataset and OOD settings.

Components				Metrics
Token Indices	Token Embeds	RAG	MoE	$\mathrm{FID}_{k}\downarrow$	$\mathrm{Div}_{k}\uparrow$	BAS $\uparrow$
✓	✗	✗	✗	79.84	13.09	0.2073
✗	✓	✗	✗	62.87	5.49	0.2269
✗	✓	✓	✗	38.74	6.16	0.2325
✗	✓	✓	✓	33.14	6.11	0.2348
✗	✓	✓	✓	30.54	6.23	0.2342

Table 5: Ablation study of different components. We progressively add each component to ChoreoLLaMA and evaluate on the InfiniteDance test set.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Duration Distribution of InfiniteDance

Classes	Genres	Duration (h)
Ballet	Ballet	8.31
Modern	Modern	3.77
Folk	Mix	0.94
	Dai	1.48
	Uygurs	2.48
	Mongol	1.34

Appendix

Table 6: Coarse classes and fine-grianed genres distribution of the InfiniteDance dataset.

Reference Dances	FID k $\downarrow$	FID m $\downarrow$	Div k $\uparrow$	BAS $\uparrow$
Top-10	38.74	13.48	6.92	0.2340
Top-100–110	47.16	17.34	6.56	0.2339
Top-500–510	51.76	18.73	6.58	0.2335
Top-1000–1010	58.97	18.92	6.39	0.2327
No RAG	62.87	147.41	4.68	0.2269

Appendix

Table 7: Ablation on different retrieved reference similarity ranks.

Distance Type	Kinematic (K)	Geometric (G)	Joint Pos. (J)
Retrieved–Retrieved	11.67	7.97	27.39
Generated–Retrieved	23.52	10.11	43.13
Generated–GT (avg.)	297.94	20.87	57.11

Appendix

Table 8: Feature-space distance analysis: generated motions show influence without copying.

Figure 7: A case show the difference between reference dance and generated dance.

Method	FID k $\downarrow$	FID m $\downarrow$	Div k $\uparrow$	Div m $\uparrow$	BAS $\uparrow$
Lodge [ 22 ]	106.85	91.30	4.14	4.43	0.2306
ChoreoLLaMA (Ours)	39.72	24.45	6.01	4.38	0.2335

Appendix

Table 9: Long-term choreography evaluation (sequences

>

30 seconds).

Figure 8: Frequency energy distribution of reference dance motions. Most energy is concentrated in the low-frequency band, justifying the choice of $\text{nbins}=2$ .

nbins	FID k $\downarrow$	FID m $\downarrow$	Div k $\uparrow$	Div m $\uparrow$	BAS $\uparrow$
2	43.19	18.38	9.05	7.03	0.2335
4	118.49	58.47	12.95	12.44	0.2375

Appendix

Table 10: Cadence-MoE frequency band ablation (60% training data subset).

Figure 9: Visualization of 3D motion capture results from monocular video using GVHMR and SMPLest-X. While this step already captures fine-grained motion details, the results still suffer from physically implausible artifacts. Therefore, we do not use these motions directly. Instead, we further refine them with a physical simulation-based correction and our Foot Restoration Diffusion Model (FRDM) to enforce accurate foot–ground contact.

Figure 10: Visualization of InfiniteDance.

Method	$\mathrm{FID}\downarrow$	$\mathrm{SSIM}\uparrow$	$\mathrm{PSNR}\uparrow$	$\mathrm{LPIPS}\downarrow$	L1 $\downarrow$	FID-FVD $\downarrow$	FID $\downarrow$
DisCo [ 47 ]	57.84	0.51	10.66	0.47	2.4e-4	43.17	522.28
Animate Anyone [ 13 ]	45.90	0.54	12.82	0.44	1.5e-4	33.32	505.01
Champ [ 55 ]	42.61	0.57	13.21	0.41	1.3e-4	31.27	447.21
Ours	37.94	0.62	14.60	0.37	9.7e-5	24.34	405.27

Appendix

Table 11: Quantitative comparison with SOTA methods.

Figure 11: Visualization of pose-guided 2D dance generation

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Nov 26, 2025Kaixing Yang, Xulong Tang, Ziqiao Peng +4MeanFlow Human Motion Generation
Renmin University of China · Malou Tech Inc · Wuhan University +1
TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba
Mar 28, 2026Ziyue Yang, Kaixing Yang, Xulong TangHuman Motion Generation
Brown University · Renmin University of China · The University of Texas at Dallas
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Jul 12, 2026Ryota Kimura, Sangheon Park, Natalia Polouliakh +1Controllable Music Generation Cross-Modal Alignment
Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology

Nov 26, 2025J/K move · Enter open · S save

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation

Kaixing Yang, Xulong Tang, Ziqiao Peng, Xiangyue Zhang+3

Renmin University of China · Malou Tech Inc · Wuhan University+1

MeanFlow Human Motion Generation

Open paper AskPDF arXiv

Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization. Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.

Abstract

Human Motion Generation

Explore similar work

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment