FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams

Genying Li; Boda Lin; Jiachen Li; Zijian Jia; Haojie Zheng; Yiming Wang; Shuchen Weng; Si Li

cs.CVAug 16, 2026

FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams

Authors: Genying Li, Boda Lin, Jiachen Li, Zijian Jia, Haojie Zheng, Yiming Wang, Shuchen Weng, Si Li

Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · School of Software and Microelectronics, Peking University

Abstract

Music-driven dance video synthesis aims to animate a reference person according to a given music clip. The task is challenging because it requires a model to jointly learn music-to-motion correspondence, identity-preserving human animation, temporal coherence, and visually realistic video generation. We present FlowDance, a music-driven dance video generation framework that integrates explicit motion modeling with reference-preserving visual synthesis through parallel pose and RGB streams. We further introduce timestep-aware pose injection to adapt structural guidance across denoising steps and persistent identity injection to preserve the reference appearance over long video. To support this task, we further build a popularity-curated, high-resolution in-the-wild dance video dataset with synchronized music, RGB videos, 3D body motion, camera parameters, and projected 2D pose annotations. Extensive experiments show that FlowDance achieves strong performance in both dance motion generation and music-driven dance video synthesis.

Figures & tables

Figure 2: FlowDanceSet construction pipeline. Collected dance videos are filtered for quality and annotated with synchronized RGB, audio, 3D motion, camera parameters, 2D poses, and dance labels.

Figure 3: Illustration of our FlowDance framework. Left: Feature extraction provides multimodal conditioning for the RGB and pose streams. Right: FlowDance comprises parallel yet coupled pose and RGB streams. The pose stream predicts music-aligned 2D motion, whose intermediate features guide RGB generation through timestep-aware M2V Aligners. Face-token injection and face-masked reconstruction further preserve identity. Insets: Details of (a) the Face Encoder and (b) the M2V Aligner.

Figure 4: Qualitative comparison on unseen K-pop and otaku examples. Red boxes mark weak motion, deformation, identity drift, or implausible structure. FlowDance preserves appearance while producing larger, coherent movements.

	Video quality							Motion quality					User preference (%)
Method	FVD $\downarrow$	IQ $\uparrow$	AQ $\uparrow$	SC $\uparrow$	BC $\uparrow$	MS $\uparrow$	DD $\uparrow$	FID ${}_{k}\downarrow$	FID ${}_{g}\downarrow$	Div ${}_{k}\uparrow$	Div ${}_{g}\uparrow$	BAS $\uparrow$	VQ $\uparrow$	ID $\uparrow$	MN $\uparrow$	M–A $\uparrow$
GT	–	0.700	0.539	0.926	0.931	0.983	0.991	–	–	6.357	7.329	0.2331	–	–	–	–
MACE-Dance ( Yang et al. 2026 )	662.4355	0.689	0.522	0.948	0.952	0.991	0.347	51.466	92.749	6.673	8.267	0.2176	5.4	3.6	5.0	5.9
MEGADance ( Yang et al. 2025 )	409.5303	0.668	0.524	0.940	0.941	0.988	0.906	9.271	13.243	6.132	7.444	0.2160	3.6	3.8	5.3	13.8
EchoMimic ( Chen et al. 2024 )	571.0114	0.693	0.510	0.954	0.954	0.991	0.008	38.699	1087.053	3.757	27.625	0.2088	11.0	20.8	1.0	0.9
Wan-S2V ( Gao et al. 2025b )	328.0243	0.671	0.537	0.924	0.936	0.980	0.979	16.069	12.8056	7.386	7.829	0.2411	15.0	12.3	24.5	9.9

Table 1: Quantitative comparison on the test set. Arrows indicate preferred directions. The best and second-best generated results in each column are highlighted in bold and underlined , respectively. GT is an unranked reference.

Variant	FVD $\downarrow$	SC $\uparrow$	MS $\uparrow$	FID ${}_{g}\downarrow$	BAS $\uparrow$
w/o pose stream	387.06	0.876	0.962	34.166	0.2081
fixed pose injection	278.63	0.946	0.983	8.943	0.2321
ArcFace only	249.86	0.937	0.990	7.616	0.2436
w/o face reconstruction loss	254.45	0.931	0.991	7.998	0.2402
Full model (ours)	243.37	0.951	0.993	7.501	0.2450

Table 2: Component ablation. Bold and underline denote the best and second-best results.

Figure 5: A 12-second sequence generated by overlap-window inference. Identity, costume, and scene remain stable as the pose evolves.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data
Jun 29, 2026Kaixing Yang, Jiashu Zhu, Xulong Tang +8Audio-Video Generation Multimodal Generation
Renmin University of China, Beijing, China · AMAP, Alibaba Group, Beijing, China · Malou Tech Inc, Plano, Texas, USA +2
Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation
Jul 10, 2026Mingyang Huang, Peng Zhang, Li Hu +2Video Diffusion Models Long Video Generation
Tongyi Lab, Alibaba Group
FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Nov 26, 2025Kaixing Yang, Xulong Tang, Ziqiao Peng +4MeanFlow Human Motion Generation
Renmin University of China · Malou Tech Inc · Wuhan University +1

Jun 29, 2026J/K move · Enter open · S save

OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

Kaixing Yang, Jiashu Zhu, Xulong Tang, Ziqiao Peng+7

Renmin University of China, Beijing, China · AMAP, Alibaba Group, Beijing, China · Malou Tech Inc, Plano, Texas, USA+2

Audio-Video Generation Multimodal Generation Video Generation Controllable Video Generation

Open paper AskPDF arXiv

Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models. To address these limitations, we introduce CIPE-Dance, a large-scale Internet-sourced dance video dataset with choreography-informed text annotations, constructed via a progressive expert pipeline. To the best of our knowledge, CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours and covering diverse dancers, environments, and dance genres. We further propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing its original controllability or visual fidelity. Motivated by the complementary roles of text as low-frequency semantics and music as high-frequency temporal dynamics, OmniDance co-designs a depth-aware specialization architecture, an anchored easy-to-hard curriculum learning strategy, and a modality-specialized time-dependent CFG strategy, enabling unified TI2V, MI2V, and MTI2V generation. Extensive experiments on CIPE-Dance demonstrate that OmniDance achieves state-of-the-art performance across all three tasks and exhibits robust multimodal integration capability. Project is available at https://github.com/AMAP-ML/OmniDance.

Abstract

Audio-Video Generation

Explore similar work

OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation