SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

Tao Feng; Xu Li; Xiangyang Luo; Ming Wen; Huadai Liu; Chen Zhang; Wei Xue

cs.SDAug 17, 2026

SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

Authors: Tao Feng, Xu Li, Xiangyang Luo, Ming Wen, Huadai Liu, Chen Zhang, Wei Xue

Organizations: Kling Team, Kuaishou Technology · The Hong Kong University of Science and Technology · Tsinghua University

Abstract

Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.

Figures & tables

Figure 1: Four generation tasks supported by SingDance. The dashed box marks singing-and-dancing, the held-out Song/Source configuration generated through compositional zero-shot inference.

Figure 2: Training architecture of SingDance. The first video frame is separately encoded as a clean reference latent, while subsequent frames are temporally compressed and noised in latent space. Wav2Vec 2.0 and MuQ features, together with the vocal-role condition, are selected by hard-compact routing and supplied to frame-wise joint audio injection. The repeated block is schematic: joint audio injection is applied immediately after each of 10 selected blocks in the 30-block DiT backbone. The denoised latents are decoded into output frames. Snowflakes and flames denote frozen and trainable modules, respectively.

Audio type	Role	Routed tokens	Task
Speech	Source	$S+R_{\mathrm{src}}$	Speaking
Speech	Listener	$S+R_{\mathrm{lis}}$	Listening/Reacting
Instrumental	Listener	$R_{\mathrm{lis}}+M$	Dancing-Only
Song	Listener	$S+R_{\mathrm{lis}}+M$	Dancing-Only
Song	Source	$S+R_{\mathrm{src}}+M$	Singing-and-Dancing

Table 1: Hard-compact routing across generation tasks.

S

M

, and

R

denote speech, music, and role tokens, respectively. The first four routes are used during training; Song/Source is activated only at inference as the held-out compositional configuration.

Figure 3: Frame-wise joint audio injection. For each generated latent frame, visual tokens act as queries and the temporally aligned routed tokens serve as joint keys and values; the cross-attention output is added residually to update the visual representation.

Singing-and-Dancing
		Reference Fidelity		Motion–Beat Alignment		Visual Quality			Human Preference
Method	Type	Subject $\uparrow$	Background $\uparrow$	BeatAlign $\uparrow$	MBCR $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	Smoothness $\uparrow$	Rhythm $\uparrow$	Visual $\uparrow$
GT	–	0.9673	0.9735	0.2795	0.2922	0.5720	0.6553	0.9858	–	–
MusicInfuser †	TA2V	–	–	0.2356	0.2337	0.5232	0.6172	0.9927	–	–
Wan-S2V	TIA2V	0.9565	0.9633	0.2640	0.1983	0.5834	0.6538	0.9860	38.67	45.33
SingDance	TIA2V	0.9682	0.9772	0.2723	0.2558	0.5846	0.6688	0.9899	61.33	54.67

Table 2: Quantitative results on SingDance-50 and Dance-100 . Human scores are preference rates (%). Best and second-best generated results are bold and underlined; GT is shown only as a reference. † MusicInfuser does not accept a reference image, so its reference-fidelity scores are not applicable.

Role	Method	LSE-C	LSE-D	BeatAlign $\uparrow$	MBCR $\uparrow$
Source	Wan-S2V	4.2809	9.4462	0.2640	0.1983
	SingDance	4.9672	8.9363	0.2723	0.2558
Listener	Wan-S2V	4.1676	9.5386	0.2480	0.2065
	SingDance	1.1768	12.4059	0.2704	0.2787

Table 3: Paired vocal-role switching on SingDance-50 . Source and listener correspond to Lip-Sync ON and OFF; LSE-C and LSE-D therefore have opposite desired directions.

Figure 4: Qualitative paired vocal-role switching on two SingDance examples. Each Lip-on/Lip-off pair shares all role-independent inputs and inference settings; only the vocal role changes.

Figure 5: Qualitative vocal-role control with the same reference image and speech audio. SingDance produces synchronized articulation for the source role and listening/reacting behavior for the listener role; videos are provided in the supplementary material.

Method	Params. $\downarrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	Subj. $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
Hallo3	14.55	5.352	9.828	0.9746	0.511	0.677
Wan-S2V	16.30	6.999	8.292	0.9812	0.580	0.698
HY-Avatar	12.99	7.210	8.494	0.9791	0.535	0.661
InfiniteTalk	18.88	8.535	7.108	0.9868	0.545	0.680
SingDance	5.63	7.995	7.551	0.9859	0.593	0.722

Table 4: Speech-driven vocal-articulation results on EMTD. Params. denotes generation-time parameters in billions; best and second-best results are bold and underlined.

Test Set	Variant	BeatAlign $\uparrow$	MBCR $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
SingDance-50	w/o $M$	0.2462	0.2386	0.5864	0.6598
	Full	0.2723	0.2558	0.5846	0.6688
Dance-100	w/o $M$	0.2853	0.2457	0.5687	0.6734
	Full	0.3002	0.2632	0.5718	0.6738

Table 5: Ablation of the routed music token on SingDance-50 and Dance-100 . Aes. and Img. denote Aesthetic and Imaging Quality; bold values indicate the better variant on each test set.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

Jul 12, 2026

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .

Ryota Kimura, Sangheon Park, Natalia Polouliakh +1

Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology

Controllable Music Generation Cross-Modal Alignment

Ask PDF

Aug 16, 2026

FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams

Music-driven dance video synthesis aims to animate a reference person according to a given music clip. The task is challenging because it requires a model to jointly learn music-to-motion correspondence, identity-preserving human animation, temporal coherence, and visually realistic video generation. We present FlowDance, a music-driven dance video generation framework that integrates explicit motion modeling with reference-preserving visual synthesis through parallel pose and RGB streams. We further introduce timestep-aware pose injection to adapt structural guidance across denoising steps and persistent identity injection to preserve the reference appearance over long video. To support this task, we further build a popularity-curated, high-resolution in-the-wild dance video dataset with synchronized music, RGB videos, 3D body motion, camera parameters, and projected 2D pose annotations. Extensive experiments show that FlowDance achieves strong performance in both dance motion generation and music-driven dance video synthesis.

Genying Li, Boda Lin, Jiachen Li +5

School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · School of Software and Microelectronics, Peking University

Audio-Video Generation Human Motion Generation Pose-Conditioned Video Generation Video Diffusion Models Identity-Preserving Video Generation

Ask PDF

Jun 22, 2026

Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics. While existing models can synthesize motion from music, they remain largely black boxes. Conversely, attempting to condition generation on both text and music frequently leads to modality collapse, where dense acoustic rhythms overwhelm sparse semantic text prompts, destroying user controllability. To resolve this spatial-temporal conflict, we propose STREAM (Structural-Temporal Rhythmic Energy-based Attention for Motion), a modality-decoupled diffusion transformer. STREAM strictly separates conditioning pathways: global text semantics dictate the kinematic structure via Adaptive Layer Normalization (AdaLN), while a novel Bimodal Energy-Based Attention Module (BEAM) routes these features to the musical beat without overwriting the semantics. We further introduce Motorica++, a newly curated dataset enriched with domain-specific dance vocabulary and frame-level semantic annotations from existing Motorica dataset. Additionally, to rigorously quantify zero-shot editability, we propose the Exchange Evaluation Protocol and Editable Dance Score (EDS). Through extensive experiments, STREAM achieves state-of-the-art alignment between motion and music while perfectly preserving choreographic semantics, positioning AI not merely as a reactive synthesizer, but as a controllable, collaborative partner for artistic direction. The source code and datasets are available at https://github.com/SeongJong-Yoo/STREAM.

Seong Jong Yoo, Siyuan Peng, Felix Gu +2

University of Maryland, College Park, MD, 20742, USA · Perception and Robotics Group

Diffusion Transformer Text-to-Motion Generation Human Motion Generation Cross-Modal Attention

Ask PDF

Abstract

Video Generation

Explore similar work

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams

Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation