SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

Tao Feng; Xu Li; Xiangyang Luo; Ming Wen; Huadai Liu; Chen Zhang; Wei Xue

cs.SDAug 17, 2026

SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

Authors: Tao Feng, Xu Li, Xiangyang Luo, Ming Wen, Huadai Liu, Chen Zhang, Wei Xue

Organizations: Kling Team, Kuaishou Technology · The Hong Kong University of Science and Technology · Tsinghua University

Abstract

Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.

Figures & tables

Figure 1: Four generation tasks supported by SingDance. The dashed box marks singing-and-dancing, the held-out Song/Source configuration generated through compositional zero-shot inference.

Figure 2: Training architecture of SingDance. The first video frame is separately encoded as a clean reference latent, while subsequent frames are temporally compressed and noised in latent space. Wav2Vec 2.0 and MuQ features, together with the vocal-role condition, are selected by hard-compact routing and supplied to frame-wise joint audio injection. The repeated block is schematic: joint audio injection is applied immediately after each of 10 selected blocks in the 30-block DiT backbone. The denoised latents are decoded into output frames. Snowflakes and flames denote frozen and trainable modules, respectively.

Audio type	Role	Routed tokens	Task
Speech	Source	$S+R_{\mathrm{src}}$	Speaking
Speech	Listener	$S+R_{\mathrm{lis}}$	Listening/Reacting
Instrumental	Listener	$R_{\mathrm{lis}}+M$	Dancing-Only
Song	Listener	$S+R_{\mathrm{lis}}+M$	Dancing-Only
Song	Source	$S+R_{\mathrm{src}}+M$	Singing-and-Dancing

Table 1: Hard-compact routing across generation tasks.

S

M

, and

R

denote speech, music, and role tokens, respectively. The first four routes are used during training; Song/Source is activated only at inference as the held-out compositional configuration.

Figure 3: Frame-wise joint audio injection. For each generated latent frame, visual tokens act as queries and the temporally aligned routed tokens serve as joint keys and values; the cross-attention output is added residually to update the visual representation.

Singing-and-Dancing
		Reference Fidelity		Motion–Beat Alignment		Visual Quality			Human Preference
Method	Type	Subject $\uparrow$	Background $\uparrow$	BeatAlign $\uparrow$	MBCR $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	Smoothness $\uparrow$	Rhythm $\uparrow$	Visual $\uparrow$
GT	–	0.9673	0.9735	0.2795	0.2922	0.5720	0.6553	0.9858	–	–
MusicInfuser †	TA2V	–	–	0.2356	0.2337	0.5232	0.6172	0.9927	–	–
Wan-S2V	TIA2V	0.9565	0.9633	0.2640	0.1983	0.5834	0.6538	0.9860	38.67	45.33
SingDance	TIA2V	0.9682	0.9772	0.2723	0.2558	0.5846	0.6688	0.9899	61.33	54.67

Table 2: Quantitative results on SingDance-50 and Dance-100 . Human scores are preference rates (%). Best and second-best generated results are bold and underlined; GT is shown only as a reference. † MusicInfuser does not accept a reference image, so its reference-fidelity scores are not applicable.

Role	Method	LSE-C	LSE-D	BeatAlign $\uparrow$	MBCR $\uparrow$
Source	Wan-S2V	4.2809	9.4462	0.2640	0.1983
	SingDance	4.9672	8.9363	0.2723	0.2558
Listener	Wan-S2V	4.1676	9.5386	0.2480	0.2065
	SingDance	1.1768	12.4059	0.2704	0.2787

Table 3: Paired vocal-role switching on SingDance-50 . Source and listener correspond to Lip-Sync ON and OFF; LSE-C and LSE-D therefore have opposite desired directions.

Figure 4: Qualitative paired vocal-role switching on two SingDance examples. Each Lip-on/Lip-off pair shares all role-independent inputs and inference settings; only the vocal role changes.

Figure 5: Qualitative vocal-role control with the same reference image and speech audio. SingDance produces synchronized articulation for the source role and listening/reacting behavior for the listener role; videos are provided in the supplementary material.

Method	Params. $\downarrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	Subj. $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
Hallo3	14.55	5.352	9.828	0.9746	0.511	0.677
Wan-S2V	16.30	6.999	8.292	0.9812	0.580	0.698
HY-Avatar	12.99	7.210	8.494	0.9791	0.535	0.661
InfiniteTalk	18.88	8.535	7.108	0.9868	0.545	0.680
SingDance	5.63	7.995	7.551	0.9859	0.593	0.722

Table 4: Speech-driven vocal-articulation results on EMTD. Params. denotes generation-time parameters in billions; best and second-best results are bold and underlined.

Test Set	Variant	BeatAlign $\uparrow$	MBCR $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
SingDance-50	w/o $M$	0.2462	0.2386	0.5864	0.6598
	Full	0.2723	0.2558	0.5846	0.6688
Dance-100	w/o $M$	0.2853	0.2457	0.5687	0.6734
	Full	0.3002	0.2632	0.5718	0.6738

Table 5: Ablation of the routed music token on SingDance-50 and Dance-100 . Aes. and Img. denote Aesthetic and Imaging Quality; bold values indicate the better variant on each test set.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Jul 12, 2026Ryota Kimura, Sangheon Park, Natalia Polouliakh +1Controllable Music Generation Cross-Modal Alignment
Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology
FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams
Aug 16, 2026Genying Li, Boda Lin, Jiachen Li +5Audio-Video Generation Human Motion Generation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · School of Software and Microelectronics, Peking University
Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation
Jun 22, 2026Seong Jong Yoo, Siyuan Peng, Felix Gu +2Diffusion Transformer Text-to-Motion Generation
University of Maryland, College Park, MD, 20742, USA · Perception and Robotics Group

Jul 12, 2026J/K move · Enter open · S save

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

Ryota Kimura, Sangheon Park, Natalia Polouliakh, Taketo Akama

Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology

Controllable Music Generation Cross-Modal Alignment

Open paper AskPDF arXiv

Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .

Abstract

Video Generation

Explore similar work

Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams

Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation