State-of-the-art models for audio-driven digital human generation have achieved photo-realistic results in talking-head synthesis. However, extending these models to singing-head synthesis remains challenging due to a significant Domain Gap: singing requires more exaggerated expressions, vivid jaw openings, and precise rhythmic synchronization. Current models, primarily trained on speech datasets, often struggle with "rhythmic drift" and constrained dynamics. To address this, we introduce Hi-Singers, the first large-scale, high-quality, in-the-wild video dataset specifically tailored for singing head synthesis. Hi-Singers undergoes a rigorous automated and manual filtering pipeline, ensuring strict thematic adherence, high-resolution rendering, and stable motion, resulting in 29,608 video segments totaling approximately 170 hours. We further establish a dedicated evaluation benchmark balanced across linguistic and musical styles. Extensive experiments across diverse architectures, including 3D-coefficient and diffusion-based models, demonstrate that Hi-Singers consistently and significantly improves performance across all dimensions. Specifically, it enables models to achieve superior visual realism, enhanced lip-sync consistency, and more precise rhythmic dynamics, effectively bridging the domain gap and setting a new performance standard for the singing synthesis task. The dataset is available at https://huggingface.co/datasets/CharlesZhang-USTC/Hi-Singers
Figures & tables
Figure 1. The framework diagram of the data processing workflow illustrates the entire process from raw internet data to the final dataset. This process can be divided into the following three stages: 1) Data collection and preprocessing 2) Automated filtering 3) Manual filtering.
Figure 2. This Sankey diagram illustrates the flow of data during the processing stages. Specifically, it shows the data filtered out and retained at each step of the filtering process.
Figure 3. This diagram displays sample video frames representing the primary music genres within the dataset (pop, classical, hip-hop and rock).
Dataset
Modality / Task
Setting
Res.
Scale
RAVDESS ( Livingstone and Russo, 2018 )
Emotion / Speech
Lab
720p
∼ 1 hr
MusicFace ( Liu et al., 2024 )
3D Mesh / Anim.
Studio
-
∼ 40 hrs
Song2Face ( Iwase et al., 2020 )
3D Animation
Lab
-
∼ 2 hrs
SingingHead ( Gao et al., 2025 )
4D Scan / 3D Anim.
Lab
1024p
∼ 27 hrs
ChorusHead ( Xing et al., 2025 )
3D Head Anim.
In-the-wild
-
∼ 8 hrs
RapVerse ( Chen et al., 2025a )
Body Motion / 3D
In-the-wild
-
∼ 27 hrs
Table 1. Comparison of existing singing-related datasets. Most prior works focus on 3D animation or small-scale lab environments, leaving a gap for large-scale 2D video generation.
Figure 4. This chart presents statistical data on the distribution of languages and music genres within the dataset. It is evident that the dataset contains roughly equal numbers of English and Chinese language entries; the music genres are more diverse, with pop music dominating as the predominant style.
Figure 5. The reference coordinate system for facial Euler angles (yaw, pitch, and roll) is illustrated in the leftmost diagram. The three scenarios requiring filtering are depicted in the subsequent diagrams, each representing excessive angular deviations relative to a face perfectly aligned with the camera within the coordinate system.
Category / Metrics
Calculation
Approach
1. Head Region Processing
Face Height
Key Points
TBF
Lip-sync Alignment
SyncNet Score
TBF
2. Motion Characteristics Filtering
Facial Movement
Mean Displacement
TBF
Facial Rotation
Face Eular Angle
TBF
Table 2. Tables for the filtering stage, metrics, calculation, and filtering approach (TBF or SBF)
Method
FID ↓
FVD ↓
Sync-C ↑
BAS ↑
MOS ↑
Additional SOTA Baselines (Domain Gap Evaluation)
SadTalker ( Zhang et al., 2023 ) (Orig.)
38.54
245.12
6.85
0.052
3.12
V-Express ( Wang et al., 2024 ) (Orig.)
25.68
192.45
6.92
0.076
3.45
Main Results and Ablations
Hallo (Original Weights)
23.21
188.34
7.58
0.139
3.85
Hallo (CelebV-HQ ( Wang et al., 2022 ) )
42.23
385.67
7.12
0.111
3.63
Table 3. Overall performance comparison on singing head generation. Hi-Singers achieves SOTA results across all dimensions. ‘Orig.’ denotes official weights trained on generic talking datasets.
Training Dataset
Metric
Language
Music Style
Chinese
English
Pop
Classical
Rock
Hip-hop
Other
Hallo (Original)
FID ↓
23.30
23.12
23.10
23.25
23.18
23.30
23.22
FVD ↓
190.18
186.50
187.20
189.50
187.80
188.50
188.70
Sync-C ↑
7.50
7.66
7.54
7.55
7.40
7.85
7.56
Sync-D ↓
7.98
7.58
7.90
7.85
8.02
7.25
7.88
BAS ↑
0.136
0.142
0.140
0.135
0.138
0.145
0.137
Table 4. Detailed Cross-Domain Analysis. We analyze the performance of Hallo (Original) and Hallo (Hi-Singers) across different categories. The averages of sub-categories are fitted to the overall performance results. BAS quantifies rhythmic synchronization.
Figure 6. Qualitative Comparison. Top: Baseline trained on generic datasets shows constrained motion and blur. Bottom: Ours trained on Hi-Singers displays vivid jaw openings and sharpness.
Existing singing voice synthesis systems often require predefined durations, explicit duration prediction, or time-aligned acoustic guidance, which limits their compatibility with practical composition workflows. We propose VocalRender, a score-native system that directly synthesizes singing from lyrics, pitches, symbolic note values, and tempo. It uses an interleaved lyric--note representation and an autoregressive diffusion model to generate continuous acoustic latents while predicting the output length, eliminating the need for explicit duration prediction. Trained on a 2,300-hour singing dataset, VocalRender achieves strong intelligibility, strong melody control, and high speaker similarity across both in-domain and out-of-domain benchmarks. Notably, it outperforms the strongest baseline by 0.42 points in naturalness CMOS, demonstrating the effectiveness of our proposed score-native architecture.
Yukun Chen, Tianrui Wang, Zhaoxi Mu +2
Xi’an Jiaotong University · Nanyang Technological University · Ant Group +1
Singing voice synthesis (SVS) has progressed rapidly, yet its ability to generalize across diverse musical genres remains underexplored. Existing benchmarks are heavily biased toward pop music, limiting systematic analysis of genre-dependent behavior. We introduce MMGenre, a benchmark for multi-genre SVS diagnosis, supported by an automatic pipeline for constructing genre-aligned music scores. MMGenre spans 10 major genres and 26 subgenres, enabling comprehensive analysis of genre-aware synthesis. Extensive evaluation of representative SVS models reveals limited genre discrimination: synthesized vocals across genres exhibit highly similar acoustic characteristics and weak separability. While zero-shot genre adaptation yields only marginal improvements, lightweight genre-specific continued training leads to substantial gains. MMGenre provides a standardized framework for multi-genre SVS evaluation and exposes critical challenges in achieving genre-aware singing voice synthesis.
Wenhao Feng, Yuxun Tang, Jiatong Shi +1
AIM3 Lab, Renmin University of China, China · Carnegie Mellon University, United States
Human speech conveys expressiveness beyond linguistic content, including personality, mood, or performance elements, such as a comforting tone or humming a song, which we formalize as role-playing and singing. We present VITA-QinYu, the first expressive end-to-end (E2E) spoken language model (SLM) that goes beyond natural conversation to support both role-playing and singing generation. VITA-QinYu adopts a hybrid speech-text paradigm that extends interleaved text-audio modeling with multi-codebook audio tokens, a design enabling richer paralinguistic representation while preserving a clear separation between modalities to avoid interference. We further develop a comprehensive data generation pipeline to synthesize a total of 15.8K hours of natural conversation, role-playing, and singing data for training. VITA-QinYu demonstrates superior expressiveness, outperforming peer SLMs by 7 percentage points on objective role-playing benchmarks, and surpassing peer models by 0.13 points on a 5-point MOS scale for singing. Simultaneously, it achieves state-of-the-art conversational accuracy and fluency, exceeding prior SLMs by 1.38 and 4.98 percentage points on the C3 and URO benchmarks, respectively. We open-source our code and models and provide an easy-to-use demo with full-stack support for streaming and full-duplex interaction.