We present BARS (Beat-Adaptive Rap Synthesis), a non-autoregressive rapping voice generation model based on Flow Matching, conditioned on a beat track (instrumental backing), lyrics, and a short reference voice for zero-shot timbre control. Unlike prior autoregressive rap synthesis, where rap pace and output duration are strictly determined by the beat, BARS enables independent control over the rap tempo, allowing flexible-length generation without explicit duration prediction or phoneme-level alignment. We encode lyrics using a pretrained ByT5 encoder with a lightweight Transformer adapter trained jointly with the model, without relying on speech-pretrained text encoders, enabling natural handling of non-standard orthography (e.g., slang, abbreviations) and numerals common in rap lyrics. To address alignment instabilities, such as token skipping or repetition, under strong beat conditioning, we propose Asymmetric Scaled RoPE for cross-attention, which introduces a positional structure to facilitate sequential lyric-audio alignment learning. Furthermore, the model supports partial re-generation via span masking, enabling lyric-level editing while preserving the surrounding vocal style. Experiments demonstrate that the proposed model achieves high lyric fidelity, natural rap flow that is rhythmically coherent with the beat, and robust tempo control across a wide range of rapping styles and speeds.
Figures & tables
Figure 1: Input construction: the rap vocal, beat track, and reference voice are encoded via a shared VAE encoder. A span mask is applied to the vocal latent, and masked regions are replaced with the Flow Matching interpolant zt . The beat latent is concatenated along the channel dimension and projected to form xt , which is prepended with zref and timestep embedding temb .
Figure 2: Architecture of a single Transformer block: RoPE is applied in self-attention with position indices reset per segment. Prior to cross-attention, conditioning tokens are removed and only audio latent tokens serve as queries, with standard RoPE on the audio side and length-scaled RoPE on the lyric key side. The conditioning tokens are reintroduced before the feed-forward network.
Figure 3: Cross-attention maps (mid-layer). (a) Abrupt horizontal shifts indicate skipping, while the double-line pattern denotes repetition via redundant attention. (b) Attention follows a smooth diagonal trajectory.
Encoder
Adapter
A-RoPE
WER ↓
Del ↓
Ins ↓
Sub ↓
GT
—
5.1 %
0.7 %
0.6 %
3.8 %
ByT5
✓
✓
5.1 %
1.1 %
1.1 %
2.9 %
✓
6.0 %
1.5 %
1.2 %
3.3 %
✓
9.7 %
1.3 %
3.2 %
5.2 %
37.7 %
5.3 %
7.4 %
25.0 %
SpT5
✓
✓
5.6 %
1.1 %
1.2 %
3.3 %
Table 1: Ablation study on encoder, adapter, and A-RoPE. WER and its decomposition into deletion (Del), insertion (Ins), and substitution (Sub) rates are reported. The proposed configuration is shown in bold. GT denotes ground truth recordings. SpT5: SpeechT5.
Figure 4: Training curves comparing models with and without A-RoPE. (a) Training loss. (b) Lyrics utilization, measured as the relative loss increase when replacing ground-truth lyrics with irrelevant lyrics in cross-attention.
Speed (CPS)
w/ A-RoPE
w/o A-RoPE
WER ↓
Del ↓
Ins ↓
WER ↓
Del ↓
Ins ↓
−2σ
20.5 %
2.2 %
9.3 %
33.6 %
5.7 %
14.6 %
−1σ
14.2 %
2.6 %
5.2 %
25.5 %
5.9 %
8.3 %
μ
12.3 %
3.3 %
2.8 %
22.0 %
7.2 %
4.7 %
+1σ
13.8 %
4.1 %
2.2 %
24.4 %
9.3 %
3.9 %
+2σ
15.4 %
5.8 %
2.2 %
26.9 %
12.6 %
3.1 %
Table 2: Tempo robustness evaluation (WER/Del/Ins, %) for the proposed model and its ablation without A-RoPE. Lyrics and beat tracks are randomly mismatched across test samples. Speed bin labels denote the offset from the training mean μ , each spanning 1 σ ( μ =17.9, σ =2.9 chars/sec).
Model
Group
WavLM Sim ↑
DNSMOS OVR ↑
GT
—
0.80 ± 0.05
3.35 ± 0.21
w/ A-RoPE
match
0.80 ± 0.05
3.49 ± 0.20
mismatch
0.75 ± 0.05
3.46 ± 0.21
w/o A-RoPE
match
0.80 ± 0.05
3.46 ± 0.20
mismatch
0.76 ± 0.05
3.42 ± 0.22
Table 3: Speaker similarity (WavLM Sim) and audio quality (DNSMOS OVR) under matched and mismatched conditions (mean ± std).
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Existing singing voice synthesis systems often require predefined durations, explicit duration prediction, or time-aligned acoustic guidance, which limits their compatibility with practical composition workflows. We propose VocalRender, a score-native system that directly synthesizes singing from lyrics, pitches, symbolic note values, and tempo. It uses an interleaved lyric--note representation and an autoregressive diffusion model to generate continuous acoustic latents while predicting the output length, eliminating the need for explicit duration prediction. Trained on a 2,300-hour singing dataset, VocalRender achieves strong intelligibility, strong melody control, and high speaker similarity across both in-domain and out-of-domain benchmarks. Notably, it outperforms the strongest baseline by 0.42 points in naturalness CMOS, demonstrating the effectiveness of our proposed score-native architecture.
Yukun Chen, Tianrui Wang, Zhaoxi Mu +2
Xi’an Jiaotong University · Nanyang Technological University · Ant Group +1