LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation

Qi Wang; Zhexu Shen; Meng Chen; Guoxin Yu; Chaoxu Pang; Weifeng Zhao; Wenjiang Zhou

cs.SDApr 13, 2026

LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation

Authors: Qi Wang, Zhexu Shen, Meng Chen, Guoxin Yu, Chaoxu Pang, Weifeng Zhao, Wenjiang Zhou

Organizations: State Key Lab of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (CAS) University of Chinese Academy of Sciences Beijing, China · Lyra Lab, Tencent Music Entertainment Shenzhen, China · Pengcheng Laboratory Shenzhen, China

Abstract

Vocal-to-accompaniment (V2A) generation, which aims to transform a raw vocal recording into a fully arranged accompaniment, inherently requires jointly addressing an accompaniment trilemma: preserving acoustic authenticity, maintaining global coherence with the vocal track, and producing dynamic orchestration across a full song. Existing open-source approaches typically make compromises among these goals. Continuous-latent generation models can capture long musical spans but often struggle to preserve fine-grained acoustic detail. In contrast, discrete autoregressive models retain local fidelity but suffer from unidirectional generation and error accumulation in extended contexts. We present LaDA-Band, an end-to-end framework that introduces Discrete Masked Diffusion to the V2A task. Our approach formulates V2A generation as Discrete Masked Diffusion, i.e., a global, non-autoregressive denoising formulation that combines the representational advantages of discrete audio codec tokens with full-sequence bidirectional context modeling. This design improves long-range structural consistency and temporal synchronization while preserving crisp acoustic details. Built on this formulation, LaDA-Band further introduces a dual-track prefix-conditioning architecture, an auxiliary replaced-token detection objective for weakly anchored accompaniment regions, and a two-stage progressive curriculum to scale Discrete Masked Diffusion to full-song vocal-to-accompaniment generation. Extensive experiments on both academic and real-world benchmarks show that LaDA-Band consistently improves acoustic authenticity, global coherence, and dynamic orchestration over existing baselines, while maintaining strong performance even without auxiliary reference audio. Codes and audio samples are available at https://github.com/Duoluoluos/TME-LaDA-Band .

Figures & tables

Figure 1. LaDA-Band enables true zero-shot generation (i.e., generating full arrangements from only vocals and text, without needing reference audio).

Figure 2. Overview of the LaDA-Band framework , illustrating (a) the training of LLaMA backbone on dual-track representations, and (b) the inference of full-song accompaniments via a masked diffusion process.

Short-form (10s) Generation Baselines (Chunked / Truncated)
Method	Formulation	Acoustic Auth.			Global Coherence			Dynamic Orch.			RTF $\downarrow$
Method	Formulation	FAD $\downarrow$	Clar. $\uparrow$	Nat. $\uparrow$	Onset F1 $\uparrow$	HPCP Sim. $\uparrow$	Coh. $\uparrow$	CLaMP $\uparrow$	Mus. $\uparrow$	Mem. $\uparrow$	RTF $\downarrow$
SongEdit-3.0B	Disc. AR	31.40	3.42	3.35	41.2	86.4	2.85	4.10	3.12	3.05	1.42
AnyAcc-3.7B	Cont. Lat.	48.20	3.15	3.18	46.5	88.1	3.02	4.55	3.20	3.18	0.48
MCLite-3.7B	Cont. Lat.	52.10	3.10	3.12	51.3	89.5	3.15	4.85	3.25	3.22	0.39
Long-form (> 4min) Full-Song Generation Baselines

Table 1. Performance comparison of LaDA-Band against SOTA open-source baselines under their representative full conditions on Suno70k (academic). RTF denotes real-time factor (

\downarrow

). All inference was performed on a single 40GB A100 GPU

Baselines
Method	Conditioning	Suno70k Academic				IHP Realworld
Method	Conditioning	CLaMP $\uparrow$	HPCP Sim. $\uparrow$	Onset F1 $\uparrow$	FAD $\downarrow$	CLaMP $\uparrow$	HPCP Sim. $\uparrow$	Onset F1 $\uparrow$	FAD $\downarrow$
ACE-Step-3.7B	Text + Ref.	5.20	93.2	60.4	46.5	6.42	93.1	56.2	41.9
ACE-Step-3.7B	Text (Zero-shot)	4.91	88.1	55.4	49.2	6.61	92.5	55.3	41.9
SongEcho-3.6B	Text + Ref. + Lyrics	10.2	94.5	67.0	45.9	6.17	95.8	43.3	46.9
SongEcho-3.6B	Text + Lyrics	10.2	90.2	59.9	56.9	6.38	91.3	49.2	46.4

Table 2. Zero-shot robustness, and generalization from academic to real-world benchmarks. We evaluate representative baselines and LaDA-Band under reduced-conditioning settings on Suno70k (academic) and IHP (real-world) to isolate robustness beyond full-strength system comparison.

Figure 3. Comparison of LaDA-Band and continuous-latent baselines on the five SongEval dimensions under the restricted Zero-shot (Text-Only) conditioning scenario .

Setting	Onset F1	FAD	CLaMP
Full LaDA-Band	82.0	22.70	14.83
w/o Stage 2 (Stage 1 only)	78.4	25.87	11.86
w/o Discrete (Cont. latents)	71.4	38.65	10.20
w/o NAR (AR decoding)	14.2	26.1	2.12
w/o Dual-track (Single-stream)	28.7	29.35	4.55
w/o RTD Loss	76.5	25.21	16.71

Table 3. Ablation study of the proposed formulation and curriculum. We replace core components and training stages to validate the gains of LaDA-Band .

Figure 4. Spectrogram comparison of modeling formulations for long-form accompaniment generation

Method	AQ	MA	ON	Overall
ACE-Step-3.7B	$2.217_{\pm 0.185}$	$2.283_{\pm 0.198}$	$2.017_{\pm 0.175}$	$2.172_{\pm 0.155}$
SongEcho-3.6B	$2.908_{\pm 0.172}$	$3.354_{\pm 0.187}$	$3.400_{\pm 0.169}$	$3.221_{\pm 0.153}$
LaDA-Band-3.0B	3.338 ±0.184	3.367 ±0.187	3.483 ±0.179	3.396 ±0.154

Table 4. Subjective MOS Results on Various Style Samples. All models adopt representative full conditioning. The 95% confidence interval for the mean.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5. Data Filtering and Augmentation Flowchart

Filename	Song ID	Audio Quality	Harmony & Alignment	Arrangement
0fc13f0f-jhl6ca.mp3	0fc13f0f
0fc13f0f-plvvpo.mp3	0fc13f0f
0fc13f0f-wn5w6r.mp3	0fc13f0f

Appendix

Table 5. Audio File Metadata for Double-Blind Supervisor Review

Method	Encoder	Backbone	Decoder
SongEditor	MuCodec encoder	Autoregressive LLaMA 3.1-1B	MuCodec decoder
ACE-Step 1.5	Music-DCAE encoder frozen mT5-base text encoder SongGen-style lyric encoder PLR-OSNet-style speaker encoder	Linear DiT	Music-DCAE decoder universal music vocoder
SongEcho	Music-DCAE encoder frozen mT5-base text encoder SongGen-style lyric encoder PLR-OSNet-style speaker encoder RVMPE pitch extractor melody encoder	ACE-Step Linear DiT with IA-EiLM	Music-DCAE decoder universal music vocoder
MuseControlLite	Music-DCAE encoder frozen mT5-base text encoder SongGen-style lyric encoder PLR-OSNet-style speaker encoder lightweight control encoder	ACE-Step Linear DiT with decoupled cross-attention adapters	Music-DCAE decoder universal music vocoder

Appendix

Table 6. Architectural summary of reproduced baselines. For SongEcho and MuseControlLite, we report the architecture under our ACE-Step-based reproduction protocol.

Figure 6. General-purpose cover generation is not directly suitable for V2A accompaniment generation. When provided with a 40-second vocal input, the system expands it into much longer full-song accompaniments (3:31 and 3:56), instead of preserving the temporal extent of the source vocal. This creates a practical gap between cover composition and directly usable vocal-aligned accompaniment generation.

Figure 7. Effect of sampling steps on generation RTF and SongEval overall under the cosine schedule. Higher SongEval overall and lower generation RTF indicate better performance. The evaluation samples are selected from blind listening data.

Schedule	Generation RTF	SongEval Overall
cosine	0.3180	4.2303
linear	0.3214	4.1548
power	0.3221	4.2030

Appendix

Table 7. Comparison of different sampling schedules in terms of generation RTF and SongEval overall for LaDA-Band-3.0B. The evaluation samples are selected from blind listening data.

Figure 8. Style distribution of 56 negative cases on Suno70k with mean MOS below 3.0. Funk/jazz/blues accounts for the majority of the low-MOS cases.

Music Accompaniment Generation

See all 12 papers

HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation
Apr 10, 2026Jian Zhu, Jianwei Cui, Yunlong Xue +4Controllable Music Generation Music Generation
Zhejiang Lab · University of Science and Technology of China · Zhejiang International Studies University
LiveBand: Live Accompaniment Generation in the Audio Domain
Jun 2, 2026Marco Pasini, Javier Nistal, Ben Hayes +3Music Generation Autoregressive Generation
Sony Computer Science Laboratories, Paris, France · Johannes Kepler University, Linz, Austria · Queen Mary University of London, UK
Decoupled Latent Flow Matching for Few-Step Joint Vocal-Accompaniment Separation
Aug 31, 2026Lishi Zuo, Youzhi Tu, Lu Yi +4Flow Matching Flow-Based Generative Modeling
Dept. of Electrical and Electronic Engineering, Hong Kong Polytechnic University, Hong Kong SAR, China

Apr 10, 2026J/K move · Enter open · S save

HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation

Jian Zhu, Jianwei Cui, Yunlong Xue, Shihao Chen+3

Zhejiang Lab · University of Science and Technology of China · Zhejiang International Studies University

Controllable Music Generation Music Generation Autoregressive Generation Conditional Generative Modeling Music Accompaniment Generation

Open paper AskPDF arXiv

Music accompaniment generation aims to automatically produce instrumental accompaniments that are rhythmically, harmonically, and timbrally coherent with a given vocal input, with broad applications in personalized music creation, arrangement assistance, and music education. Existing approaches, primarily operating in the symbolic domain or relying on single-stage audio generation frameworks, commonly suffer from insufficient high-level semantic structure modeling, limited acoustic detail reconstruction, and weak conditional controllability. To address these limitations, this paper proposes HAFM, a Hierarchical Autoregressive Foundation Model for vocal-conditioned music accompaniment generation. The model employs a dual-rate tokenization strategy in which

50

Hz HuBERT semantic tokens capture high-level musical structure and

75

Hz EnCodec acoustic tokens encode fine-grained acoustic content, enabling explicit disentanglement of semantic and acoustic representations. Building on this foundation, a three-stage cascaded generation framework is designed to progressively generate semantic tokens, coarse acoustic tokens, and fine acoustic tokens, refining the accompaniment from global structure to local detail. . Objective evaluation on the MUSDB18 dataset demonstrates that the full three-stage model achieves a Fr{é}chet Audio Distance (FAD) score of 1.71, representing an 18.6% relative improvement over the two-stage baseline (FAD = 2.10). Subjective listening tests show that the generated accompaniments achieve a 51.5% preference rate against ground-truth accompaniments in head-to-head comparisons, and substantially outperform the random baseline in terms of rhythmic alignment, harmonic compatibility, and overall musical coherence. The source code and demo are available at https://github.com/HackerHyper/HAFM.git.

Abstract

Music Accompaniment Generation

Explore similar work

HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation

LiveBand: Live Accompaniment Generation in the Audio Domain

Decoupled Latent Flow Matching for Few-Step Joint Vocal-Accompaniment Separation