Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering

Yangbin Yu; Mingyu Yang

cs.SDSep 14, 2026

Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering

Authors: Yangbin Yu, Mingyu Yang

Organizations: Work done at Tencent.

Abstract

We present Open-Qwen-Music, an open reconstruction of Qwen-Music and a fully specified research system for text-to-music generation that couples LLM-based semantic composition with diffusion-based acoustic rendering. The system comprises a 25 Hz single-codebook music tokenizer, a 3B-parameter autoregressive Music LLM, and a diffusion renderer producing 48 kHz stereo audio, following the cross-module interfaces reported by Qwen-Music. The strongest systems of this design remain closed, and prominent open music-generation projects release weights and inference code without their training corpora or end-to-end training implementations. This limits independent and controlled study of how information loss and prediction errors propagate from semantic representation through autoregressive planning to acoustic rendering. To our knowledge, Open-Qwen-Music is the first fully open release of an LLM-composition-plus-diffusion-rendering text-to-music system. Beyond model weights and inference code, the release includes the training datasets and provenance manifests, complete data-processing, annotation, training, inference, and evaluation pipelines, configurations, and pretrained weights for every learned module. Artifact manifests bind the identities of these artifacts across the complete workflow. Together, these artifacts establish a reproducible implementation of the modular architecture and provide an empirical basis for component-level analysis and future evaluation. We present the system as a transparent, executable research baseline and a starting point for the community, not as evidence of quality parity with Qwen-Music. Open-Qwen-Music is an ongoing effort, and we will continue to improve its generation quality, controllability, and robustness. All release artifacts are available at https://github.com/biang15343100-source/Open-Qwen-Music.

Figures & tables

Dataset	Audio records	Audio hours	Definition and main use
Dataset-Inventory	1,481,301	63,170.407	Broad 26-source inventory for tokenizer and Spec-VAE training.
Dataset-Annotated	240,325	9,506.162	Source-capped, annotated subset of Dataset-Inventory.
Dataset-Annotation-Filtered	44,355	2,155.176	Strict annotation-filtered subset of Dataset-Annotated: 35,484 vocal and 8,871 instrumental records.
MiniMax	25,606	1,103.057	Independent synthetic augmentation: 10,197 Chinese, 9,955 English, and 5,454 instrumental records.
Research package	62,417	2,958.571	Access-screened Dataset-Annotation-Filtered audio plus MiniMax audio; 631 additional metadata-only records are not included in the audio count.

Table 1: Dataset summary. Dataset-Annotation-Filtered is contained in Dataset-Annotated and is not counted as additional audio when both are used.

Representation	Rate	Specification	Downstream role
Semantic Tokens	25 Hz	one codebook, 32,768	Music LLM and DiT
Melody Tokens	6.25 Hz	256; unvoiced $=255$	Melody-CoT
Spectral latent	25 Hz	128 dimensions	Spec Decoder
Waveform	48 kHz	stereo	playback and evaluation

Table 2: Cross-module interfaces.

Phase	Objective	Main reconstruction choice
Pretraining	Bidirectional BestRQ	Broad pretraining with waveform masking and four-frame targets.
Domain adaptation	BestRQ	Fresh optimizer and sampler-matched feature statistics.
Causal adaptation	Causal BestRQ	Strictly causal attention; unchanged targets.
Semantic supervision	Multi-task prediction	4096-unit multilingual CTC, Mel, and Chroma losses with weights $1/1/1$ .
Discretization	Vector quantization	16-dimensional cosine VQ, data-dependent initialization, exact all-rank usage statistics, and no dead-code revival.

Table 3: Training recipe for the semantic tokenizer.

Setting	Music LLM
Initialization	Qwen2.5-Omni thinker
Learning rate / schedule	$1\times 10^{-4}$ / constant
Parallelism	64 H20 GPUs, DDP, BF16
Context	4,096 tokens; $\leq$ 30 s / 750 Semantic frames

Table 4: Optimization recipe for the autoregressive planner.

Component	Objective	Main reconstruction choice
Spec-VAE	Reconstruction	Full VAE on 48 kHz stereo; global batch 256.
Spec-VAE	Decoder fine-tuning	Encoder frozen; STFT-domain adversarial training at batch 64.
DiT	Conditional flow matching	24-block, width-1536 backbone; exponential-moving-average weights.
Refiner	Band-aware residual correction	Frozen VAE with fresh optimizers; mid-band magnitude and phase correction.

Table 5: Training recipe for the acoustic renderer.

Signal	Result	Note
Lyric LCS recall	0.889	precision 0.879; CI $[0.877,0.902]$
Vocal gender	88.3%	CI $[85.9,90.3]$
Genre	75.2%	GTZAN subset
Mood	76.9%	arousal after dead-zone mapping
Confidence–error correlation	Spearman $-0.764$	two-transcription confidence

Table 6: Automatic-annotation audit. Each row retains its own task-specific denominator and is not combined into an overall annotation score.

Evaluation panel	Examples	$\Delta$ PER (pp)	$\Delta$ Mel	$\Delta$ Chroma
Common music holdout	2,372	2.352	—	0.00124
Exposure panel	256	2.153	0.0306	0.00159
Real-recording panel	101	2.879	0.0267	0.00143

Table 7: Quantized-minus-continuous (

Q-P

) prediction-error differences. PER differences are percentage points. Mel was not measured on the common music holdout.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Records	Hours
AI-Music Deduplicated ( AI-Music, 2026 )	508,557	25,141.23
JamendoMaxCaps ( Roy et al., 2025 )	261,739	13,683.13
Muse ( Jiang et al., 2026 )	117,067	7,589.09
Udio Dataset ( Blanchon, 2024 )	134,329	3,914.82
FMA Full ( Defferrard et al., 2017 )	65,282	3,607.52
GlobalDISCO ( Solak et al., 2025 )	73,024	3,340.90

Appendix

Table 8: The 26 retained sources in Dataset-Inventory. Hours are rounded to two decimals. Each row cites the primary paper, the ingested release page, or both. Derivative and mirror caveats are stated below.

Branch	Required evidence	Main numeric rules
Vocal	Melody-CoT eligible; at least two sections; no under-transcribed, dropped, or hallucinated lyric windows; English, Chinese, or bilingual.	Lyric confidence $\geq 0.905$ ; lyric coverage $\geq 0.15$ ; genre $\geq 0.80$ ; instrument $\geq 0.75$ ; binary vocal gender $\geq 0.89$ .
Instrumental	Non-synthetic; no usable lyrics or vocal tags; fewer than three text units.	Instrumental-gender confidence $\geq 0.90$ ; vocal ratio $<0.15$ ; vocal-stem-to-mix ratio $<-15$ dB.

Appendix

Table 9: Main Dataset-Annotation-Filtered eligibility rules. The implementation also applies silence, timeline, and annotation-consistency checks.

	Stage	Function
1	Index	Ingest source manifests or file trees, assign stable sample identities, and record source and audio metadata.
2	Source separation	Extract the vocal stem and vocal-activity evidence used by the lyric and voice-analysis branches.
3	Structure analysis	Detect musical-section boundaries and normalize predicted section labels to the shared section taxonomy.
4	Vocal-stem transcription	Produce the primary lyric transcription from the separated vocal stem.
5	Mixture transcription	Independently transcribe the original mixture for cross-checking the vocal-stem result.
6	Lyric fusion	Reconcile the two transcription streams with available source text, determine language, and produce the selected lyric sequence with quality measurements.

Appendix

Table 10: Stages of the released annotation pipeline. The order follows the dependency graph; stages whose prerequisites are satisfied may run in parallel.

Field group	Contents	Role
Identity	Schema version, stable sample identifier, and source-dataset label.	Joins annotations to the corresponding corpus item while preserving the record contract.
Audio	Audio reference, duration in seconds, sample rate, and channel count.	Defines the media item and its technical properties.
Language	Resolved lyric language, when available.	Supports language-aware normalization, filtering, and corpus sampling.
Description	Free-form musical description produced by the tagging branch.	Provides text conditioning beyond the controlled tag vocabulary.
Musical tags	Genre, mood, instrument, vocal gender, and vocal timbre; low-confidence attributes are omitted.	Supplies independently controllable prompt attributes without treating missing evidence as a negative label.
Structured lyrics	Lyrics serialized in section order with section labels.	Provides the text and musical-form representation consumed by downstream sequence construction.

Appendix

Table 11: Field groups in each exported annotation record.

Phase	Objective and training unit	Updates	Batch/GPU	Global batch	Peak LR / warmup
Masked pretraining	Bidirectional BestRQ on deterministic 30-s crops	100,000	2	128	$2\!\times\!10^{-4}$ / 10,000
Music-domain adaptation	BestRQ on deterministic 30-s crops from the annotated domain	20,000	4	256	$2\!\times\!10^{-5}$ / 500
Causal adaptation	Causal BestRQ with unchanged acoustic targets	20,000	4	256	$2\!\times\!10^{-5}$ / 500
Supervised-head warmup	Frozen backbone; CTC head on complete tracks up to 300 s	2,000	1	64	$1\!\times\!10^{-3}$ / 100
Joint semantic supervision	CTC, Mel, and Chroma losses with weights $1/1/1$	40,000	1	64	$5\!\times\!10^{-5}$ / 3,000
Discrete codebook training	CTC/Mel/Chroma/VQ weights $0.5/1/1/1$	25,000	1	64	$3\!\times\!10^{-5}$ / 2,000

Appendix

Table 12: Sequential optimization phases for the semantic tokenizer. Batch/GPU is the per-process batch size; warmup and updates count optimizer steps.

Stage	Initialization and data progression	Stage updates	Cumulative updates	Global batch
Stage 1	Qwen2.5-Omni-3B initialization; beginning of the corpus stream	5,000	5,000	128
Stage 2	Stage-1 model initialization; continuation of the corpus stream	5,000	10,000	128

Appendix

Table 13: Two-stage Music LLM optimization. Cumulative updates count the uninterrupted sequence of consumed training batches.

Module or stage	Objective	Updates	GPUs	Batch/GPU	Accum.	Global batch	Precision
VAE Stage 1	Full-model reconstruction and KL training	90,000	64	4	1	256	FP32
VAE Stage 2	Decoder-only adversarial refinement	5,000	64	1	1	64	FP32
Renderer	Conditional flow matching	29,000	64	1	4	256	BF16
Refiner	Band-aware residual correction	1,250	64	1	1	64	FP32

Appendix

Table 14: Distributed training schedules for the acoustic modules.

Setting	Value
Music LLM sequence layout	Direct text-to-semantic
Sampling	temperature 0.7; top- $p$ 0.98; top- $k$ 0
Semantic repetition control	penalty 0.1 over a 64-token window
Acoustic solver	Euler, 8 steps
Classifier-free guidance	1.0
Loudness condition	$-15.4335$ LUFS

Appendix

Table 15: Inference settings used for text-to-music generation.

Text-to-Music Generation

Text-to-music generation turns natural-language descriptions into audio or symbolic scores, with control over style, instruments, mood, lyrics, and structure.

See all 48 papers

Jul 13, 2026

Qwen-Music Technical Report

We introduce Qwen-Music, a music generation model that produces high-fidelity songs with complete vocals. It supports text-to-music generation from descriptions, lyrics, and musical attributes, and cover song generation with different styles and vocal characteristics. Qwen-Music comprises three components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. The tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information. The LLM performs autoregressive modeling with a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving musicality, structural coherence, and reference-melody preservation. The renderer enriches discrete semantic tokens with acoustic details to produce high-fidelity stereo waveforms. We train the LLM using a quality-aware pre-training curriculum followed by progressive post-training with supervised initialization, offline DPO, and online GSPO to improve musicality and instruction following. On 600 evaluation inputs, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover generation, Qwen-Music preserves reference melodies more accurately than Suno V5.5, Suno V5, and MiniMax Cover on the AI-generated reference set, and outperforms MiniMax Cover on most metrics on the real-world popular-song reference set.

Jin Xu, Kangdi Wang, Ruibin Yuan +24

Qwen Team

Text-to-Music Generation Controllable Music Generation Music Generation

Ask PDF

Sep 11, 2026

StepAudio 3 Music Technical Report

We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.

Chengli Feng, Zhiyue Wu, Jiahao Song +10

StepFun · The Chinese University of Hong Kong · University of California San Diego +1

Text-to-Music Generation Music Generation Controllable Music Generation Audio Diffusion Models

Ask PDF

May 20, 2026

Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches

Text-to-music generation has advanced rapidly, with modern autoregressive and diffusion-based models producing convincing music from natural-language prompts. However, much of this progress relies on large-scale training data and external pretraining, making it difficult to isolate which design choices remain effective when data and pretraining are controlled. We study this setting using a Diffusion Transformer backbone with lyric and timbre conditioning, adapted to an instrumental-only text-to-music task in which the auxiliary lyric and timbre branches receive only degenerate conditioning signals. Through controlled ablations, we find that models retrained without these branches score lower across AudioBox aesthetics, LLM-as-judge, and human MOS, and that reinvesting the saved parameters as additional DiT depth recovers only marginally. This suggests the auxiliary branches may act as training-time architectural anchors whose contribution goes beyond their explicit conditioning content. We validate the same model through comparisons with external instrumental baselines and through our submission to the ICME 2026 Academic Text-to-Music (ATTM) Grand Challenge, where our Performance submission ranked first under both the objective metrics and the subsequent organizer-administered MOS over 35 raters, attaining the highest overall MOS across all challenge submissions, while our Efficiency submission was a finalist that tied for second under the objective metrics.

Junyoung Koh

Department of Artificial Intelligence, Yonsei University MAAP KRAFTON Seoul, Republic of Korea

Text-to-Music Generation Diffusion Transformer Music Generation Conditional Generative Modeling

Ask PDF

Abstract

Text-to-Music Generation

Explore similar work

Qwen-Music Technical Report

StepAudio 3 Music Technical Report

Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches