We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.
Figures & tables
Figure 1: Relative performance across MuQ-MuLan, AudioBox-Aesthetics, and SongBench. Outer arcs identify the three benchmark families. Scores are divided by the best observed score per metric (1.0), not a theoretical maximum; the radial axis is truncated to 0.60–1.00. Purple denotes StepAudio 3 Music. Compare systems within each metric; polygon area is not an aggregate score. Curves provide a visual approximation; exact scores are in table 3 .
Figure 2: Overview of StepAudio 3 Music. Lyrics, a text prompt, and optional task-specific references are serialized for a trainable Mixture-of-Experts autoregressive model. In ABC-CoT mode, the model first produces an explicit arrangement plan and then predicts a 50-Hz sequence from a 65536-entry single codebook. A separately trained renderer generates 50-Hz StepAudio VAE latents with a flow-matching DiT and decodes them into 48-kHz waveform audio. The renderer is held fixed during autoregressive-model training.
Figure 3: Discrete bottleneck designs considered for the music tokenizer. The controlled comparison uses a common 25-Hz frame rate for all three variants. (a) Single-codebook VQ produces one semantic token q1 per frame. (b) Semantic RVQ quantizes residuals within the same semantically supervised latent space; the aggregate embedding ∑iei(qi) is processed by the upper encoder under the shared multi-task objectives. (c) Acoustic RVQ freezes the semantic tokenizer and quantizes the residual between an acoustic feature and the aligned semantic feature. Teal elements denote semantic representations and tokens, green denotes acoustic features, and blue denotes acoustic residual tokens. A separate DiT detokenizer is trained for the final comparison after each tokenizer is frozen and is omitted from the diagram.
Figure 4: Music data cleaning and structured annotation pipeline. Raw audio is filtered for bandwidth and content quality, separated into vocal and accompaniment stems, routed to language-appropriate transcription systems, and aligned with predicted song sections to produce structured training records. Dashed branches indicate rejection or auxiliary data paths.
Figure 5: Three-stage training pipeline for the StepAudio Music Tokenizer. Stage 1 adapts BEST-RQ pretraining with bidirectional Conformer blocks. Contiguous waveform spans are replaced with Gaussian noise before Mel extraction, and the BEST-RQ loss is evaluated only at the corresponding positions in the aligned 50-Hz sequence. Stage 2 introduces full-song CTC, Mel, and Chroma supervision. Stage 3 inserts a single-codebook VQ bottleneck with 65536 entries between the lower and upper 12-layer Conformer stacks, producing one discrete music token per frame at 50 Hz. Each stage is initialized from the checkpoint produced by the preceding stage.
Stage
Tokens
Context
Batch
LR schedule
Large-scale Pretraining
600B
16,384
768
2.45×10−5 , constant
Multi-task Mid-Pretraining
500B
32,768
512
2.45×10−5→2.0×10−5 , cosine
High-quality Annealing
80B
32,768
512
warm-up to 2.0×10−5 , then cosine to 2.0×10−6
Table 1: LLM pretraining schedule for StepAudio 3 Music. Token counts refer to training consumption; Stage 3 includes high-quality replay.
Subset
Tokens
Principal conditioning
Lyrics-to-music
6B
Lyrics and text prompt
Instrumental generation
2B
text prompt
Lyrics-to-music with ABC-CoT
4B
Lyrics, text prompt, and arrangement plan
Instrumental generation with ABC-CoT
2B
text prompt and arrangement plan
Cover-song generation
4B
Reference song
Vocal-to-mix generation
2B
Dry vocal track
Table 2: Composition of the supervised fine-tuning corpus. The listed subsets collectively contain approximately 20B tokens. Token counts are approximate.
Table 3: Objective evaluation on common benchmark subsets. Caption adherence and AudioBox-Aesthetics are reported on 339 lyrics-to-song conditions; SongBench is reported on 316 vocal conditions. Values are means. Bold and underlining indicate the best and second-best scores, respectively, for metrics where higher is better. Production Complexity is descriptive and is not ranked. The dashed line separates StepAudio 3 Music from the comparison systems.
Figure 6: Artificial Analysis Music Arena Vocals leaderboard. Supplied preliminary benchmark results based on blind preference votes. StepAudio 3 Music obtains a Quality Elo of 1105 through direct evaluation, without any adaptation to AA prompts.
Model
Latent rate
MCD ↓
MS-Mel-L1 ↓
MS-STFT-L1 ↓
SI-SNR ↑
SDR ↑
UTMOS ↑
Direct VAE reconstruction
StepAudio VAE (25 Hz)
25 Hz
4.79±1.31
0.96±0.12
0.51±0.31
4.21±4.01
5.54±2.96
–
StepAudio VAE (50 Hz)
50 Hz
3.59±1.32
0.88±0.11
0.42±0.26
7.68±4.23
8.57±3.53
–
Token-conditioned detokenization with the StepAudio VAE (50 Hz)
0.9B DiT
50 Hz
5.48±1.12
1.32±0.27
0.24±0.07
−19.49±9.06
−2.97±1.16
3.45±0.60
4B DiT
50 Hz
6.72±0.90
1.63±0.31
0.27±0.11
−23.27±12.23
−2.47±1.08
3.07±0.45
Table 4: Ablations of the music detokenizer. Results are reported as mean ± standard deviation over 54 audio segments. Lower MCD, MS-Mel-L1, and MS-STFT-L1 are better; higher SI-SNR, SDR, and UTMOS are better. Direct VAE reconstruction and token-conditioned detokenization are reported as separate experiment groups; UTMOS is reported only for the latter.
Configuration
SB-Musicality ↑
SB-Mean ↑
CE ↑
PQ ↑
MuQ-MuLan ↑
SFT
5.6395
6.5445
7.6259
8.2700
0.4157
+ DPO (final)
5.7465
6.6438
7.7086
8.3868
0.4465
Table 5: Effect of DPO after supervised fine-tuning. SB denotes SongBench, and SB-Mean is the mean over its seven dimensions. CE and PQ are the Content Enjoyment and Production Quality axes of AudioBox-Aesthetics. Higher values are better for all reported metrics.
Dimension
Direct
ABC-CoT
ABC-CoT + LLM Editing
Melody
6.5394
6.6054
6.6454
Arrangement
6.9122
6.9320
7.0211
Vocal
7.0172
7.0256
7.0282
Instrumental
7.0086
7.0122
7.0396
Mixing
6.7284
6.7698
6.8483
Structure
6.5544
6.5868
6.6411
Table 6: SongBench results for direct generation, ABC-CoT planning, and LLM-edited ABC-CoT planning using the DPO model. Scores are reported on a 1–10 scale; higher is better. Mean is the arithmetic average over the seven dimensions. Bold indicates the highest value in each row.
Text-to-music generation turns natural-language descriptions into audio or symbolic scores, with control over style, instruments, mood, lyrics, and structure.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16×2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Symbolic models make melody, harmony, rhythm, and form explicit but typically stop before a finished recording; audio models produce complete songs while leaving composition implicit. We introduce YuE2, which unifies symbolic and audio music generation at frontier quality through symbolic planning. A single AR-NAR Mixture-of-Transformers (MoT) first writes a readable score specifying melody and harmony, expands it into semantic music tokens, and realizes it as full-song audio. In comparisons using the same checkpoint, experts prefer symbolic planning for overall quality and musicality, with 49.3% of overall preferences versus 34.6% without planning. Experts also favor the unified model over a separate language model and diffusion Transformer. On WildSongBench, YuE2 scores 6.73 on SongBench Global Avg, exceeding all evaluated public baselines. Selecting from eight candidates (best-of-8), YuE2 reaches 6.96, the highest observed mean among all evaluated systems. Expert listening further establishes its competitiveness with proprietary song generators, favoring best-of-8 over Suno v4.5 and yielding nearly balanced preferences against Suno v5. To learn this generation process from recordings without aligned scores, we introduce MERT2 and SheetSage2 to supply semantic and symbolic supervision. MERT2 sets a new state of the art in music representation learning, surpassing previous best results on 14 of 15 MARBLE metrics; SheetSage2 leads 12 of 15 benchmark-metric pairs in our lead-sheet transcription comparison. The same checkpoint follows score edits while largely preserving unedited musical content and generates zero-shot covers without cover-specific training. Its readable score also enables agentic music editing, with external language models translating user feedback into revisions of the composition.
Ruibin Yuan, Jiahao Pan, Junyan Jiang +32
HKUST HKGAI M-A-P Tokenwave.AI New York University Stanford University MBZUAI ACE Studio
We introduce Qwen-Music, a music generation model that produces high-fidelity songs with complete vocals. It supports text-to-music generation from descriptions, lyrics, and musical attributes, and cover song generation with different styles and vocal characteristics. Qwen-Music comprises three components: Qwen-Music-Tokenizer, Qwen-Music-LLM, and Qwen-Music-Render. The tokenizer compresses audio into a 25 Hz single-codebook stream of Music Semantic Tokens that preserve semantic and melodic information. The LLM performs autoregressive modeling with a melody-token-based chain-of-thought (Melody-CoT) mechanism that plans melodies before full-song generation, improving musicality, structural coherence, and reference-melody preservation. The renderer enriches discrete semantic tokens with acoustic details to produce high-fidelity stereo waveforms. We train the LLM using a quality-aware pre-training curriculum followed by progressive post-training with supervised initialization, offline DPO, and online GSPO to improve musicality and instruction following. On 600 evaluation inputs, Qwen-Music achieves state-of-the-art results in 13 of 16 objective musicality and audio-quality metrics. Professional evaluators also prefer Qwen-Music over leading proprietary systems. For cover generation, Qwen-Music preserves reference melodies more accurately than Suno V5.5, Suno V5, and MiniMax Cover on the AI-generated reference set, and outperforms MiniMax Cover on most metrics on the real-world popular-song reference set.