StepAudio 3 Music Technical Report

Chengli Feng; Zhiyue Wu; Jiahao Song; Zheqi Dai; Boyang Wang; Ruibin Yuan; Junming Gong; Wenxiao Zhao

eess.ASSep 11, 2026

StepAudio 3 Music Technical Report

Authors: Chengli Feng, Zhiyue Wu, Jiahao Song, Zheqi Dai, Boyang Wang, Ruibin Yuan, Junming Gong, Wenxiao Zhao,

Organizations: StepFun · The Chinese University of Hong Kong · University of California San Diego · ACE

Abstract

We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.

Figures & tables

Figure 1: Relative performance across MuQ-MuLan, AudioBox-Aesthetics, and SongBench. Outer arcs identify the three benchmark families. Scores are divided by the best observed score per metric (1.0), not a theoretical maximum; the radial axis is truncated to 0.60–1.00. Purple denotes StepAudio 3 Music. Compare systems within each metric; polygon area is not an aggregate score. Curves provide a visual approximation; exact scores are in table 3 .

Figure 2: Overview of StepAudio 3 Music. Lyrics, a text prompt, and optional task-specific references are serialized for a trainable Mixture-of-Experts autoregressive model. In ABC-CoT mode, the model first produces an explicit arrangement plan and then predicts a 50-Hz sequence from a 65536-entry single codebook. A separately trained renderer generates 50-Hz StepAudio VAE latents with a flow-matching DiT and decodes them into 48-kHz waveform audio. The renderer is held fixed during autoregressive-model training.

Figure 3: Discrete bottleneck designs considered for the music tokenizer. The controlled comparison uses a common 25-Hz frame rate for all three variants. (a) Single-codebook VQ produces one semantic token $q_{1}$ per frame. (b) Semantic RVQ quantizes residuals within the same semantically supervised latent space; the aggregate embedding $\sum_{i}e_{i}(q_{i})$ is processed by the upper encoder under the shared multi-task objectives. (c) Acoustic RVQ freezes the semantic tokenizer and quantizes the residual between an acoustic feature and the aligned semantic feature. Teal elements denote semantic representations and tokens, green denotes acoustic features, and blue denotes acoustic residual tokens. A separate DiT detokenizer is trained for the final comparison after each tokenizer is frozen and is omitted from the diagram.

Figure 4: Music data cleaning and structured annotation pipeline. Raw audio is filtered for bandwidth and content quality, separated into vocal and accompaniment stems, routed to language-appropriate transcription systems, and aligned with predicted song sections to produce structured training records. Dashed branches indicate rejection or auxiliary data paths.

Figure 5: Three-stage training pipeline for the StepAudio Music Tokenizer. Stage 1 adapts BEST-RQ pretraining with bidirectional Conformer blocks. Contiguous waveform spans are replaced with Gaussian noise before Mel extraction, and the BEST-RQ loss is evaluated only at the corresponding positions in the aligned 50-Hz sequence. Stage 2 introduces full-song CTC, Mel, and Chroma supervision. Stage 3 inserts a single-codebook VQ bottleneck with 65536 entries between the lower and upper 12-layer Conformer stacks, producing one discrete music token per frame at 50 Hz. Each stage is initialized from the checkpoint produced by the preceding stage.

Stage	Tokens	Context	Batch	LR schedule
Large-scale Pretraining	600B	16,384	768	$2.45\times 10^{-5}$ , constant
Multi-task Mid-Pretraining	500B	32,768	512	$2.45\times 10^{-5}\!\rightarrow\!2.0\times 10^{-5}$ , cosine
High-quality Annealing	80B	32,768	512	warm-up to $2.0\times 10^{-5}$ , then cosine to $2.0\times 10^{-6}$

Table 1: LLM pretraining schedule for StepAudio 3 Music. Token counts refer to training consumption; Stage 3 includes high-quality replay.

Subset	Tokens	Principal conditioning
Lyrics-to-music	6B	Lyrics and text prompt
Instrumental generation	2B	text prompt
Lyrics-to-music with ABC-CoT	4B	Lyrics, text prompt, and arrangement plan
Instrumental generation with ABC-CoT	2B	text prompt and arrangement plan
Cover-song generation	4B	Reference song
Vocal-to-mix generation	2B	Dry vocal track

Table 2: Composition of the supervised fine-tuning corpus. The listed subsets collectively contain approximately 20B tokens. Token counts are approximate.

Metric	StepAudio 3 Music	Suno V5.5	Suno V5	Mureka V9	MiniMax Music 3
Caption adherence [ 28 ] (music–text similarity; $n=339$ )
MuQ-MuLan $\uparrow$	0.4465	0.4437	0.4287	0.3752	0.3703
AudioBox-Aesthetics [ 27 ] (1–10; $n=339$ )
Content Enjoyment $\uparrow$	7.7086	7.5646	7.6823	7.4363	7.5654
Content Usefulness $\uparrow$	8.0052	7.9584	7.8745	7.6692	7.8677
Production Quality $\uparrow$	8.3868	8.2988	8.2975	8.0086	8.2976

Table 3: Objective evaluation on common benchmark subsets. Caption adherence and AudioBox-Aesthetics are reported on 339 lyrics-to-song conditions; SongBench is reported on 316 vocal conditions. Values are means. Bold and underlining indicate the best and second-best scores, respectively, for metrics where higher is better. Production Complexity is descriptive and is not ranked. The dashed line separates StepAudio 3 Music from the comparison systems.

Figure 6: Artificial Analysis Music Arena Vocals leaderboard. Supplied preliminary benchmark results based on blind preference votes. StepAudio 3 Music obtains a Quality Elo of 1105 through direct evaluation, without any adaptation to AA prompts.

Model	Latent rate	MCD $\downarrow$	MS-Mel-L1 $\downarrow$	MS-STFT-L1 $\downarrow$	SI-SNR $\uparrow$	SDR $\uparrow$	UTMOS $\uparrow$
Direct VAE reconstruction
StepAudio VAE (25 Hz)	25 Hz	$4.79\pm 1.31$	$0.96\pm 0.12$	$0.51\pm 0.31$	$4.21\pm 4.01$	$5.54\pm 2.96$	–
StepAudio VAE (50 Hz)	50 Hz	$3.59\pm 1.32$	$0.88\pm 0.11$	$0.42\pm 0.26$	$7.68\pm 4.23$	$8.57\pm 3.53$	–
Token-conditioned detokenization with the StepAudio VAE (50 Hz)
0.9B DiT	50 Hz	$\mathbf{5.48\pm 1.12}$	$\mathbf{1.32\pm 0.27}$	$\mathbf{0.24\pm 0.07}$	$\mathbf{-19.49\pm 9.06}$	$-2.97\pm 1.16$	$\mathbf{3.45\pm 0.60}$
4B DiT	50 Hz	$6.72\pm 0.90$	$1.63\pm 0.31$	$0.27\pm 0.11$	$-23.27\pm 12.23$	$\mathbf{-2.47\pm 1.08}$	$3.07\pm 0.45$

Table 4: Ablations of the music detokenizer. Results are reported as mean

\pm

standard deviation over 54 audio segments. Lower MCD, MS-Mel-L1, and MS-STFT-L1 are better; higher SI-SNR, SDR, and UTMOS are better. Direct VAE reconstruction and token-conditioned detokenization are reported as separate experiment groups; UTMOS is reported only for the latter.

Configuration	SB-Musicality $\uparrow$	SB-Mean $\uparrow$	CE $\uparrow$	PQ $\uparrow$	MuQ-MuLan $\uparrow$
SFT	5.6395	6.5445	7.6259	8.2700	0.4157
+ DPO (final)	5.7465	6.6438	7.7086	8.3868	0.4465

Table 5: Effect of DPO after supervised fine-tuning. SB denotes SongBench, and SB-Mean is the mean over its seven dimensions. CE and PQ are the Content Enjoyment and Production Quality axes of AudioBox-Aesthetics. Higher values are better for all reported metrics.

Dimension	Direct	ABC-CoT	ABC-CoT + LLM Editing
Melody	6.5394	6.6054	6.6454
Arrangement	6.9122	6.9320	7.0211
Vocal	7.0172	7.0256	7.0282
Instrumental	7.0086	7.0122	7.0396
Mixing	6.7284	6.7698	6.8483
Structure	6.5544	6.5868	6.6411

Table 6: SongBench results for direct generation, ABC-CoT planning, and LLM-edited ABC-CoT planning using the DPO model. Scores are reported on a 1–10 scale; higher is better. Mean is the arithmetic average over the seven dimensions. Bold indicates the highest value in each row.

Text-to-Music Generation

Text-to-music generation turns natural-language descriptions into audio or symbolic scores, with control over style, instruments, mood, lyrics, and structure.

See all 48 papers

StepAudio 3 Gen Technical Report
Sep 11, 2026Bin Lin, Bo Zhao, Boyang Wang +68TTS Synthesis Zero-Shot TTS
YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality
Sep 27, 2026Ruibin Yuan, Jiahao Pan, Junyan Jiang +32Audio Representation Learning Controllable Music Generation
HKUST HKGAI M-A-P Tokenwave.AI New York University Stanford University MBZUAI ACE Studio
Qwen-Music Technical Report
Jul 13, 2026Jin Xu, Kangdi Wang, Ruibin Yuan +24Text-to-Music Generation Controllable Music Generation
Qwen Team

Sep 11, 2026J/K move · Enter open · S save

StepAudio 3 Gen Technical Report

Bin Lin, Bo Zhao, Boyang Wang, Boyang Zhang+67

TTS Synthesis Zero-Shot TTS Controllable Speech Generation Music Generation Audio Generation

Open paper AskPDF arXiv

We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared

16 \times 2048

residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.

Abstract

Text-to-Music Generation

Explore similar work

StepAudio 3 Gen Technical Report

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Qwen-Music Technical Report