A Tutorial on Generative Models for Music Audio

Mateusz Modrzejewski; Joanna Kurkowska; Dorien Herremans

eess.ASOct 28, 2026

A Tutorial on Generative Models for Music Audio

Authors: Mateusz Modrzejewski, Joanna Kurkowska, Dorien Herremans

Organizations: Warsaw University of Technology, Institute of Computer Science, and ZAiKS Lab · ZAiKS Lab · Singapore University of Technology and Design

Abstract

Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.

Figures & tables

Figure 1: Composition of the reviewed literature by year and category. Counts describe the monograph’s bibliography only. Entries before 2023 are the precursors cited for context, and the nine entries published before 2019 are not shown. The year 2026 is a partial publication year, and Section 2 distinguishes the system cutoff from later contextual additions.

Symbol	Meaning
$x$ , $T$ , $f_{\mathrm{s}}$	waveform, its number of samples, the sample rate
$c$	the condition: text, lyrics, audio, symbolic signals, or video
$E$ , $D$	encoder and decoder of a codec or autoencoder
$e_{n}$ , $N$ , $f_{\mathrm{fr}}$	encoder frames, their number, the frame rate
$Q$ , $K$ , $\mathcal{C}^{(q)}$	number of codebooks, codebook size, the $q$ -th codebook
$z^{(q)}_{n}$ , $u_{i}$	token of stream $q$ at frame $n$ ; the flattened token sequence

Table 1: Notation. The same symbols are used in every section. Where a cited paper uses other symbols, a notation remark in the text gives the correspondence.

Figure 2: Data paths of the five generative paradigms (D2). Boxes are the stages a sample passes through, from the condition to the audio, and arrows give the order of computation. Notation follows Section 4 .

Figure 3: Two schedules for predicting four token streams of six frames with one autoregressive model. Left: the flattened schedule predicts the four tokens of each frame one after another, so a clip of

N

frames costs

NQ

steps (here 24). Right: the delay schedule predicts one token of every stream at each step, with stream

q

delayed by

q-1

frames, so the same clip costs

N+Q-1

steps (here 9); the hatched cells are the steps at which a delayed stream has nothing to predict yet, and the numbers are frame indices. At each step of the delay schedule the later streams see the earlier streams of the same frame, which is what makes the single pass possible at a small cost in quality.

Figure 4: Structure of the monograph. Dimensions D1 to D3 (representation, generative paradigm, conditioning) describe how a model works (Sections 3 and 5 ), and the application classes D4 organize the review (Sections 6 to 10 ). The release attributes (D5) are recorded in the comparison tables of Sections 6 to 9 , evaluation is treated in Section 11 , and data, provenance, and open challenges are treated in Sections 12 and 13 . Questions Q1 to Q4 of Section 1 are taken up in Sections 3 to 10 , Q5 in Section 11 , and Q6 in Section 12 .

Functional category	Input (D3)	Class (D4)	Examples and section
Text-prompt instrumental	global text	T1	MusicGen ( Copet et al., 2023 ) , Stable Audio ( Evans et al., 2024a ) ; Section 6
Lyrics-to-song	lyrics, structure tags, text	T2	YuE ( Yuan et al., 2025 ) , DiffRhythm ( Ning et al., 2025 ) ; Section 7
Audio-to-audio	reference or live audio	T2, T3, T4	SingSong ( Donahue et al., 2023 ) (vocals to accompaniment, tabulated with the accompaniment systems of T3), LiveBand ( Pasini et al., 2026 ) ; Sections 7 , 8 , 9
Music editing	audio and an instruction	T3	InstructME ( Han et al., 2023 ) , MusicMagus ( Zhang et al., 2024b ) ; Section 8
Video-to-music	video or image	T5	VidMuse ( Tian et al., 2025b ) , V2M-Zero ( Lin et al., 2026 ) ; Section 10
Understanding and generation	dialogue and audio	T6	M2UGen ( Liu et al., 2023b ) , MusicAgent ( Yu et al., 2023 ) ; Section 10

Table 2: Functional (input-to-output) categories of music generation systems and their place in the taxonomy.

Dim.	Value	Mechanism explained in	Systems reviewed in
D1	Discrete codec tokens	Sections 3 and 4	Sections 19 , 20 , and 7
D1	Continuous latents	Sections 3 and 4	Sections 21 and 7
D1	Hybrid plan or sketch	Section 12	Sections 25 and 27
D1	Symbolic	Section 12	Section 36
D2	Autoregressive	Section 4	Section 19
D2	Masked and parallel	Section 4	Section 20

Table 3: Comparison questions and the sections that answer them. For each value of the five dimensions, the section that explains the mechanism and the section that reviews the systems that use it.

Figure 5: Landmark music generation systems of the period reviewed, by generative paradigm (rows), application class (color), and openness (filled: open weights; hollow: closed; grey: partial release, withdrawn release, or release not stated in the paper). Dates are first public release, placed to the month.

Figure 6: Coverage of the reviewed systems across generative paradigm (D2) and application class (D4), for the classes tabulated in Tables 4 to 6 . A system that spans classes is counted once, under its table row (M2UGen and the agent systems under T3, production co-creation, for their editing and co-creation use); the cross-modal (T5) and unified (T6) systems of Section 10 without a table row do not appear. The last row counts the rows whose paradigm is undisclosed (the two commercial products) or outside the nine groups, so that the figure covers all 105 rows. Bubble area and number give the count of systems per cell; assignments are the table rows. Empty cells mark combinations with no tabulated system. Counts are derived from the rows of the tables.

System, D1	Attributes (D3, D5)
Autoregressive token language models
MusicGen ( Copet et al., 2023 ) D1 Tok	Inputs text; melody; Duration 30 s window; longer by windowed continuation (official code) ctx ; Output 32 kHz mono; stereo models by fine-tuning (paper Section 4.3), released as separate checkpoints; Latency batch, offline; Open ✓; Data L 20k h lic.; Post-training –.
MusicLM ( Agostinelli et al., 2023 ) D1 Tok	Inputs text; melody (hummed); Duration 5 min demonstrated gen ; Output 24 kHz mono; Latency batch, offline; Open $\times$ ; Data U 280k h; Post-training –.
UniAudio ( Yang et al., 2024 ) D1 Tok	Inputs text and task tokens; Duration ?; Output ?; Latency batch, offline; Open R; Data P 165k h mix (all-audio multi-task corpus); Post-training –.
Masked and parallel decoding
MAGNeT ( Ziv et al., 2024 ) D1 Tok	Inputs text; Duration 30 s ctx ; Output 32 kHz; Latency batch, faster than real time; Open ✓; Data L 20k h lic.; Post-training –.

Table 4: Instrumental text-to-music systems (T1). Coverage and source dates are defined in Section 2 and Appendix 15 .

System, D1	Attributes (D3, D5)
Make-An-Audio ( Huang et al., 2023b ) D1 Lat	Inputs text (prompt enhancement); Duration 10 s gen ; Output 16 kHz; Latency batch, offline; Open ✓; Data P 3k h mix (AudioSet, AudioCaps, Clotho, ESC-50, BBC and others); Post-training –.
Noise2Music ( Huang et al., 2023a ) D1 Wav	Inputs text (pseudo-captions); Duration 30 s gen ; Output 24 kHz; Latency batch, offline; Open $\times$ ; Data U 340k h; Post-training –.
AudioLDM 2 ( Liu et al., 2024b ) D1 Lat	Inputs text; Duration 10 s gen ; Output 16 kHz (paper); 48 kHz checkpoint in the repository; Latency batch, offline; Open ✓; Data P 29.5k h mix (includes speech and sound effects); Post-training –.
JEN-1 ( Li et al., 2024a ) D1 Lat	Inputs text; audio (continuation, inpainting); Duration ?; Output 48 kHz stereo; Latency batch, offline; Open $\times$ ; Data L 15k tracks; Post-training –.
Moûsai ( Schneider et al., 2024 ) D1 Lat	Inputs text; Duration about 44 s per evaluated window; minutes by chaining eval ; Output 48 kHz stereo; Latency batch, offline; Open ✓; Data U 2.5k h coll.; Post-training –.
MusicLDM ( Chen et al., 2024b ) D1 Lat	Inputs text; Duration 10 s gen ; Output 16 kHz; Latency batch, offline; Open ✓; Data L 455.6 h coll (Audiostock); Post-training –.

Table 4: Instrumental text-to-music systems (T1) (continued).

System, D1	Attributes (D3, D5)
Stable Audio Open ( Evans et al., 2025 ) D1 Lat	Inputs text; timing; Duration 47 s gen ; Output 44.1 kHz stereo; Latency batch, offline; Open ✓; Data CC/P 7.3k h mix (Freesound, FMA); Post-training –.
Flow matching and rectified flow
FluxMusic ( Fei et al., 2024b ) D1 Lat	Inputs text; Duration 10 s gen ; Output 16 kHz; Latency batch, offline; Open ✓; Data P/U 22k h (ASM, MTT, MSD, FMA, Music4All plus private); Post-training –.
JASCO ( Tal et al., 2024 ) D1 Lat	Inputs text; chords; melody; drums; audio; Duration about 10 s gen ; Output 32 kHz; Latency batch, offline; Open ✓; Data L 16k h lic. (Meta Music Initiative, Shutterstock, Pond5 per the model card); Post-training –.
MusicFlow ( Prajwal et al., 2024 ) D1 Lat	Inputs text; audio (continuation); Duration 10 s training chunks ctx ; Output 32 kHz mono; Latency batch, offline; Open A; Data U 20k h; Post-training –.
Stable Audio 3 ( Evans et al., 2026 ) D1 Lat	Inputs text; inpainting; variable length; Duration 2 min (small); 6 min 20 s (medium, large) gen ; Output 44.1 kHz stereo; Latency batch, faster than real time; Open S; Data L/CC; Post-training adv : adversarial post-training.

Table 4: Instrumental text-to-music systems (T1) (continued).

Figure 7: The planner-renderer data path and where its training and inference differ. In training (top), the planner learns to predict a plan

\pi

(semantic tokens, a melody, or a score) from the condition, and the renderer learns to produce audio from plans taken from the data, which are clean. At inference (bottom), the renderer receives the planner’s own prediction

\hat{\pi}

, which contains errors the renderer never saw in training. This mismatch is the exposure bias that the full-context renderer of Section 25 is trained against. It is the cost of the separation that gives the design its long-range structure.

System, D1	Attributes (D3, D5)
Autoregressive token language models
Chord-Conditioned Song Generator ( Gao et al., 2024 ) D1 Tok	Inputs lyrics; chords; Duration 12 s segments gen ; Output ?; Latency batch, offline; Open ?; Data U 554k songs; Post-training –.
SongCreator ( Lei et al., 2024 ) D1 Tok	Inputs lyrics; prompts; audio; Duration up to 30 s gen ; Output 24 kHz tokens, 44.1 kHz render; Latency batch, offline; Open $\times$ ; Data U 8.5k h; Post-training –.
LeVo ( Lei et al., 2025 ) D1 Tok	Inputs lyrics; text; audio prompt; Duration ?; Output 48 kHz; Latency batch, offline; Open W; Data P/U 110k h; Post-training pref : multi-preference DPO.
SongGen ( Liu et al., 2025d ) D1 Tok	Inputs lyrics; text; voice prompt; Duration up to 30 s gen ; Output 16 kHz (X-Codec); Latency batch, offline; Open ✓; Data P 2k h; Post-training –.
YuE ( Yuan et al., 2025 ) D1 Tok	Inputs lyrics; structure tags; in-context audio; Duration up to 5 min generated; about 163 s context window gen ; Output 16 kHz, upsampled to 44.1 kHz; Latency batch, offline; Open ✓; Data U; Post-training –.

Table 5: Song-generation systems (T2). Coverage and source dates are defined in Section 2 and Appendix 15 . Legend as in Table 4 .

System, D1	Attributes (D3, D5)
HeartMuLa ( Yang et al., 2026b ) D1 Tok	Inputs lyrics; tags (multilingual); Duration up to 6 min gen ; Output 48 kHz stereo; Latency batch, offline; Open ✓; Data U 100k h; Post-training pref : DPO.
LeVo 2 ( Lei et al., 2026 ) D1 Tok	Inputs lyrics; text; Duration full-length songs (length not stated) gen ; Output 48 kHz; Latency batch, offline; Open W; Data U 500k h; Post-training sft/pref : SFT, offline DPO, semi-online DPO.
Muse ( Jiang et al., 2026 ) D1 Tok	Inputs lyrics; style descriptions; Duration about 4 min songs gen ; Output MuCodec tokens (rate not stated); Latency batch, offline; Open ✓; Data L/Syn 7.8k h lic.; Post-training –.
Shao ( Liu et al., 2026a ) D1 Tok	Inputs lyrics; duration; text; Duration about 6 min (381 s at 21.5 Hz token rate) gen ; Output 44.1 kHz; Latency batch, offline; Open ✓; Data U 1.2M h int.; Post-training –.
Latent and cascaded diffusion
MIDI-SAG ( Tsai et al., 2026 ) D1 Lat	Inputs vocal MIDI; chords; structure; Duration 47 s single pass; 90 to 120 s with continuation gen ; Output ?; Latency batch, offline; Open ✓; Data U 2.5k h; Post-training –.

Table 5: Song-generation systems (T2) (continued).

System, D1	Attributes (D3, D5)
Jam ( Liu et al., 2025c ) D1 Lat	Inputs lyrics (word-level timing); text; Duration up to 3 min 50 s gen ; Output 44.1 kHz stereo; Latency batch, offline; Open ?; Data U 54k h; Post-training pref : DPO with SongEval scores.
VersBand ( Zhang et al., 2025e ) D1 Lat	Inputs prompts (text, audio); Duration phrases up to 20 s gen ; Output 24 kHz; Latency batch, offline; Open C; Data P/U 1k h; Post-training –.
MPEcho ( Lee et al., 2026b ) D1 Lat	Inputs reference melody; phonemes; text; Duration ?; Output ?; Latency batch, offline; Open ✓; Data P 1.4k h; Post-training –.
SegTune ( Wang et al., 2026e ) D1 Lat	Inputs lyrics; segment prompts; Duration 30 s to 6 min training data range; generation maximum not stated ctx ; Output 44 kHz (as written in the paper); Latency batch, offline; Open ?; Data U 27k h pretrain.; 4k h ft.; Post-training pref : iterative DPO.
UniSinger ( Zhang et al., 2026c ) D1 Lat	Inputs lyrics; speaker prompt; Duration 15 to 20 s evaluation clips; generation length not stated eval ; Output 44.1 kHz; Latency batch, offline; Open ?; Data U 30k h coll.; 20k h train.; 5k h accomp.; Post-training –.
WanSong ( Chen et al., 2026a ) D1 Lat	Inputs lyrics; text; Duration up to 5 min gen ; Output 44.1 kHz stereo; Latency batch, faster than real time; Open ?; Data U; Post-training pref : RLHF (DPO and RL with reward models).

Table 5: Song-generation systems (T2) (continued).

System, D1	Attributes (D3, D5)
SongBloom ( Yang et al., 2025a ) D1 Hyb	Inputs lyrics (structured); 10 s reference audio; text; Duration up to 150 s gen ; Output 48 kHz stereo; Latency batch, offline; Open ✓; Data U 100k h; Post-training –.
ACE-Step 1.5 ( Gong et al., 2026a ) D1 Hyb	Inputs lyrics; text; Duration up to 10 min gen ; Output 48 kHz stereo; Latency batch, faster than real time; Open ✓; Data U; Post-training pref : intrinsic RL (GRPO, DiffusionNTF).
Dai et al. full-song framework ( Dai et al., 2026 ) D1 Hyb	Inputs lyrics; text; attributes; cover; Duration up to about 328 s gen ; Output 48 kHz stereo; Latency batch, offline; Open ?; Data U; Post-training sft/pref : SFT, DPO, GRPO, on-policy distillation.
Qwen-Music ( Xu et al., 2026 ) D1 Hyb	Inputs lyrics; text; melody tokens; Duration renderer training samples up to 6 min; generation maximum not stated ctx ; Output 48 kHz stereo; Latency batch, offline; Open ?; Data U; Post-training sft/pref : SFT, offline DPO, online GSPO.
SketchSong ( Duan et al., 2026a ) D1 Hyb	Inputs lyrics; text; Duration song-level (length not stated) gen ; Output LeVo codec pipeline (rate not stated); Latency batch, offline; Open ?; Data U 54k h; Post-training –.
Undisclosed architecture (products)

Table 5: Song-generation systems (T2) (continued).

System, D1	Attributes (D3, D5)
Autoregressive token language models
Coco-Mulla ( Lin et al., 2023 ) D1 Tok	Inputs chords; drums; Duration 20 s gen ; Output 32 kHz (MusicGen EnCodec backbone); Latency batch, offline; Open C; Data U 17 h; Post-training –.
SingSong ( Donahue et al., 2023 ) D1 Tok	Inputs input vocals; Duration 10 s clips eval ; Output 16 kHz; Latency batch, offline; Open $\times$ ; Data U 46k h; Post-training –.
Arrange-Inpaint-Refine ( Lin et al., 2024 ) D1 Tok	Inputs frame-level content controls; score; Duration 15 s gen ; Output 32 kHz (MusicGen EnCodec backbone); Latency batch, offline; Open ?; Data P 1,289 songs; Post-training –.
Instruct-MusicGen ( Zhang et al., 2024a ) D1 Tok	Inputs instruction; audio; Duration 5 s gen ; Output 32 kHz (MusicGen EnCodec backbone); Latency batch, offline; Open R; Data P/Syn; Post-training –.
MusiConGen ( Lan et al., 2024 ) D1 Tok	Inputs rhythm; chords; Duration 30 s gen ; Output 32 kHz (EnCodec 32k); Latency batch, offline; Open ✓; Data U 250 h; Post-training –.

Table 6: Control, editing, stem, and live systems (T3 and T4). Coverage and source dates are defined in Section 2 and Appendix 15 . Legend as in Table 4 .

System, D1	Attributes (D3, D5)
Lyria RealTime ( Lyria Team et al., 2025 ) D1 Tok	Inputs text; audio prompts, continuously; Duration arbitrary; 2 s chunks, 10 s context stream ; Output 48 kHz stereo (SpectroStream); Latency streaming (generates at least as fast as playback); Open $\times$ ; Data ?; Post-training –.
Magenta RealTime ( Lyria Team et al., 2025 ) D1 Tok	Inputs text; audio prompts, continuously; Duration arbitrary; 2 s chunks, 10 s context stream ; Output 48 kHz stereo (SpectroStream); Latency streaming (generates at least as fast as playback); Open ✓; Data U 190k h; Post-training –.
MusicGen-Stem ( Rouard et al., 2025 ) D1 Tok	Inputs text; stems; Duration 30 s gen ; Output 32 kHz mono per stem; Latency batch, offline; Open ✓; Data U 17k h; Post-training –.
Streaming accompaniment (Wu et al.) ( Wu et al., 2025e ) D1 Tok	Inputs live audio input; Duration streaming; 10 s training windows ctx ; Output 32 kHz (DAC); Latency streaming (future-visibility and chunk trade-off); Open ✓; Data P 145 h; Post-training –.
Masked and parallel decoding
VampNet ( Garcia et al., 2023 ) D1 Tok	Inputs audio prompt; mask pattern (no text); Duration 10 s samples eval ; Output 44.1 kHz; Latency batch, faster than real time; Open ✓; Data U 797k tracks; Post-training –.