Controllable Music Generation

Controllable music generation studies how to make generated music follow specific instructions, from chords, melodies, rhythm, and instrumentation to lyrics, style, emotion, and song structure. Systems use text-conditioned diffusion and autoregressive models, symbolic planning and retrieval, reference-audio or MIDI conditioning, and inference-time methods such as latent or activation steering; interactive approaches also use dialogue, motion, or live feedback. Evaluation spans symbolic tasks, song and instrument synthesis, editing, and video- or dance-to-music alignment, with tests for both control adherence and musical or audio quality. A central challenge is achieving precise, independently adjustable controls without harming naturalness, coherence, or fidelity, while developing evaluations that distinguish genuine instruction-following from models’ default tendencies.