Controllable Music Generation

Follow this topic Ask about this topic

Controllable music generation studies how to make generated music follow specific instructions, from chords, melodies, rhythm, and instrumentation to lyrics, style, emotion, and song structure. Systems use text-conditioned diffusion and autoregressive models, symbolic planning and retrieval, reference-audio or MIDI conditioning, and inference-time methods such as latent or activation steering; interactive approaches also use dialogue, motion, or live feedback. Evaluation spans symbolic tasks, song and instrument synthesis, editing, and video- or dance-to-music alignment, with tests for both control adherence and musical or audio quality. A central challenge is achieving precise, independently adjustable controls without harming naturalness, coherence, or fidelity, while developing evaluations that distinguish genuine instruction-following from models’ default tendencies.

Latest papers

91 papers10 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

Fast Diffusion-GAN Model for Generating Symbolic Music with Emotions
Oct 21, 2023Jincheng Zhang, György Fazekas, Charalampos SaitisTimbre-Conditioned Music Generation Music Generation
Centre for Digital Music, Queen Mary University of London, UK

Oct 21, 2023J/K move · Enter open · S save

Fast Diffusion-GAN Model for Generating Symbolic Music with Emotions

Jincheng Zhang, György Fazekas, Charalampos Saitis

Centre for Digital Music, Queen Mary University of London, UK

Timbre-Conditioned Music Generation Music Generation Symbolic Music Generation Diffusion Models Controllable Music Generation Few-Step Diffusion Sampling

Open paper AskPDF arXiv

Diffusion models have shown promising results for a wide range of generative tasks with continuous data, such as image and audio synthesis. However, little progress has been made on using diffusion models to generate symbolic music since this class of generative models struggles with discrete data while its iterative sampling process is computationally expensive. In this work, we propose a diffusion model integrated with a Generative Adversarial Network, aiming to mitigate the slow sampling drawback of diffusion models applied to symbolic music generation. We enable the use of diffusion model by representing symbolic music as image-like pianorolls with velocity values instead of discrete event tokens and then used a Variational Autoencoder to obtain their latent representations to train the proposed diffusion model. Our results demonstrate the successful control of our diffusion model to generate symbolic music with a desired emotion. Our model uses merely 15 denoising diffusion steps to produce high-quality music, while most diffusion models for symbolic music generation require hundreds or thousands of steps.