Controllable Music Generation

Diffusion models have shown promising results for a wide range of generative tasks with continuous data, such as image and audio synthesis. However, little progress has been made on using diffusion models to generate symbolic music since this class of generative models struggles with discrete data while its iterative sampling process is computationally expensive. In this work, we propose a diffusion model integrated with a Generative Adversarial Network, aiming to mitigate the slow sampling drawback of diffusion models applied to symbolic music generation. We enable the use of diffusion model by representing symbolic music as image-like pianorolls with velocity values instead of discrete event tokens and then used a Variational Autoencoder to obtain their latent representations to train the proposed diffusion model. Our results demonstrate the successful control of our diffusion model to generate symbolic music with a desired emotion. Our model uses merely 15 denoising diffusion steps to produce high-quality music, while most diffusion models for symbolic music generation require hundreds or thousands of steps.