Music Inpainting With Similarity-Guided Latent Diffusion
Organizations: Acoustic Lab, Dept. Information and Communications Engineering, Aalto University, Finland · Acoustics and Audio Group, University of Edinburgh, United Kingdom
Abstract
We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves candidate replacement segments from the wider signal based on contextual similarity and uses them to guide the diffusion process toward coherent reconstructions. Being model-agnostic, the framework is compatible with any pre-trained diffusion model operating in either the waveform or latent domain. Objective evaluations demonstrate that SimDPS outperforms widely adopted RePaint and DPS methods across acoustic features and neural-embedding distance metrics. Subjective evaluations on multi-genre music datasets confirm that SimDPS enhances perceptual plausibility compared to previous diffusion-based methods and demonstrates robustness to poorly matched similarity candidates. These findings pave the way for improved music restoration and context-aware audio editing.
Figures & tables
| Method | MFCC dist. ( ) | Chroma sim. ( ) | Rhythm corr. ( ) | Boundary err. ( ) | MusicFM dist. ( ) | CLAP sim. ( ) |
|---|---|---|---|---|---|---|
| Latent-Interp. | ||||||
| Latent-Sim. | ||||||
| RePaint | ||||||
| DPS | ||||||
| SimDPS-Solo (proposed) | ||||||
| SimDPS-Multi (proposed) |