Differentiable Digital Signal Processing Mixture Model-Guided Diffusion for Synthesis Parameter Estimation from Harmonic Sound Mixtures

Kengo Takemoto; Tomohiko Nakamura; Hiroshi Saruwatari

cs.SDSep 13, 2026

Differentiable Digital Signal Processing Mixture Model-Guided Diffusion for Synthesis Parameter Estimation from Harmonic Sound Mixtures

Authors: Kengo Takemoto, Tomohiko Nakamura, Hiroshi Saruwatari

Organizations: The University of Tokyo, Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan

Abstract

A differentiable digital signal processing (DDSP) autoencoder reconstructs a monophonic harmonic sound through three types of synthesis parameters: fundamental frequency, loudness, and timbre features. To handle mixtures of harmonic sounds within the DDSP approach, we have previously proposed a DDSP mixture model (DDSPMM). It represents a mixture as the sum of source signals synthesized by the decoders of pretrained DDSP autoencoders. Although DDSPMM enables direct estimation of synthesis parameters of each source from mixtures, it does not explicitly model temporal variations in the synthesis parameters and can produce excessive temporal fluctuations. In this paper, we propose a method for estimating synthesis parameters with temporally plausible trajectories by incorporating a denoising diffusion probabilistic model (DDPM) into the DDSPMM-based estimation. The DDPM is trained as a generative model of synthesis parameters. During estimation, the proposed method guides the DDPM reverse diffusion process with the reconstruction error between the observed mixture and the mixture synthesized by DDSPMM from the current estimates. Experiments on woodwind and string instrument ensembles showed that the DDPM-based regularization improves synthesis parameter estimation by imposing temporal plausibility on the estimated trajectories.

Figures & tables

Figure 1: (A): noise predicter based on the U-Net architecure. DS and US denotes downsampling and upsampling, respectively. (B): convolutional block used in the noise predictor. PE and Conv2D denote positional encoding and two-dimensional convolutional layer, respectively.

Figure 2: Overview of the proposed synthesis parameter estimation algorithm. At each step, reverse diffusion is guided by the multi-scale spectral loss between the observed mixture and the mixture synthesized from the current estimates.

Instruments	Duration [ $\mathrm{s}$ ]	Method	$F_{\text{o}}$ [ $\mathrm{cent}$ ] ( $\downarrow$ )	Loudness [ $\mathrm{dB}$ ] ( $\downarrow$ )	MFCCs ( $\downarrow$ )	Zimtohrli score ( $\uparrow$ )
Flute / Flute (Fugue)	172	SS–DDSP	$281\pm 165$	$6.11\pm 0.330$	$3.70\pm 0.457$	$2.65\pm 0.115$
		DDSPMM	$18.5\pm 6.39$	$5.35\pm 0.368$	$2.09\pm 0.0860$	$3.70\pm 0.0251$
		Proposed	$\bm{14.0\pm 5.97}$	$\bm{4.08\pm 0.833}$	$\bm{2.01\pm 0.0211}$	$\bm{3.81\pm 0.0575}$
		Reconst.	-	-	$1.63\pm 0.0400$	$4.05\pm 0.00784$
Clarinet / Violin (Pavane)	133	SS–DDSP	$12.5\pm 8.95$	$9.07\pm 3.64$	$3.29\pm 0.101$	$3.17\pm 0.180$
		DDSPMM	$\bm{9.82\pm 2.18}$	$5.70\pm 1.22$	$2.12\pm 0.146$	$\bm{3.77\pm 0.159}$

Table 1: Estimation errors and Zimtohrli scores for the synthesis parameter estimation methods. The Instruments column lists the instrument types in each mixture; the labels in parentheses indicate the corresponding musical pieces in the URMP dataset.

Panel of Figure 3 — Figure 3: Illustrative examples of violin $F_{\text{o}}$ , loudness and MFCCs trajectories estimated from the Flute / Violin / Clarinet mixture in Rondeau. The proposed method suppresses excessive temporal fluctuations while preserving performance-related variations in $F_{\text{o}}$ , loudness, and MFCC.

Parameter Estimation

Parameter estimation studies how to infer unknown quantities in statistical, physical, and machine-learning models from observed data.

See all 144 papers

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning
Aug 4, 2026Tristan Wu, Daniel Chin, Junan Zhang +3Diffusion-Based Inverse Problems Diffusion Models
Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +2
Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
Aug 5, 2026Yisu Zong, Jinjie Shi, Joshua ReissControllable Music Generation Audio Diffusion Models
Centre for Digital Music, Queen Mary University of London, UK
Physics-Informed Neural Engine Sound Modeling with Differentiable Pulse-Train Synthesis
Mar 10, 2026Robin Doerfler, Lonce WyseAudio Generation Physics-Informed Generative Modeling
Audio Research & Development, Impulse Audio Lab GmbH, Munich, Germany · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain

Aug 4, 2026J/K move · Enter open · S save

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

Tristan Wu, Daniel Chin, Junan Zhang, Junyan Jiang+2

Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen+2

Diffusion-Based Inverse Problems Diffusion Models

Open paper AskPDF arXiv

Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.

Abstract

Parameter Estimation

Explore similar work

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects

Physics-Informed Neural Engine Sound Modeling with Differentiable Pulse-Train Synthesis