A differentiable digital signal processing (DDSP) autoencoder reconstructs a monophonic harmonic sound through three types of synthesis parameters: fundamental frequency, loudness, and timbre features. To handle mixtures of harmonic sounds within the DDSP approach, we have previously proposed a DDSP mixture model (DDSPMM). It represents a mixture as the sum of source signals synthesized by the decoders of pretrained DDSP autoencoders. Although DDSPMM enables direct estimation of synthesis parameters of each source from mixtures, it does not explicitly model temporal variations in the synthesis parameters and can produce excessive temporal fluctuations. In this paper, we propose a method for estimating synthesis parameters with temporally plausible trajectories by incorporating a denoising diffusion probabilistic model (DDPM) into the DDSPMM-based estimation. The DDPM is trained as a generative model of synthesis parameters. During estimation, the proposed method guides the DDPM reverse diffusion process with the reconstruction error between the observed mixture and the mixture synthesized by DDSPMM from the current estimates. Experiments on woodwind and string instrument ensembles showed that the DDPM-based regularization improves synthesis parameter estimation by imposing temporal plausibility on the estimated trajectories.
Figures & tables
Figure 1: (A): noise predicter based on the U-Net architecure. DS and US denotes downsampling and upsampling, respectively. (B): convolutional block used in the noise predictor. PE and Conv2D denote positional encoding and two-dimensional convolutional layer, respectively.
Figure 2: Overview of the proposed synthesis parameter estimation algorithm. At each step, reverse diffusion is guided by the multi-scale spectral loss between the observed mixture and the mixture synthesized from the current estimates.
Instruments
Duration [ s ]
Method
Fo [ cent ] ( ↓ )
Loudness [ dB ] ( ↓ )
MFCCs ( ↓ )
Zimtohrli score ( ↑ )
Flute / Flute (Fugue)
172
SS–DDSP
281±165
6.11±0.330
3.70±0.457
2.65±0.115
DDSPMM
18.5±6.39
5.35±0.368
2.09±0.0860
3.70±0.0251
Proposed
14.0±5.97
4.08±0.833
2.01±0.0211
3.81±0.0575
Reconst.
-
-
1.63±0.0400
4.05±0.00784
Clarinet / Violin (Pavane)
133
SS–DDSP
12.5±8.95
9.07±3.64
3.29±0.101
3.17±0.180
DDSPMM
9.82±2.18
5.70±1.22
2.12±0.146
3.77±0.159
Table 1: Estimation errors and Zimtohrli scores for the synthesis parameter estimation methods. The Instruments column lists the instrument types in each mixture; the labels in parentheses indicate the corresponding musical pieces in the URMP dataset.
Figure 3: Illustrative examples of violin Fo , loudness and MFCCs trajectories estimated from the Flute / Violin / Clarinet mixture in Rondeau. The proposed method suppresses excessive temporal fluctuations while preserving performance-related variations in Fo , loudness, and MFCC.
Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.
Tristan Wu, Daniel Chin, Junan Zhang +3
Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +2
Automatic music mixing aims to combine multitrack recordings into a balanced and coherent musical piece. Because the content of different songs and the subjective preferences of mixing engineers jointly shape the final outcome, a practical system should deliver well-balanced mixes while allowing for controllable stylistic variation. However, most existing methods treat automatic mixing and mixing style control as separate tasks, making it difficult for a single system to produce high-quality mixes while remaining editable and style-aware. To address this limitation, this paper presents Diff2Mix, a generative automatic mixing system based on diffusion models and a differentiable mixing console. This system offers two levels of optional user control: a reference audio enables overall production style control, and the differentiable mixing console provides explicit audio effects parameters for interpretability and fine-grained optimization. We demonstrate our system's competitive performance through both objective and subjective evaluations in terms of mixing quality and control ability. We provide code and audio samples at our project page https://zys711.github.io/Diff2Mix .
Yisu Zong, Jinjie Shi, Joshua Reiss
Centre for Digital Music, Queen Mary University of London, UK
Engine sounds originate from sequential exhaust pressure pulses rather than sustained harmonic oscillations. While neural synthesis methods typically aim to approximate the resulting spectral characteristics, we propose directly modeling the underlying pulse shapes and temporal structure. We present the Pulse-Train-Resonator (PTR) model, a differentiable synthesis architecture that generates engine audio as parameterized pulse trains aligned to engine firing patterns and propagates them through recursive Karplus-Strong resonators simulating exhaust acoustics. The architecture integrates physics-informed inductive biases including harmonic decay, thermodynamic pitch modulation, valve-dynamics envelopes, exhaust system resonances and derived engine operating modes such as throttle operation and Deceleration Fuel Cutoff (DFCO). Validated on three diverse engine types totaling 7.5 hours of audio, PTR achieves a 21% improvement in harmonic reconstruction and a 5.7% reduction in total loss over a harmonic-plus-noise baseline model, while providing interpretable parameters corresponding to physical phenomena. Complete code, model weights, and audio examples are openly available.
Robin Doerfler, Lonce Wyse
Audio Research & Development, Impulse Audio Lab GmbH, Munich, Germany · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain