INSTRUMENTAL: Automatic Synthesizer Parameter Recovery from Audio via Evolutionary Optimization

Philipp Bogdan

cs.SDMar 16, 2026

INSTRUMENTAL: Automatic Synthesizer Parameter Recovery from Audio via Evolutionary Optimization

Authors: Philipp Bogdan

Abstract

Existing audio-to-MIDI tools extract notes but discard the timbral characteristics that define an instrument's identity. We present Instrumental, a system that recovers continuous synthesizer parameters from audio by coupling a differentiable 28-parameter subtractive synthesizer with CMA-ES, a derivative-free evolutionary optimizer. We optimize a composite perceptual loss combining mel-scaled STFT, spectral centroid, and MFCC divergence, achieving a matching loss of 2.09 on real recorded audio. We systematically evaluate eight hypotheses for improving convergence and find that only parametric EQ boosting yields meaningful improvement. Our results show that CMA-ES outperforms gradient descent on this non-convex landscape, that more parameters do not monotonically improve matching, and that spectral analysis initialization accelerates convergence over random starts.

Figures & tables

Panel of Figure 1 — Figure 1: Pipeline: audio $\to$ Demucs separation $\to$ pitch detection $\to$ CMA-ES optimization loop $\to$ output parameters.

Panel of Figure 2 — Figure 2: CMA-ES convergence. 90% of improvement occurs in the first 10K evaluations (shaded). The remaining 90K evals yield only 0.07 loss reduction.

Params	Modules added	Loss	$\boldsymbol{\Delta}$
15	Osc, filter, ADSR, reverb	4.60	–
18	+ unison, noise floor	2.34	$-49\%$
24	+ pulse width, filter slope	2.13	$-9\%$
28	+ 2-band parametric EQ	2.09	$-2\%$
29	+ distortion, delay, vibrato	div.	–

Table 1: Ablation over synthesizer parameters. Loss is the matching loss averaged over three target notes (lower is better).

Modification	$\Delta$ Loss
Chebyshev waveshaping	$+29\%$
Parametric EQ (2 bands)	$\mathbf{-8\%}$ $\checkmark$
TFS / envelope loss	$+17\%$
SPSA fine-tuning	$+11\%$
Multi-start CMA-ES ( $8{\times}$ )	$+29\%$
FM synthesis (2-op)	$+17\%$

Table 2: Hypothesis tests (10K evals each, baseline loss 2.27).

Panel of Figure 3 — Figure 3: Harmonic amplitudes: original vs. matched. H1–H3 are closely reproduced; H4–H8 diverge by 2–5 $\times$ , revealing the architectural floor of subtractive synthesis.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Landing page. Users drop an audio file (MP3 or WAV) and select Single Sound or Sequence mode.

Figure 5: Optimization in progress. The progress bar, evaluation count, current best loss, and elapsed time update live via WebSocket as CMA-ES runs across 7 CPU cores.

Figure 6: Result view. The recovered waveform is displayed above a 2-octave piano keyboard. Each key plays the matched synthesizer at that pitch using WebAudio. The clock button (bottom-right) loads the best pre-computed preset (loss 2.09).

Parameter Estimation

Parameter estimation studies how to infer unknown quantities in statistical, physical, and machine-learning models from observed data.

See all 144 papers

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search
Sep 25, 2026Ben Hayes, Haokun Tian, Stefan LattnerAudio Representation Learning Joint-Embedding Predictive Architecture
Sony Computer Science Laboratories, Paris, France · Queen Mary University of London, UK
FM Synthesizer Audio-Parameter Shared Embeddings
Aug 18, 2026David Braun, Adam FinkelsteinMusic Information Retrieval Graph Neural Networks
Dept. of Computer Science Princeton University Princeton, NJ, USA
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
Oct 7, 2026Tristan Wu, Daniel Chin, Liwei Lin +2Music Generation
The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +1

Sep 25, 2026J/K move · Enter open · S save

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

Ben Hayes, Haokun Tian, Stefan Lattner

Sony Computer Science Laboratories, Paris, France · Queen Mary University of London, UK

Audio Representation Learning Joint-Embedding Predictive Architecture Parameter Estimation Latent Space Optimization

Open paper AskPDF arXiv

Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audio representations are often difficult to optimize, while direct search requires rendering every candidate. We introduce Synth-JEPA, which learns mutually predictive audio and parameter representations from paired synthesizer data. At inference, candidate parameters are scored directly in this learned space, yielding a renderer-free objective whose audio geometry is shaped by parameter correspondences rather than generic audio similarity. We evaluate Synth-JEPA on Surge XT using held-out synthesizer sounds and out-of-domain NSynth and FSD50K targets, against inverse models, direct search, and learned proxy objectives. Synth-JEPA outperforms all baselines in-domain and remains competitive out-of-domain. Its matching quality continues to improve with additional test-time search, allowing compute to be traded for match quality. In pairwise listening tests, listeners preferred Synth-JEPA in 85% of trials overall. Together, these results show that an audio representation with a parameter-induced geometry allows synthesizer sound matching to be approached as an effective renderer-free search problem.

Abstract

Parameter Estimation

Explore similar work

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

FM Synthesizer Audio-Parameter Shared Embeddings

AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text