Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audio representations are often difficult to optimize, while direct search requires rendering every candidate. We introduce Synth-JEPA, which learns mutually predictive audio and parameter representations from paired synthesizer data. At inference, candidate parameters are scored directly in this learned space, yielding a renderer-free objective whose audio geometry is shaped by parameter correspondences rather than generic audio similarity. We evaluate Synth-JEPA on Surge XT using held-out synthesizer sounds and out-of-domain NSynth and FSD50K targets, against inverse models, direct search, and learned proxy objectives. Synth-JEPA outperforms all baselines in-domain and remains competitive out-of-domain. Its matching quality continues to improve with additional test-time search, allowing compute to be traded for match quality. In pairwise listening tests, listeners preferred Synth-JEPA in 85% of trials overall. Together, these results show that an audio representation with a parameter-induced geometry allows synthesizer sound matching to be approached as an effective renderer-free search problem.
Figures & tables
Figure 1: Synth-JEPA training and inference. Ea and Ep are audio and parameter encoders, while fa→p and fp→a are cross-domain predictors. At inference, target audio is encoded once and candidate parameters are optimized directly against the learned objective, without rendering candidate audio during search.
In-domain presets
NSynth
Freesound (FSD50K)
Family
Method
MSS ↓
wMFCC ↓
CLAP ↑
MSS ↓
wMFCC ↓
CLAP ↑
MSS ↓
wMFCC ↓
CLAP ↑
Renders / target
Parameters
Synth-JEPA
SIGReg
8.44
11.07
0.793
10.25
15.54
0.465
13.59
14.81
0.250
0
53.3M
EMA Teacher
11.52
15.29
0.625
14.02
19.33
0.323
15.39
17.33
0.179
0
53.3M
Amortized
AST Regression
17.94
23.16
0.499
28.06
30.64
0.104
21.48
26.56
0.086
0
25.0M
Flow Matching
12.75
14.78
0.707
16.67
21.24
0.356
17.94
18.84
0.204
0
44.0M
Renderer search
Log-Mel distance
15.86
12.17
0.553
17.81
14.10
0.199
14.55
11.61
0.178
2048
—
Table 1: Sound matching results for in-domain Surge XT presets drawn from the parameter prior, and for out-of-domain NSynth and FSD50K targets, 1 024 targets per set. Bold and underlining mark the best and second-best result in each column.
Figure 2: Sound matching performance when scaling the number of model evaluations (search budget / ODE solver steps). Wall-clock time reported per-target on one RTX 3090. Labels indicate number of model/objective evaluations per target.
Figure 3: Sound matching performance with inference-time best-of- k selection of rendered audio.
Figure 4: Pairwise listening tests. Participants compared a reference to two sound matching attempts and selected the closest. Points show proportion of trials in which Synth-JEPA was preferred, with 95% confidence intervals computed by bootstrap resampling of participants.
Given a target sound, finding the synthesizer preset that best reproduces it remains a core problem in sound design. Existing methods treat synthesis parameters as flat vectors, discarding the signal routing and parameter interactions that produce audio. We make two contributions. First, to learn a representation of parameters including their signal routing, we design a graph neural network whose message passing structure imitates FM signal processing. Second, we adapt the multimodal objective from SLAP to learn joint embeddings of audio and FM synthesizer parameters, enabling preset retrieval from a gallery. We focus on the Yamaha DX7, where six identical sinusoid operators interact according to one of 32 routing topologies. Our graph encoder's message passing weights are shared across all nodes and layers, enabling processing of arbitrary topologies of any size. When every topology is seen during training, the DX7-GNN and two baselines achieve strong audio-to-preset retrieval. When some topologies are held out for testing, the DX7-GNN substantially outperforms both baselines despite having the fewest parameters. Our ablations further support the claim that imitating FM signal flow in a parameter encoder improves generalization to unseen topologies.
David Braun, Adam Finkelstein
Dept. of Computer Science Princeton University Princeton, NJ, USA
Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.
Tristan Wu, Daniel Chin, Junan Zhang +3
Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +2
In out-of-domain (OOD) sound-matching, a synthesizer is optimized to mimic a sound it did not generate. OOD evaluation of loss functions is underexplored in part because the standard "parameter loss" metric requires a shared parameter space between target and imitator, which OOD settings lack. We introduce Partial Parameter Distance (PPD), which applies parameter loss only to the critical parameters that mismatched synthesizers share (e.g., filter cutoffs), enabling automatically evaluated OOD experiments; we verify its results with blinded listening tests. Across seven scenarios involving band-pass filtering, amplitude modulation, and pitch-bending, we evaluate four differentiable loss functions (SIMSE_Spec, L1_Spec, JTFS, DTW_Envelope). Loss-function effectiveness remains tightly coupled to the method of synthesis: SIMSE_Spec excels at filter-cutoff recovery, DTW_Envelope at amplitude-modulation recovery, and JTFS at smooth pitch trajectories. Parameter-based evaluation agrees with listening tests on the top-ranked loss function in five of seven scenarios, demonstrating its utility as a diagnostic tool.