Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

Ben Hayes; Haokun Tian; Stefan Lattner

cs.SDSep 25, 2026

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

Authors: Ben Hayes, Haokun Tian, Stefan Lattner

Organizations: Sony Computer Science Laboratories, Paris, France · Queen Mary University of London, UK

Abstract

Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audio representations are often difficult to optimize, while direct search requires rendering every candidate. We introduce Synth-JEPA, which learns mutually predictive audio and parameter representations from paired synthesizer data. At inference, candidate parameters are scored directly in this learned space, yielding a renderer-free objective whose audio geometry is shaped by parameter correspondences rather than generic audio similarity. We evaluate Synth-JEPA on Surge XT using held-out synthesizer sounds and out-of-domain NSynth and FSD50K targets, against inverse models, direct search, and learned proxy objectives. Synth-JEPA outperforms all baselines in-domain and remains competitive out-of-domain. Its matching quality continues to improve with additional test-time search, allowing compute to be traded for match quality. In pairwise listening tests, listeners preferred Synth-JEPA in 85% of trials overall. Together, these results show that an audio representation with a parameter-induced geometry allows synthesizer sound matching to be approached as an effective renderer-free search problem.

Figures & tables

Figure 1: Synth-JEPA training and inference. $\mathcal{E}_{a}$ and $\mathcal{E}_{p}$ are audio and parameter encoders, while $f_{a\rightarrow p}$ and $f_{p\rightarrow a}$ are cross-domain predictors. At inference, target audio is encoded once and candidate parameters are optimized directly against the learned objective, without rendering candidate audio during search.

		In-domain presets			NSynth			Freesound (FSD50K)
Family	Method	MSS $\downarrow$	wMFCC $\downarrow$	CLAP $\uparrow$	MSS $\downarrow$	wMFCC $\downarrow$	CLAP $\uparrow$	MSS $\downarrow$	wMFCC $\downarrow$	CLAP $\uparrow$	Renders / target	Parameters
Synth-JEPA	SIGReg	8.44	11.07	0.793	10.25	15.54	0.465	13.59	14.81	0.250	0	53.3M
Synth-JEPA	EMA Teacher	11.52	15.29	0.625	14.02	19.33	0.323	15.39	17.33	0.179	0	53.3M
Amortized	AST Regression	17.94	23.16	0.499	28.06	30.64	0.104	21.48	26.56	0.086	0	25.0M
Amortized	Flow Matching	12.75	14.78	0.707	16.67	21.24	0.356	17.94	18.84	0.204	0	44.0M
Renderer search	Log-Mel distance	15.86	12.17	0.553	17.81	14.10	0.199	14.55	11.61	0.178	2048	—

Table 1: Sound matching results for in-domain Surge XT presets drawn from the parameter prior, and for out-of-domain NSynth and FSD50K targets, 1 024 targets per set. Bold and underlining mark the best and second-best result in each column.

Figure 2: Sound matching performance when scaling the number of model evaluations (search budget / ODE solver steps). Wall-clock time reported per-target on one RTX 3090. Labels indicate number of model/objective evaluations per target.

Figure 3: Sound matching performance with inference-time best-of-

k

selection of rendered audio.

Figure 4: Pairwise listening tests. Participants compared a reference to two sound matching attempts and selected the closest. Points show proportion of trials in which Synth-JEPA was preferred, with 95% confidence intervals computed by bootstrap resampling of participants.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

FM Synthesizer Audio-Parameter Shared Embeddings
Aug 18, 2026David Braun, Adam FinkelsteinMusic Information Retrieval Graph Neural Networks
Dept. of Computer Science Princeton University Princeton, NJ, USA
DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning
Aug 4, 2026Tristan Wu, Daniel Chin, Junan Zhang +3Diffusion-Based Inverse Problems Diffusion Models
Computational Media and Art, The Hong Kong University of Science and Technology (Guangzhou) · New York University Shanghai · The Chinese University of Hong Kong, Shenzhen +2
Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance
Aug 27, 2026Amir Salimi, Daniel Penner, Kalvin Eng +2Audio Generation
University of Alberta

Aug 18, 2026J/K move · Enter open · S save

FM Synthesizer Audio-Parameter Shared Embeddings

David Braun, Adam Finkelstein

Dept. of Computer Science Princeton University Princeton, NJ, USA

Music Information Retrieval Graph Neural Networks

Open paper AskPDF arXiv

Given a target sound, finding the synthesizer preset that best reproduces it remains a core problem in sound design. Existing methods treat synthesis parameters as flat vectors, discarding the signal routing and parameter interactions that produce audio. We make two contributions. First, to learn a representation of parameters including their signal routing, we design a graph neural network whose message passing structure imitates FM signal processing. Second, we adapt the multimodal objective from SLAP to learn joint embeddings of audio and FM synthesizer parameters, enabling preset retrieval from a gallery. We focus on the Yamaha DX7, where six identical sinusoid operators interact according to one of 32 routing topologies. Our graph encoder's message passing weights are shared across all nodes and layers, enabling processing of arbitrary topologies of any size. When every topology is seen during training, the DX7-GNN and two baselines achieve strong audio-to-preset retrieval. When some topologies are held out for testing, the DX7-GNN substantially outperforms both baselines despite having the fewest parameters. Our ablations further support the claim that imitating FM signal flow in a parameter encoder improves generalization to unseen topologies.

Abstract

Audio Representation Learning

Explore similar work

FM Synthesizer Audio-Parameter Shared Embeddings

DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning

Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance