PEACE: Joint Embeddings of DSP Effects Code and Audio

David Braun; Adam Finkelstein

cs.SDOct 2, 2026

PEACE: Joint Embeddings of DSP Effects Code and Audio

Authors: David Braun, Adam Finkelstein

Organizations: Princeton University

Abstract

This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.

Figures & tables

Figure 1: SLAP architecture. Each modality’s online branch pairs encoder-projector $\mathcal{E}_{M}$ (producing $y_{M}$ then $z_{M}$ ) with predictor $\mathcal{P}_{M}$ to produce $q_{M}$ . The EMA target branch $\bar{\mathcal{E}}_{M}$ produces stop-gradient $\bar{z}_{M}$ with no predictor. The $\mathcal{L}$ terms show loss gradients flowing through the predictors into the online branches, not the EMA targets.

Figure 2: BoxGraph of a stereo effect where reverb is defined in the library. Each node shows its I/O signature (

X{\to}Y

means

X

inputs,

Y

outputs), edge labels indicate argument positions, and a composition node’s signature is fully determined by the operator and children’s signatures. The graph is not a traditional “audio processor graph” but instead a composition of functional transformations, each resulting in a new function-like box.

Audio Encoder	Code Encoder	Audio $\rightarrow$ Code				Code $\rightarrow$ Audio				Gap ( $\downarrow$ )	Separability ( $\downarrow$ )
		Recall ( $\uparrow$ )		Norm. rank ( $\downarrow$ )		Recall ( $\uparrow$ )		Norm. rank ( $\downarrow$ )
		@1	@10	Med.	Mean	@1	@10	Med.	Mean	Dist.	Mod.	Source
AFx-Rep (scratch)	T5 (scratch)	41.2	71.9	0.02	1.5	41.3	71.4	0.02	1.2	0.07	72.2	56.2
AFx-Rep (scratch)	T5 (fine-tune)	50.6	78.1	0.00	1.1	48.4	76.3	0.02	1.0	0.07	74.0	57.0
AFx-Rep (scratch)	BoxGraph	48.5	77.7	0.02	1.0	50.8	77.5	0.00	0.8	0.06	80.4	58.5
AFx-Rep (fine-tune)	T5 (scratch)	46.9	74.9	0.02	2.5	46.9	76.1	0.02	1.5	0.11	81.6	54.4

Table 1: Cross-modal retrieval (for metrics, see § 4.3 ). Dist. : Euclidean distance between the

q_{A}

and

q_{C}

centroids. Mod. and Source are linear-probe accuracies (50% = linearly indistinguishable, 100% = perfectly separable). Bold indicates best per column per section. Recall, rank, and separability in %.

	R@1 ( $\uparrow$ )		R@10 ( $\uparrow$ )
Effect	Box	T5	Box	T5
AutoPan	20.6	19.5	62.8	60.7
Chorus	27.7	34.5	74.4	78.3
Compressor	2.5	2.6	16.8	14.6
Delay	31.2	35.1	71.4	74.2
Distortion	4.6	3.7	21.0	20.8

Table 2: Per-effect average retrieval in a gallery of size 1024, in %. Chance is

\sim

0.1% R@1 and

\sim

1.0% R@10.

Figure 3: Retrieval vs. chain length

N

(gallery size 1024). Shaded region marks chain lengths seen during training.

		R@1 ( $\uparrow$ )		R@10 ( $\uparrow$ )
Model	Code	$y_{A}$	$q_{A}$	$y_{A}$	$q_{A}$
PEACE scratch	BoxGraph	19.4	23.1	50.6	53.5
PEACE scratch	T5	18.4	19.1	46.5	48.3
PEACE frozen	BoxGraph	8.1	13.9	28.7	38.6
PEACE frozen	T5	8.1	13.4	28.7	36.6
AFx-Rep	–	8.4	–	28.3	–

Table 3: RIR retrieval (gallery 1,831), in %.

y_{A}

= encoder latent (HTS-AT for LAION-CLAP);

q_{A}

= predictor output, or the projector output

z_{A}

for Fx-Encoder++ and LAION-CLAP, which lack a predictor.

Retrieval (no chain supervision)
	Exact ( $\uparrow$ )		Order-tolerant ( $\uparrow$ )
Method	R@1	R@10	R@1	R@10
AFx-Rep frozen + Box	12.6	48.7	19.5	56.8
AFx-Rep scratch + Box	17.4	61.4	27.2	67.4
AFx-Rep scratch + T5	12.4	43.2	20.7	49.1
Supervised classifier (cross-entropy)

Table 4: Chain topology retrieval in % from 420 ordered pairs of distinct effects with all parameter values masked. Chance is 0.24% R@1 and 2.4% R@10. Supervised rows use only an audio encoder and average three seeds, which differ by as much as 4.9 points in R@1. Order-tolerant counts a retrieval as correct if the top-

K

contains any chain with the same set of effects regardless of order.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 207 papers

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control
Mar 10, 2026Shihao He, Yihan Xia, Fang Liu +2Music Information Retrieval Audio Representation Learning
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
PHALAR: Phasors for Learned Musical Audio Representations
May 5, 2026Davide Marincione, Michele Mancusi, Giorgio Strano +4Audio Representation Learning Representation Learning
Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.
DashengTokenizer: One layer is enough for unified audio understanding and generation
Feb 27, 2026Heinrich Dinkel, Xingwei Sun, Gang Li +8Audio Tokenization Audio Representation Learning
MiLM Plus, Xiaomi Inc., Beijing, China

Mar 10, 2026J/K move · Enter open · S save

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

Shihao He, Yihan Xia, Fang Liu, Taotao Wang+1

College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China

Music Information Retrieval Audio Representation Learning

Open paper AskPDF arXiv

Digital audio workstations expose rich effect chains, yet a semantic gap remains between perceptual user intent and low-level signal-processing parameters. We study retrieval-grounded audio effect control, where the output is an editable plugin configuration rather than a finalized waveform. Our focus is Texture Resonance Retrieval (TRR), an audio representation built from Gram matrices of projected mid-level Wav2Vec2 activations. This design preserves texture-relevant co-activation structure. We evaluate TRR on a guitar-effects benchmark with 1,063 candidate presets and 204 queries. The evaluation follows Protocol-A, a cross-validation scheme that prevents train-test leakage. We compare TRR against CLAP and internal retrieval baselines (Wav2Vec-RAG, Text-RAG, FeatureNN-RAG), using min-max normalized metrics grounded in physical DSP parameter ranges. Ablation studies validate TRR's core design choices: projection dimensionality, layer selection, and projection type. A near-duplicate sensitivity analysis confirms that results are robust to trivial knowledge-base matches. TRR achieves the lowest normalized parameter error among evaluated methods. A multiple-stimulus listening study with 26 participants provides complementary perceptual evidence. We interpret these results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.

Abstract

Audio Representation Learning

Explore similar work

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

PHALAR: Phasors for Learned Musical Audio Representations

DashengTokenizer: One layer is enough for unified audio understanding and generation