TAPE: Learning General Audio Effect Representations

Phillip Long; Yongyi Zang; Qihui Yang; Christodoulos Benetatos; Junfei Liu; Randal Leistikow; Julian McAuley; Perry R. Cook

cs.SDOct 29, 2026

TAPE: Learning General Audio Effect Representations

Authors: Phillip Long, Yongyi Zang, Qihui Yang, Christodoulos Benetatos, Junfei Liu, Randal Leistikow, Julian McAuley, Perry R. Cook

Organizations: University of California, San Diego · Smule Labs

Abstract

Musicians and mixing engineers often can decipher the signal processing chain they hear on a reference recording, then adapt it to new material. We present TAPE (Transformations of Audio Parameter Embeddings), which estimates this processing from the processed reference audio alone. An encoder trained to identify processors in eight major effect families maps the reference to an embedding. A differentiable digital signal processing (DSP) chain decodes the embedding into editable parameters in one forward pass and applies them to a different unprocessed recording. Its modules are equalization, compression, waveshaping, a small residual network, and reverberation. TAPE achieves lower spectral error than DeepAFx-ST, the previous state of the art for single-step regression, and lower spectral error than per-example search over a chain approximating ST-ITO's, while running 275 $\times$ faster than that search. Because the decoder outputs DSP parameters, its chain can also run causally: a C++ implementation with the trained weights processed 128-sample buffers 1.8 $\times$ faster than real time, with no missed deadlines.

Figures & tables

Figure 1: Overview of TAPE. The encoder

f

maps a processed reference

\mathrm{wet}_{A}

to an embedding

z

, and small heads map

z

to the controls and gates of a five-module DSP chain that processes a different recording

\mathrm{dry}_{B}

. In training (dashed), the same plugin configuration

\theta

renders

A

and

B

, and the output is compared with

\mathrm{wet}_{B}

. Information about the source of

A

z

cannot help reconstruct

B

, but no loss removes it from

z

Figure 2: t-SNE of held-out embeddings ( $n=3{,}295$ ), colored by effect family. The single miscellaneous plugin is not plotted.

Figure 3: Controls predicted from

z

for three held-out embeddings (EQ Plugin, Compressor, Reverb). (a) Five-band EQ magnitude response. (b) Module gates (distortion is the waveshaper, NN the residual). (c) RT60 per band.

	Study 1 ( $N=8$ )			Study 3 ( $N=7$ )
Condition	Mean	SD	95% CI	Mean	SD
Adversarially fine-tuned	72.5	23.0	[66.3, 78.5]	72.4	27.3
Unprocessed	47.6	27.0	[38.8, 56.4]	–	–
Reconstruction only	41.3	25.0	[33.0, 50.0]	38.4	24.6

Table 1: Listening results for cross-content transfer (similarity to the reference, 0–100). Means and SDs are over all ratings, and 95% CIs are bootstrap intervals over per-listener means. Scores are for the least-squares variant. Study 1 includes the unprocessed input as a control; Study 3 repeats the main contrast on new pairs.

Method	Objective	Renders	Time	STFT L1	Peak	Peak, 24 kHz	AFx	Wins
Unprocessed $\mathrm{dry}_{B}$	–	0	–	2.335	1.808	1.724	0.894	5
TAPE, one pass	–	1	0.52	1.120	1.391	1.310	0.660	–
TAPE, one pass, TF32 disabled	–	1	0.53	1.177	1.448	1.323	–	–
Search over ST-ITO’s basic chain, full setting
ST-ITO search	AFx-Rep	4,097	143	2.906	2.317	2.150	0.259	2
ST-ITO search	$f$	4,097	205	2.498	1.703	1.580	0.652	5

Table 2: Search at the full setting and the released DeepAFx-ST models (60 held-out pairs, means over pairs). STFT L1 is measured against the withheld

\mathrm{wet}_{B}

at 48 kHz. Peak normalizes output and target to unit peak at 48 kHz, and Peak, 24 kHz first resamples both to 24 kHz and then normalizes each to unit peak. AFx is the cosine distance to

\mathrm{wet}_{A}

in AFx-Rep (not computed for DeepAFx-ST or with TF32 disabled). Wins counts pairs on which a method has a lower STFT L1 (48 kHz, unnormalized) than TAPE. Renders count evaluations of a chain per example; the corrective searches add TAPE’s pass and skip the final render when they keep TAPE’s output. Time is the median wall-clock seconds per five-second clip. ‡ Controls predicted at 24 kHz, rendered on the 48 kHz input. Appendix 20 gives the reduced setting and searches in TAPE’s own space.

Figure 4: Reconstruction error against wall-clock cost (60 pairs). Lines follow each search as its budget grows at the reduced setting: 8 to 128 renders of the basic chain, 8 to 32 latent candidates, or 1 to 16 TTA steps. Solid lines optimize AFx-Rep and dashed lines optimize

f

. Squares (ST-ITO search) and diamonds (TAPE, then chain search) mark the full setting of Table 2 , filled for AFx-Rep and open for

f

. The star is TAPE’s single forward pass, and the dotted line is the unprocessed input.

Buffer	Budget (ms)	Time (s)	RTF	p99 (ms)	Worst (ms)	Missed
64	1.33	3.32	1.50	0.92	1.76	21 of 225,000
128	2.67	2.84	1.76	1.56	2.38	0 of 112,500
256	5.33	2.61	1.92	2.85	3.64	0 of 56,280
512	10.67	2.52	1.99	5.47	6.28	0 of 28,140
1,024	21.33	2.48	2.02	10.72	11.77	0 of 14,100

Table 3: Streaming the trained chain on one CPU core at 48 kHz, over the 60 five-second dry inputs of Section 7.2 , in simulated host callbacks. Time is the mean processing time per clip, excluding the once-per-reference conditioning, and RTF is the clip duration divided by that time. p99 is the 99th percentile of callback time, averaged over clips; Worst is the slowest callback over all clips; Missed counts callbacks that exceeded the budget.

Depth	Objective	Plugin acc. (%)	Config acc. (%)	Val. loss
6	Plugin AM-Softmax	78.7	0.01	0.875
8	Plugin AM-Softmax	72.5	0.00	1.115
6	Config AM-Softmax	63.9	39.0	3.908
8	Config AM-Softmax	55.9	25.6	4.341
6	Hierarchical	44.3	0.89	5.074
8	Hierarchical	46.9	0.57	5.237

Table 4: Encoder depth and training objective.

Figure 5: Module responses of the decoder. From left to right: five-band peaking EQ; compressor gain reduction (small shaded band: softplus versus hard knee); five parallel waveshapers; log-magnitude spectrogram of a parametric reverberation impulse.

Rank	Architecture	Temporal	SI-SDR (dB)	LSD	L1	CLAP Dist.
1	Causal TCN (used)	Yes	$-19.29$	0.871	0.184	0.405
2	Grouped SRU	Yes	$-19.31$	0.894	0.183	0.366
3	S4D hybrid	Yes	$-19.37$	0.968	0.181	0.392
4	AdaLN feed-forward	No	$-19.72$	0.940	0.181	0.379
Ranks 5–23 (19 architectures): SI-SDR from $-20.52$ to $-27.04$ dB

Table 5: Residual architecture comparison (23 pairs; the four highest-ranked of 23 architectures). Temporal means recurrent, convolutional-over-time, or attention-over-time capacity.

Condition	STFT L1	Time L1	CLAP Dist.
Separate	2.314	0.117	0.333
Joint finetune	1.224	0.116	0.308
Joint scratch	1.052	0.132	0.208

Table 6: Joint versus separate parameter-head training (200 validation-split clips, same-content reconstruction). CLAP Dist. is

1-\cos

between CLAP embeddings of prediction and reference.

Condition	Mean	SD	95% CI
Joint finetune	66.4	25.6	[57.4, 74.2]
Separate	51.1	26.1	[41.4, 60.2]
Unprocessed	49.7	26.0	[41.1, 57.9]
Joint scratch	48.8	26.5	[39.0, 58.6]

Table 7: Listening results for the curricula of the DSP chain on same-content validation clips (Study 2,

N=6

; similarity to the reference, 0–100). Means and SDs are over all ratings, and 95% CIs are bootstrap intervals over per-listener means.

Condition	STFT L1 (mean)	STFT L1 (median)	Time L1
Parametric prior	4.451	2.873	0.00198
Refined impulse	4.927	4.158	0.00099

Table 8: Parametric reverberation prior versus the refined impulse (50 clips). Each clip is a validation-split dry recording convolved with an impulse response from the reverberation plugins;

z

is extracted from that convolved clip, so this is same-content reconstruction, and the impulse set is not split. STFT L1 and time L1 compare the predicted impulse response with the true one, not the rendered audio. Study 4, which rated the rendered clips, detected no significant listening difference.

Family	STFT L1	Time L1	CLAP Dist.	$N$
Preamplification	0.713	0.057	0.028	1
Dynamics	0.927	0.027	0.073	11
Modulation	1.010	0.058	0.341	6
Reverberation	1.015	0.391	0.184	3
Saturation/Distortion	1.149	0.228	0.133	2
Chain	1.176	0.057	0.308	27

Table 9: Per-family cross-content reconstruction after adversarial fine-tuning (60 pairs, R3GAN variant). Chain is an evaluation grouping for held-out configurations that route several processors in series, not one of the eight plugin families of Table 10 . One pair with an unresolved category is included in the global row only.

Family	Plugins	Representative types
Dynamics / Compression	19	Feed-forward, vari-mu, optical, bus comp
Equalization	14	Parametric, graphic, vintage transformer
Modulation	9	Chorus, phaser, flanger, tremolo
Preamplification	9	Channel strip, transformer, tube pre
Reverberation	8	Algorithmic, plate, hall, room
Analog tape	5	Tape saturation, wow/flutter

Table 10: Plugin inventory by family (68 total).

ID	Display name	Cat.	$P_{\mathrm{eff}}$
4k_eq	4K EQ	eq	20
airwindows_chorus	Airwindows Consolidated (Chorus)	chorus	10
airwindows_consolidated_compressor	Airwindows Consolidated (Compressor)	compressor	10
airwindows_consolidated_eq	Airwindows Consolidated (EQ Algorithms)	eq	10
airwindows_consolidated_modulation	Airwindows Consolidated (Modulation Algorithms)	modulation	10
airwindows_consolidated_preamp	Airwindows Consolidated (Console/Preamp)	preamp	10

Table 11: Sixty-eight training plugins with stable identifiers, display names, condensed category tags, and differentiable control counts (

P_{\mathrm{eff}}

Component	Specification (8-layer; 6-layer omits two blocks)
Frontend	Hann FFT, $N_{\mathrm{FFT}}=2048$ , hop $960$ at $48\,\mathrm{kHz}$ ; $80$ mel bins from $0$ to $24\,\mathrm{kHz}$ , decibel scale with an $80$ dB floor.
Stem	Convolutional downsampler, $8\times$ temporal shrink, width $128$ .
Conformer	Width $256$ , depth $N\in\{8,6\}$ , 4 heads, depthwise conv kernel $9$ , FFN $\times 4$ , rotary/content mix $50$ : $50$ , DropPath $0.1$ .
Pooling	Learned query token, then an affine map to $z\in\mathbb{R}^{256}$ .
Size	About $5$ to $6\times 10^{6}$ parameters for $N=8$ ; $3$ to $4\times 10^{6}$ for $N=6$ .

Table 12: Encoder front end and Conformer stack.

Stage	Ranges
EQ	Gain $\pm 12$ dB; $Q\in[0.1,10]$ ; frequency $20\,\mathrm{Hz}$ to $20\,\mathrm{kHz}$
Compressor	Threshold $[-60,0]$ dBFS; ratio $[1,20]$ ; attack $[1,100]\,\mathrm{ms}$ ; release $[10,1000]\,\mathrm{ms}$
Distortion	Drive up to $+36$ dB; tone low-pass $500\,\mathrm{Hz}$ to $20\,\mathrm{kHz}$ ; makeup $-12$ to $+6$ dB

Table 13: Actuator spans after sigmoidal remapping.

$k$	Name	$s_{k}(x)$
1	Tanh saturation	$g_{o}\tanh(g_{i}x+b)$
2	Hard clip	$g_{o}\,\mathrm{clip}(g_{i}x+b,-1,1)$
3	Asymmetric tube	$g_{o}\tanh\!\big(g_{\pm}(x+b)\big)$ , $g_{+}$ if $x+b\geq 0$ , else $g_{-}$
4	Diode	$g_{o}\,\dfrac{u}{\sqrt{u^{2}+\varepsilon}}\big(1-e^{-\kappa\|u\|}\big)$ , $u=x+b$
5	Sine wavefolder	$g_{o}\sin(g_{i}x+b)$

Table 14: Waveshaper curves mixed by the softmax in Equation 6 .

Search over ST-ITO’s basic chain
Method	Objective	Renders	Time	STFT L1	$\Delta$ [95% CI]	Peak	AFx	$f$
Unprocessed $\mathrm{dry}_{B}$	–	0	–	2.335	$+1.215$ [ $+0.929$ , $+1.537$ ]	1.808	0.894	0.797
TAPE, one pass	–	1	0.52	1.120	–	1.391	0.660	0.372
ST-ITO search	AFx-Rep	128	4.2	3.278	$+2.158$ [ $+1.687$ , $+2.709$ ]	2.395	0.394	0.673
ST-ITO search	$f$	128	6.1	3.143	$+2.023$ [ $+1.689$ , $+2.382$ ]	2.255	0.819	0.510 †
Random search	AFx-Rep	128	4.1	3.033	$+1.913$ [ $+1.545$ , $+2.312$ ]	2.288	0.508	0.733

Table 15: One forward pass versus per-example search at the reduced setting (60 held-out pairs; means over pairs). STFT L1 is measured against the withheld

\mathrm{wet}_{B}

; Peak normalizes output and target to unit peak first.

\Delta

is the change in STFT L1 relative to TAPE, with a 95% bootstrap interval. AFx and

f

are cosine distances to

\mathrm{wet}_{A}

in AFx-Rep and in our encoder; each search lowers the distance it optimizes. Renders count full evaluations of the basic chain or of

g

per example; TTA also takes 16 backward passes. Time is the median wall-clock seconds per five-second clip. † Searches of the basic chain with

f

optimize

f

on peak-normalized audio, which reaches 0.29 and 0.27; the column shows the unnormalized distance.

		48 kHz				24 kHz, peak-normalized
				ST-ITO search				DeepAFx-ST, music
Family	$N$	Dry	TAPE	AFx-Rep	$f$	Dry	TAPE	Official	48 kHz
Dynamics	11	2.087	0.927	2.448	2.426	1.280	0.947	1.527	1.309
Equalization	9	2.358	1.299	3.917	2.160	2.572	1.842	2.767	2.554
Modulation	6	1.763	1.010	2.762	2.327	0.967	1.043	1.358	1.210
Reverberation	3	2.055	1.014	2.034	1.362	1.579	0.914	2.012	1.728

Table 16: Full-setting search and DeepAFx-ST by effect family (60 pairs; lower is better). Left: STFT L1 at 48 kHz, with 4,097 renders of the basic chain per search. Right: peak-normalized STFT L1 after resampling output and target to 24 kHz, for the official recipe and the 48 kHz rendering ablation of the music model. One pair with an unresolved category is omitted.

Family	$N$	Dry	TAPE	AFx-Rep	$f$	AFx-Rep	$f$	Oracle
				ST-ITO search		Latent CMA-ES	Latent TTA
Dynamics	11	2.087	0.927	2.997	3.403	0.948	0.927	0.743
Equalization	9	2.358	1.299	4.030	3.141	1.390	1.412	1.047
Modulation	6	1.763	1.010	3.037	3.000	1.014	1.242	0.877
Reverberation	3	2.055	1.014	1.929	1.922	1.037	1.017	0.955
Saturation/Distortion	2	1.547	1.149	1.573	2.287	1.152	0.943	0.923

Table 17: STFT L1 by effect family for the comparison of Table 15 (60 pairs; lower is better). Searches use 128 renders of the basic chain, 32 latent candidates, or 16 TTA steps. One pair with an unresolved category is omitted.

Stage	Median	5th percentile
Equalizer (biquad cascade vs. FFT-grid response)	54.4	29.7
Compressor (single precision)	105.2	82.3
Tone filter (one-pole vs. zero-phase)	8.2	7.0
Mean removal (DC blocker vs. clip mean)	19.7	15.2
Waveshaper stage, all changes	12.6	6.1

Table 18: Signal-to-error ratio (dB) between streamed stages and the offline formulation used in training, over 100 random settings on 5 s clips.

Differentiable Digital Signal Processing

See all 9 papers

PEACE: Joint Embeddings of DSP Effects Code and Audio
Oct 2, 2026David Braun, Adam FinkelsteinAudio Representation Learning Music Information Retrieval
Princeton University
TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control
Mar 10, 2026Shihao He, Yihan Xia, Fang Liu +2Music Information Retrieval Audio Representation Learning
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
InstructFX2FX: A Multi-Turn Text-to-Effect System for Sequential Audio Effect Refinement
Jun 20, 2026Song-Ze Yu, Milan Liessens Dujardin, Yuxuan Cai +4Text-to-Audio Generation Large Language Model-Guided Optimization
Center for New Music and Audio Technologies (CNMAT) University of California, Berkeley

Oct 2, 2026J/K move · Enter open · S save

PEACE: Joint Embeddings of DSP Effects Code and Audio

David Braun, Adam Finkelstein

Princeton University

Audio Representation Learning Music Information Retrieval

Open paper AskPDF arXiv

This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.

Abstract

Differentiable Digital Signal Processing

Explore similar work

PEACE: Joint Embeddings of DSP Effects Code and Audio

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

InstructFX2FX: A Multi-Turn Text-to-Effect System for Sequential Audio Effect Refinement