This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.
Figures & tables
Figure 1: SLAP architecture. Each modality’s online branch pairs encoder-projector EM (producing yM then zM ) with predictor PM to produce qM . The EMA target branch EˉM produces stop-gradient zˉM with no predictor. The L terms show loss gradients flowing through the predictors into the online branches, not the EMA targets.
Figure 2: BoxGraph of a stereo effect where reverb is defined in the library. Each node shows its I/O signature ( X→Y means X inputs, Y outputs), edge labels indicate argument positions, and a composition node’s signature is fully determined by the operator and children’s signatures. The graph is not a traditional “audio processor graph” but instead a composition of functional transformations, each resulting in a new function-like box.
Audio Encoder
Code Encoder
Audio → Code
Code → Audio
Gap ( ↓ )
Separability ( ↓ )
Recall ( ↑ )
Norm. rank ( ↓ )
Recall ( ↑ )
Norm. rank ( ↓ )
@1
@10
Med.
Mean
@1
@10
Med.
Mean
Dist.
Mod.
Source
AFx-Rep (scratch)
T5 (scratch)
41.2
71.9
0.02
1.5
41.3
71.4
0.02
1.2
0.07
72.2
56.2
AFx-Rep (scratch)
T5 (fine-tune)
50.6
78.1
0.00
1.1
48.4
76.3
0.02
1.0
0.07
74.0
57.0
AFx-Rep (scratch)
BoxGraph
48.5
77.7
0.02
1.0
50.8
77.5
0.00
0.8
0.06
80.4
58.5
AFx-Rep (fine-tune)
T5 (scratch)
46.9
74.9
0.02
2.5
46.9
76.1
0.02
1.5
0.11
81.6
54.4
Table 1: Cross-modal retrieval (for metrics, see § 4.3 ). Dist. : Euclidean distance between the qA and qC centroids. Mod. and Source are linear-probe accuracies (50% = linearly indistinguishable, 100% = perfectly separable). Bold indicates best per column per section. Recall, rank, and separability in %.
R@1 ( ↑ )
R@10 ( ↑ )
Effect
Box
T5
Box
T5
AutoPan
20.6
19.5
62.8
60.7
Chorus
27.7
34.5
74.4
78.3
Compressor
2.5
2.6
16.8
14.6
Delay
31.2
35.1
71.4
74.2
Distortion
4.6
3.7
21.0
20.8
Table 2: Per-effect average retrieval in a gallery of size 1024, in %. Chance is ∼ 0.1% R@1 and ∼ 1.0% R@10.
Figure 3: Retrieval vs. chain length N (gallery size 1024). Shaded region marks chain lengths seen during training.
R@1 ( ↑ )
R@10 ( ↑ )
Model
Code
yA
qA
yA
qA
PEACE scratch
BoxGraph
19.4
23.1
50.6
53.5
PEACE scratch
T5
18.4
19.1
46.5
48.3
PEACE frozen
BoxGraph
8.1
13.9
28.7
38.6
PEACE frozen
T5
8.1
13.4
28.7
36.6
AFx-Rep
–
8.4
–
28.3
–
Table 3: RIR retrieval (gallery 1,831), in %. yA = encoder latent (HTS-AT for LAION-CLAP); qA = predictor output, or the projector output zA for Fx-Encoder++ and LAION-CLAP, which lack a predictor.
Exact ( ↑ )
Order-tolerant ( ↑ )
Method
R@1
R@10
R@1
R@10
Retrieval (no chain supervision)
AFx-Rep frozen + Box
12.6
48.7
19.5
56.8
AFx-Rep scratch + Box
17.4
61.4
27.2
67.4
AFx-Rep scratch + T5
12.4
43.2
20.7
49.1
Supervised classifier (cross-entropy)
Table 4: Chain topology retrieval in % from 420 ordered pairs of distinct effects with all parameter values masked. Chance is 0.24% R@1 and 2.4% R@10. Supervised rows use only an audio encoder and average three seeds, which differ by as much as 4.9 points in R@1. Order-tolerant counts a retrieval as correct if the top- K contains any chain with the same set of effects regardless of order.
Digital audio workstations expose rich effect chains, yet a semantic gap remains between perceptual user intent and low-level signal-processing parameters. We study retrieval-grounded audio effect control, where the output is an editable plugin configuration rather than a finalized waveform. Our focus is Texture Resonance Retrieval (TRR), an audio representation built from Gram matrices of projected mid-level Wav2Vec2 activations. This design preserves texture-relevant co-activation structure. We evaluate TRR on a guitar-effects benchmark with 1,063 candidate presets and 204 queries. The evaluation follows Protocol-A, a cross-validation scheme that prevents train-test leakage. We compare TRR against CLAP and internal retrieval baselines (Wav2Vec-RAG, Text-RAG, FeatureNN-RAG), using min-max normalized metrics grounded in physical DSP parameter ranges. Ablation studies validate TRR's core design choices: projection dimensionality, layer selection, and projection type. A near-duplicate sensitivity analysis confirms that results are robust to trivial knowledge-base matches. TRR achieves the lowest normalized parameter error among evaluated methods. A multiple-stimulus listening study with 26 participants provides complementary perceptual evidence. We interpret these results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.
Shihao He, Yihan Xia, Fang Liu +2
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
Stem retrieval, the task of matching missing stems to a given audio submix, is a key challenge currently limited by models that discard temporal information. We introduce PHALAR, a contrastive framework achieving a relative accuracy increase of up to ≈70% over the state-of-the-art while requiring <50% of the parameters and a 7× training speedup. By utilizing a Learned Spectral Pooling layer and a complex-valued head, PHALAR enforces pitch-equivariant and phase-equivariant biases. PHALAR establishes new retrieval state-of-the-art across MoisesDB, Slakh, and ChocoChorales, correlating significantly higher with human coherence judgment than semantic baselines. Finally, zero-shot beat tracking and linear chord probing confirm that PHALAR captures robust musical structures beyond the retrieval task.
Davide Marincione, Michele Mancusi, Giorgio Strano +4
Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.
This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.