Exploring the Design Space of Representation Learning for Audio Transformations

Sungho Lee; Marco Martínez-Ramírez; Junghyun Koo; Wei-Hsiang Liao; Kyogu Lee; Yuki Mitsufuji

cs.SDAug 28, 2026

Exploring the Design Space of Representation Learning for Audio Transformations

Authors: Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo, Wei-Hsiang Liao, Kyogu Lee, Yuki Mitsufuji

Organizations: Department of Intelligence and Information, Seoul National University, Seoul, South Korea · Sony AI, Tokyo, Japan · Sony Group Corporation, Tokyo, Japan

Abstract

Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the proposed framework.

	Condition
Configuration	WS	WI	CS	CI	R	Avg
$\mathcal{L}_{T}$	83.8	57.6	46.3	48.8	54.4	58.2
$\mathcal{L}_{P}$	87.7	62.6	43.5	46.6	58.1	59.7
$\mathcal{L}_{y}$	81.8	50.7	26.4	24.4	46.0	45.9
$\mathcal{L}_{T}$ $+$ $\mathcal{L}_{P}$	87.1	64.3	45.4	49.8	58.7	61.1
$\mathcal{L}_{T}$ $+$ $\mathcal{L}_{y}$	86.6	61.5	46.0	49.2	57.8	60.2

Table 1: Retrieval accuracy across source conditions.

Figure 2: Retrieval accuracy across chain lengths.

	Chain estimation										Parameter estimation								Source cls.
Configuration	IoU	Exact	F1	EQ	Filt.	Dyn.	Dist.	Rev.	Del.	Mod.	MAE	EQ	Filt.	Dyn.	Dist.	Rev.	Del.	Mod.	$z_{T}$	$z_{y}$
$\mathcal{L}_{T}$	51.5	16.1	58.3	45.4	76.1	52.1	30.8	56.2	74.0	73.6	17.7	21.5	21.0	18.7	0 5.1	22.5	19.6	15.2	67.2	93.7
$\mathcal{L}_{P}$	53.3	17.8	59.6	47.9	78.0	55.7	29.0	55.5	74.5	76.3	17.5	21.5	20.8	18.5	0 5.3	22.0	19.2	14.9	83.1	93.2
$\mathcal{L}_{y}$	50.5	15.4	57.6	46.7	76.0	50.7	29.1	52.5	73.2	75.4	17.7	21.7	20.9	18.8	0 5.4	22.3	19.8	15.0	93.5	94.9
$\mathcal{L}_{T}$ $+$ $\mathcal{L}_{P}$	53.0	17.3	59.9	49.1	77.3	55.6	30.4	55.3	74.8	76.5	17.5	21.5	20.8	18.6	0 5.3	22.0	19.2	15.0	85.0	94.0
$\mathcal{L}_{T}$ $+$ $\mathcal{L}_{y}$	52.7	16.5	59.6	44.4	76.9	57.7	32.7	55.1	74.6	75.9	17.6	21.5	21.0	18.7	0 5.2	22.3	19.4	15.0	88.9	94.0

Table 3: Downstream task and source classification results with frozen embeddings and MLP probes.

Figure 3: UMAP visualizations of embeddings.

	Blind		Non-blind
Metric	$z_{T}$	$z_{y}$	$\hat{z}_{T}$	$z_{y}$	$z_{T}$
Accuracy (WS)	87.3	87.7	88.2	86.3	98.0
Accuracy (WI)	63.2	62.5	62.3	55.4	94.7
Accuracy (CS)	45.5	36.0	45.2	26.8	88.6
Accuracy (CI)	48.7	40.4	45.6	22.9	85.9
Accuracy (R)	57.3	56.2	57.2	42.6	89.7

Table 4: Embedding and pretraining comparison.

	Condition
Configuration	WS	WI	CI
$z_{T}$	0.390 $\pm$ 0.010	0.575 $\pm$ 0.016	0.647 $\pm$ 0.018
$z_{y}$	0.383 $\pm$ 0.010	0.626 $\pm$ 0.019	0.720 $\pm$ 0.021
AFx-Rep [ 5 ]	0.393 $\pm$ 0.011	0.663 $\pm$ 0.018	0.832 $\pm$ 0.023
Fx-Enc++ [ 6 ]	0.443 $\pm$ 0.013	0.803 $\pm$ 0.027	1.144 $\pm$ 0.037
CLAP [ 1 ]	0.539 $\pm$ 0.013	0.807 $\pm$ 0.020	1.041 $\pm$ 0.028

Table 5: Style transfer results.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

Evaluating Compositional Structure in Audio Representations
Mar 14, 2026Chuyang Chen, Bea Steers, Brian McFee +1Compositional Representation Learning Audio Representation Learning
Music and Audio Research Laboratory, New York University, USA
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Aug 20, 2026Umberto Cappellazzo, Xubo Liu, Stavros Petridis +1Self-Supervised Speech Representation Learning Autoregressive Modeling
Imperial College London · University of Surrey
Probing Spatial Structure in Pretrained Audio Representations
Jun 4, 2026Chuyang Chen, Sivan Ding, Adrian S. Roman +1Audio Representation Learning Representation Probing
Music and Audio Research Laboratory, New York University, USA

Mar 14, 2026J/K move · Enter open · S save

Evaluating Compositional Structure in Audio Representations

Chuyang Chen, Bea Steers, Brian McFee, Juan Bello

Music and Audio Research Laboratory, New York University, USA

Compositional Representation Learning Audio Representation Learning

Open paper AskPDF arXiv

We propose a benchmark for evaluating compositionality in audio representations. Audio compositionality refers to representing sound scenes in terms of constituent sources and attributes, and combining them systematically. While central to auditory perception, this property is largely absent from current evaluation protocols. Our framework adapts ideas from vision and language to audio through two tasks: A-COAT, which tests consistency under additive transformations, and A-TRE, which probes reconstructibility from attribute-level primitives. Both tasks are supported by large synthetic datasets with controlled variation in acoustic attributes, providing the first benchmark of compositional structure in audio embeddings.

Abstract

Audio Representation Learning

Explore similar work

Evaluating Compositional Structure in Audio Representations

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Probing Spatial Structure in Pretrained Audio Representations