Organizations: Department of Intelligence and Information, Seoul National University, Seoul, South Korea · Sony AI, Tokyo, Japan · Sony Group Corporation, Tokyo, Japan
Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.
Figures & tables
Figure 1: Overview of the proposed framework.
Condition
Configuration
WS
WI
CS
CI
R
Avg
LT
83.8
57.6
46.3
48.8
54.4
58.2
LP
87.7
62.6
43.5
46.6
58.1
59.7
Ly
81.8
50.7
26.4
24.4
46.0
45.9
LT+LP
87.1
64.3
45.4
49.8
58.7
61.1
LT+Ly
86.6
61.5
46.0
49.2
57.8
60.2
Table 1: Retrieval accuracy across source conditions.
Figure 2: Retrieval accuracy across chain lengths.
Chain estimation
Parameter estimation
Source cls.
Configuration
IoU
Exact
F1
EQ
Filt.
Dyn.
Dist.
Rev.
Del.
Mod.
MAE
EQ
Filt.
Dyn.
Dist.
Rev.
Del.
Mod.
zT
zy
LT
51.5
16.1
58.3
45.4
76.1
52.1
30.8
56.2
74.0
73.6
17.7
21.5
21.0
18.7
0 5.1
22.5
19.6
15.2
67.2
93.7
LP
53.3
17.8
59.6
47.9
78.0
55.7
29.0
55.5
74.5
76.3
17.5
21.5
20.8
18.5
0 5.3
22.0
19.2
14.9
83.1
93.2
Ly
50.5
15.4
57.6
46.7
76.0
50.7
29.1
52.5
73.2
75.4
17.7
21.7
20.9
18.8
0 5.4
22.3
19.8
15.0
93.5
94.9
LT+LP
53.0
17.3
59.9
49.1
77.3
55.6
30.4
55.3
74.8
76.5
17.5
21.5
20.8
18.6
0 5.3
22.0
19.2
15.0
85.0
94.0
LT+Ly
52.7
16.5
59.6
44.4
76.9
57.7
32.7
55.1
74.6
75.9
17.6
21.5
21.0
18.7
0 5.2
22.3
19.4
15.0
88.9
94.0
Table 3: Downstream task and source classification results with frozen embeddings and MLP probes.
We propose a benchmark for evaluating compositionality in audio representations. Audio compositionality refers to representing sound scenes in terms of constituent sources and attributes, and combining them systematically. While central to auditory perception, this property is largely absent from current evaluation protocols. Our framework adapts ideas from vision and language to audio through two tasks: A-COAT, which tests consistency under additive transformations, and A-TRE, which probes reconstructibility from attribute-level primitives. Both tasks are supported by large synthetic datasets with controlled variation in acoustic attributes, providing the first benchmark of compositional structure in audio embeddings.
Chuyang Chen, Bea Steers, Brian McFee +1
Music and Audio Research Laboratory, New York University, USA
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Pretrained spatial audio encoders are increasingly used as general-purpose representations for perceptual tasks, yet their spatial encoding capabilities remain poorly understood. We introduce the Spatial Audio Representation Learning (SARL) benchmark, a controlled framework for evaluating spatial information in pretrained audio models. SARL probes source-level factors (azimuth, elevation, distance, class) and room-level factors (RT60, volume, shape). Experiments across diverse encoders reveal three patterns: input configuration and training paradigm shape spatial encoding; source factors are consistently easier to decode than room factors; and sensitivity analysis under controlled perturbations shows heterogeneous responses to source and room variation. These results reveal systematic biases in current pretrained audio representations. SARL is released as an open-source benchmark for reproducible evaluation of spatial audio representations.
Chuyang Chen, Sivan Ding, Adrian S. Roman +1
Music and Audio Research Laboratory, New York University, USA