Evaluating Compositional Structure in Audio Representations

Chuyang Chen; Bea Steers; Brian McFee; Juan Bello

cs.SDMar 14, 2026

Evaluating Compositional Structure in Audio Representations

Authors: Chuyang Chen, Bea Steers, Brian McFee, Juan Bello

Organizations: Music and Audio Research Laboratory, New York University, USA

Abstract

We propose a benchmark for evaluating compositionality in audio representations. Audio compositionality refers to representing sound scenes in terms of constituent sources and attributes, and combining them systematically. While central to auditory perception, this property is largely absent from current evaluation protocols. Our framework adapts ideas from vision and language to audio through two tasks: A-COAT, which tests consistency under additive transformations, and A-TRE, which probes reconstructibility from attribute-level primitives. Both tasks are supported by large synthetic datasets with controlled variation in acoustic attributes, providing the first benchmark of compositional structure in audio embeddings.

Figures & tables

Model (checkpoint)	Architecture	Training Objective	#Params	A-COAT $\uparrow$	A-TRE $\uparrow$
Downsample	–	None	–	1.00 $\pm$ 0.01	0.23 $\pm$ 0.16
Random	–	None	–	0.00 $\pm$ 0.04	0.00 $\pm$ 0.04
PANNs (Cnn14)	CNN	Supervised classification (AudioSet)	81M	0.27 $\pm$ 0.24	0.93 $\pm$ 0.04
PaSST (PaSST-S)	Transformer	Supervised classification (AudioSet)	86M	0.26 $\pm$ 0.19	0.87 $\pm$ 0.05
CLAP (630k-AS-best)	Transformer	Contrastive audio–text pretraining	31M	0.39 $\pm$ 0.20	0.90 $\pm$ 0.05
Whisper (large-v2)	Hybrid	Automatic speech recognition (ASR)	635M	0.32 $\pm$ 0.22	0.98 $\pm$ 0.01

Table 1: Performance comparison across baselines and pretrained audio encoders. The Downsample baseline attains a perfect A-COAT score by construction, while the Random baseline produces near-zero scores on both metrics.

Figure 1: Model score distributions for A-COAT (a) and A-TRE (b) as notched box plots. Boxes show the interquartile range with median in red; notches give an approximate 95% confidence interval.

Figure 2: Model scores as a function of diversity. (a) A-COAT vs $H^{\mathrm{quad}}$ : most models exhibit consistent negative slopes except BEATs . (b) A-TRE vs $H$ : slopes vary across models, indicating differing sensitivity to diversity. Red lines show linear fits with 95% confidence intervals.

Compositional Representation Learning

Compositional representation learning studies how models encode reusable parts and relations to understand combinations they have not seen.

See all 84 papers

Do Audio Representations Compose Additively?
Sep 23, 2026Chenhao Xue, Zhijin Guo, Joyraj Chakraborty +2Audio Representation Learning Compositional Representation Learning
Department of Engineering Science, University of Oxford, Oxford, UK · University of Essex, Colchester, UK
PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio
Mar 5, 2026Yuanjian Chen, Yang Xiao, Han Yin +3Audio Reasoning Audio-Language Model Evaluation
Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1
Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions
Sep 28, 2026Michel Olvera, Paraskevas Stamatiadis, Changhong Wang +1Audio-Language Model Evaluation Audio Reasoning
S2A, LTCI, IDS · S2A, IDS

Sep 23, 2026J/K move · Enter open · S save

Do Audio Representations Compose Additively?

Chenhao Xue, Zhijin Guo, Joyraj Chakraborty, Martin Reed+1

Department of Engineering Science, University of Oxford, Oxford, UK · University of Essex, Colchester, UK

Audio Representation Learning Compositional Representation Learning

Open paper AskPDF arXiv

Compositionality, the ability to represent complex acoustic scenes as combinations of simpler sound sources, is central to auditory perception and classical additive signal models. Still, it remains unclear whether modern pre-trained audio representations internalize additive structure without compositional supervision. Existing evaluation frameworks of audio compositional reasoning largely focus on cross-modal audio-text alignment, leaving open whether audio representations themselves exhibit additive compositional structure independent of text grounding, analogous to vector arithmetic in word representations. To investigate this, we adopt a two-step diagnostic for frozen audio representations. First, we quantify linear alignment between representations and sound source labels using canonical correlation analysis. Second, we test additive compositional generalization via leave-one-combination-out reconstruction, grouping clips by exact source-label set, averaging their representations, and predicting held-out means from per-source contributions fitted only on training combinations. With larger combination holdouts, CLAP outperforms the permuted and label-overlap baselines on FSD50K, while the speech models do not outperform the label-overlap baseline. We examine representations generated by Wav2Vec2, HuBERT, and CLAP on FSD50K and CHiME-Home datasets. All three models show consistently higher linear correlation and more accurate leave-one-combination-out reconstructions than the permuted baselines. However, only CLAP shows large cosine similarity gains, which could be associated with its training on many kinds of audio and text. Finally, we note that all three models exhibit reconstruction residuals, revealing limits of additive compositionality such as nonlinear or non-compositional audio structure.

Abstract

Compositional Representation Learning

Explore similar work

Do Audio Representations Compose Additively?

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions