How Class Ontology and Data Scale Affect Audio Transfer Learning
Authors: Manuel Milling, Andreas Triantafyllopoulos, Alexander Gebhard, Simon Rampp, Björn W. Schuller
Organizations: CHI – Chair of Health Informatics, Technical University of Munich, Munich, Germany · MCML – Munich Center for Machine Learning, Munich, Germany · MDSI – Munich Data Science Institute, Munich, Germany · GLAM – Group on Language, Audio, & Music, Imperial College, London, UK
Transfer learning is a crucial concept within deep learning that allows artificial neural networks to benefit from a large pre-training data basis when confronted with a task of limited data. Despite its ubiquitous use and clear benefits, there are still many open questions regarding the inner workings of transfer learning and, in particular, regarding the understanding of when and how well it works. To that extent, we perform a rigorous study focusing on audio-to-audio transfer learning, in which we pre-train various model states on (ontology-based) subsets of AudioSet and fine-tune them on three computer audition tasks, namely acoustic scene recognition, bird activity recognition, and speech command recognition. We report that increasing the number of samples and classes in the pre-training data both have a positive impact on transfer learning. This is, however, generally surpassed by similarity between pre-training and the downstream task, which can lead the model to learn comparable features.
Figures & tables
Abbreviation
# data points
Branches
# classes
UAF1
AS
1 716 492
all
527
0.194
AS-speech
836 623
S
8
0.222
AS-humansounds
910 601
HS
72
0.230
AS-things
246 927
SoT
167
0.309
AS-animal
81 237
A
65
0.442
AS-naturalsounds
31 310
NS
18
0.493
TABLE I : Subsets of AudioSet considered for pre-training, including information about the corresponding ontology branches, number of training samples, positive classes, and subset-specific performance on the pre-training task. Unweighted average F1 (UAF1)-score is based on a threshold of 0.5. Abbreviations: speech (S) , human sounds (HS) , sound of things (SoT) , animal (A) , natural sounds (NS) .
Fig. 1 : Excerpt of the AudioSet ontology, including specific nodes being considered in the selection of pre-training data. We colour the nodes whose branches explicitly serve as a data basis for pre-training schematically, based on whether they are focused on sounds related to humans (blue), nature (red), or mechanical things (green).
Training Setting
Pre-training
Fine-Tuning
Datasets
AS (-subset)
ASC , BAD , SCR
Architecture
CNN10
Optimiser
Adam
Learning Rate
10−3
Batch Size
128
32
Epochs
30
50
TABLE II : Overview of the training setup for the pre-training and fine-tuning stage of transfer learning experiments. Abbreviations: AudioSet (AS) acoustic scene classification (ASC) , speech command recognition (SCR) , bird activity detection (BAD) , binary cross-entropy (BCE) , balanced cross-entropy (BalancedCE) .
Fig. 2 : Performance of fine-tuning experiments on the three datasets acoustic scene classification (ASC) , bird activity detection (BAD) , and speech command recognition (SCR) w. r. t. the number of pre-training samples. We consider different pre-training states trained on randomly sampled (orange) and ontology-based (blue) subsets of AudioSet (AS) . Results are averaged across three random seeds. Note the difference in performance scale across the fine-tuning tasks.
Fig. 3 : Performance of fine-tuning experiments on the three datasets acoustic scene classification (ASC) , bird activity detection (BAD) and speech command recognition (SCR) w. r. t. the number of pre-training classes. We consider different pre-training states trained on ontology-based subsets of AudioSet (AS) . Results are averaged across three random seeds. Note the difference in performance scale across the fine-tuning tasks.
Fig. 4 : Pair-wise cosine distance between the first convolutional layer of model states, pre-trained on a different subset of AudioSet.
Current audio pre-training seeks to learn unified representations for broad audio understanding tasks, but it remains fragmented and is fundamentally bottlenecked by its reliance on weak, noisy, and scale-limited labels. Drawing lessons from vision's foundational pre-training blueprint, we argue that the audio field must first establish its own large-scale, strong supervision framework. We introduce a new data-centric pipeline that leverages a high-fidelity captioner to create SOTA-quality captions and the first Unified Tag System (UTS) that bridges speech, music, and environmental sounds. We then conduct a systematic comparative study of different pre-training objectives on these strong source data. Our experiments suggest that data quality and coverage are the primary drivers of performance, while the choice of objective dictates downstream task specialization.
Xuanru Zhou, Yiwen Shao, Wei-Cheng Tseng +1
UT-Austin, Austin, USA · Tencent AI Lab, Bellevue, USA
In order to gain fresh insights about the information processing characteristics of different audio classification models, we propose transferability analysis. Given a minimal, sufficient signal for a classification on a model f, transferability analysis asks whether other models accept this minimal signal as having the same classification as it did on f. We define what it means for a sufficient signal to be transferable and perform a large study over 3 different classification tasks: music genre, emotion recognition and deepfake detection. We find that transferability rates vary depending on the task, with sufficient signals for music genre being transferable ≈26% of the time. The other tasks reveal much higher variance in transferability and reveal that some models, in particular on deepfake detection, have different transferability behavior. We call these models `flat-earther' models. We investigate deepfake audio in more depth, and show that transferability analysis also allows to us to discover information theoretic differences between the models which are not captured by the more familiar metrics of accuracy and precision.
Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-language models can build effective general-purpose audio encoders, nor a systematic understanding of how pretraining objectives behave across diverse tasks and scales. We identify three key barriers: limited scale of audio-text corpora, limited coverage of audio attributes in existing caption corpora, and lack of systematic exploration and evaluation. To fill this gap, we present the first principled empirical study of ALP. We first introduce CaptionStew, a 10.7M caption dataset aggregating open-source audio-text corpora across multiple domains and captioning focuses. We then conduct the first comprehensive evaluation comparing contrastive and captioning objectives for learning audio representation across speech, music, and environmental sound tasks. Our results not only demonstrate that ALP yields competitive, transferable representations, but reveal critical trade-offs: contrastive learning offers superior data efficiency, while captioning exhibits better scalability. Furthermore, we find that the benefits of supervised initialization often diminish at larger scales, challenging common practices. By grounding these claims in empirical evidence, we establish a viable pathway toward general-purpose audio representation learning, guiding future research.
Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1