cs.LGMar 26, 2026

How Class Ontology and Data Scale Affect Audio Transfer Learning

Authors: Manuel Milling, Andreas Triantafyllopoulos, Alexander Gebhard, Simon Rampp, Björn W. Schuller

Organizations: CHI – Chair of Health Informatics, Technical University of Munich, Munich, Germany · MCML – Munich Center for Machine Learning, Munich, Germany · MDSI – Munich Data Science Institute, Munich, Germany · GLAM – Group on Language, Audio, & Music, Imperial College, London, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transfer learning is a crucial concept within deep learning that allows artificial neural networks to benefit from a large pre-training data basis when confronted with a task of limited data. Despite its ubiquitous use and clear benefits, there are still many open questions regarding the inner workings of transfer learning and, in particular, regarding the understanding of when and how well it works. To that extent, we perform a rigorous study focusing on audio-to-audio transfer learning, in which we pre-train various model states on (ontology-based) subsets of AudioSet and fine-tune them on three computer audition tasks, namely acoustic scene recognition, bird activity recognition, and speech command recognition. We report that increasing the number of samples and classes in the pre-training data both have a positive impact on transfer learning. This is, however, generally surpassed by similarity between pre-training and the downstream task, which can lead the model to learn comparable features.

Figures & tables

Abbreviation# data pointsBranches# classesUAF1
AS1 716 492all5270.194
AS-speech836 623S80.222
AS-humansounds910 601HS720.230
AS-things246 927SoT1670.309
AS-animal81 237A650.442
AS-naturalsounds31 310NS180.493
TABLE I : Subsets of AudioSet considered for pre-training, including information about the corresponding ontology branches, number of training samples, positive classes, and subset-specific performance on the pre-training task. Unweighted average F1 (UAF1)-score is based on a threshold of 0.5. Abbreviations: speech (S) , human sounds (HS) , sound of things (SoT) , animal (A) , natural sounds (NS) .
Fig. 1 : Excerpt of the AudioSet ontology, including specific nodes being considered in the selection of pre-training data. We colour the nodes whose branches explicitly serve as a data basis for pre-training schematically, based on whether they are focused on sounds related to humans (blue), nature (red), or mechanical things (green).
Training SettingPre-trainingFine-Tuning
DatasetsAS (-subset)ASC , BAD , SCR
ArchitectureCNN10
OptimiserAdam
Learning Rate10−310^{-3}
Batch Size12832
Epochs3050
TABLE II : Overview of the training setup for the pre-training and fine-tuning stage of transfer learning experiments. Abbreviations: AudioSet (AS) acoustic scene classification (ASC) , speech command recognition (SCR) , bird activity detection (BAD) , binary cross-entropy (BCE) , balanced cross-entropy (BalancedCE) .
Fig. 2 : Performance of fine-tuning experiments on the three datasets acoustic scene classification (ASC) , bird activity detection (BAD) , and speech command recognition (SCR) w. r. t. the number of pre-training samples. We consider different pre-training states trained on randomly sampled (orange) and ontology-based (blue) subsets of AudioSet (AS) . Results are averaged across three random seeds. Note the difference in performance scale across the fine-tuning tasks.
Fig. 3 : Performance of fine-tuning experiments on the three datasets acoustic scene classification (ASC) , bird activity detection (BAD) and speech command recognition (SCR) w. r. t. the number of pre-training classes. We consider different pre-training states trained on ontology-based subsets of AudioSet (AS) . Results are averaged across three random seeds. Note the difference in performance scale across the fine-tuning tasks.
Fig. 4 : Pair-wise cosine distance between the first convolutional layer of model states, pre-trained on a different subset of AudioSet.
More in this topic

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Explore similar work

CardsList

  1. Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods

    Mar 26, 2026Xuanru Zhou, Yiwen Shao, Wei-Cheng Tseng +1Audio Representation LearningAudio Classification

    UT-Austin, Austin, USA · Tencent AI Lab, Bellevue, USA

  2. If It's Good Enough for You, It's Good Enough for Me: Transferability of Audio Sufficiencies across Models

    Apr 3, 2026David A. Kelly, Hana ChocklerAudio Classification

    King’s College London UK

  3. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model EvaluationAudio Representation Learning

    University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1