Audio Classification

Ask about this topic

Overview

Audio classification studies how to assign sounds to labels, from spoken words and music genres to bird calls, machine faults, and respiratory conditions. Systems commonly use spectrograms, mel-frequency cepstral coefficients, or raw waveforms with convolutional networks, transformers, and pretrained or self-supervised audio encoders; limited-label settings also use transfer learning, few-shot methods, and audio-language models. Benchmarks such as AudioSet, ESC-50, UrbanSound8K, and ICBHI test different tasks, but strong benchmark scores may not carry over across recording devices, locations, speakers, or species. A central challenge is building reliable models from scarce or weak labels while avoiding data leakage and shortcuts, and keeping models efficient enough for clinical and on-device use.

Latest papers

CardsList

147 papers18 papers in the last four weeks, up 6% on the four weeks before. 0.1% of all new papers.

Apr 6, 2026

YMIR: A new Benchmark Dataset and Model for Arabic Yemeni Music Genre Classification Using Convolutional Neural Networks

Automatic music genre classification is a major task in music information retrieval; however, most current benchmarks and models have been developed primarily for Western music, leaving culturally specific traditions underrepresented. In this paper, we introduce the Yemeni Music Information Retrieval (YMIR) dataset, which contains 1,475 carefully selected audio clips covering five traditional Yemeni genres: Sanaani, Hadhrami, Lahji, Tihami, and Adeni. The dataset was labeled by five Yemeni music experts following a clear and structured protocol, resulting in strong inter-annotator agreement (Fleiss kappa = 0.85). We also propose the Yemeni Music Classification Model (YMCM), a convolutional neural network (CNN)-based system designed to classify music genres from time-frequency features. Using a consistent preprocessing pipeline, we perform a systematic comparison across six experimental groups and five different architectures, resulting in a total of 30 experiments. Specifically, we evaluate several feature representations, including Mel-spectrograms, Chroma, FilterBank, and MFCCs with 13, 20, and 40 coefficients, and benchmark YMCM against standard models (AlexNet, VGG16, MobileNet, and a baseline CNN) under the same experimental conditions. The experimental findings reveal that YMCM is the most effective, achieving the highest accuracy of 98.8% with Mel-spectrogram features. The results also provide practical insights into the relationship between feature representation and model capacity. The findings establish YMIR as a useful benchmark and YMCM as a strong baseline for classifying Yemeni music genres.

Moeen AL-Makhlafi, Abdulrahman A. AlKannad, Eiad Almekhlafi +2

School of Artificial Intelligence, Xidian University, Xi’an, China. · Department of Information Science and Technology, Northwest University, Xi’an 710127, China.

Music Information RetrievalAudio ClassificationConvolutional Neural Networks
AskPDF
Apr 3, 2026

If It's Good Enough for You, It's Good Enough for Me: Transferability of Audio Sufficiencies across Models

In order to gain fresh insights about the information processing characteristics of different audio classification models, we propose transferability analysis. Given a minimal, sufficient signal for a classification on a model ff, transferability analysis asks whether other models accept this minimal signal as having the same classification as it did on ff. We define what it means for a sufficient signal to be transferable and perform a large study over 33 different classification tasks: music genre, emotion recognition and deepfake detection. We find that transferability rates vary depending on the task, with sufficient signals for music genre being transferable ≈26%\approx26\% of the time. The other tasks reveal much higher variance in transferability and reveal that some models, in particular on deepfake detection, have different transferability behavior. We call these models `flat-earther' models. We investigate deepfake audio in more depth, and show that transferability analysis also allows to us to discover information theoretic differences between the models which are not captured by the more familiar metrics of accuracy and precision.

David A. Kelly, Hana Chockler

King’s College London UK

Audio Classification
AskPDF
Apr 2, 2026

Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

Outbound AI calling systems must distinguish voicemail greetings from live human answers in real time to avoid wasted agent interactions and dropped calls. We present a lightweight approach that extracts 15 temporal features from the speech activity pattern of a pre-trained neural voice activity detector (VAD), then classifies with a shallow tree-based ensemble. Across two evaluation sets totaling 764 telephony recordings, the system achieves a combined 96.1% accuracy (734/764), with 99.3% (139/140) on an expert-labeled test set and 95.4% (595/624) on a held-out production set. In production validation over 77,000 calls, it maintained a 0.3% false positive rate and 1.3% false negative rate. End-to-end inference completes in 46 ms on a commodity dual-core CPU with no GPU, supporting 380+ concurrent WebSocket calls. In our search over 3,780 model, feature, and threshold combinations, feature importance was concentrated in three temporal variables. Adding transcription keywords or beep-based features did not improve the best real-time configuration and increased latency substantially. Our results suggest that temporal speech patterns are a strong signal for distinguishing voicemail greetings from live human answers.

Kumar Saurav

CTO, ClearGrid

Speech ProcessingAudio Classification
AskPDF
Mar 31, 2026

Can LLM Agents Identify Spoken Dialects like a Linguist?

Due to the scarcity of labeled dialectal speech, audio dialect classification is a challenging task for most languages, including Swiss German. In this work, we explore the ability of large language models (LLMs) as agents in understanding the dialects and whether they can show comparable performance to models such as HuBERT in dialect classification. In addition, we provide an LLM baseline and a human linguist one. Our approach uses phonetic transcriptions produced by ASR systems and combines them with linguistic resources such as dialect feature maps, vowel history, and rules. Our findings indicate that, when linguistic information is provided, the LLM predictions improve. The human baseline shows that automatically generated transcriptions can be beneficial for such classifications, but also presents opportunities for improvement.

Tobias Bystrich, Lukas Hamm, Maria Hassan +3

Department of Computer Science, University of Bonn, Germany · Research Center Deutscher Sprachatlas, Marburg University, Germany · Bonn-Aachen International Center for Information Technology, University of Bonn, Germany +1

Audio ClassificationLLM Agents
AskPDF
Mar 26, 2026

How Class Ontology and Data Scale Affect Audio Transfer Learning

Transfer learning is a crucial concept within deep learning that allows artificial neural networks to benefit from a large pre-training data basis when confronted with a task of limited data. Despite its ubiquitous use and clear benefits, there are still many open questions regarding the inner workings of transfer learning and, in particular, regarding the understanding of when and how well it works. To that extent, we perform a rigorous study focusing on audio-to-audio transfer learning, in which we pre-train various model states on (ontology-based) subsets of AudioSet and fine-tune them on three computer audition tasks, namely acoustic scene recognition, bird activity recognition, and speech command recognition. We report that increasing the number of samples and classes in the pre-training data both have a positive impact on transfer learning. This is, however, generally surpassed by similarity between pre-training and the downstream task, which can lead the model to learn comparable features.

Manuel Milling, Andreas Triantafyllopoulos, Alexander Gebhard +2

CHI – Chair of Health Informatics, Technical University of Munich, Munich, Germany · MCML – Munich Center for Machine Learning, Munich, Germany · MDSI – Munich Data Science Institute, Munich, Germany +1

Transfer LearningAudio Classification
AskPDF
Mar 26, 2026

Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods

Current audio pre-training seeks to learn unified representations for broad audio understanding tasks, but it remains fragmented and is fundamentally bottlenecked by its reliance on weak, noisy, and scale-limited labels. Drawing lessons from vision's foundational pre-training blueprint, we argue that the audio field must first establish its own large-scale, strong supervision framework. We introduce a new data-centric pipeline that leverages a high-fidelity captioner to create SOTA-quality captions and the first Unified Tag System (UTS) that bridges speech, music, and environmental sounds. We then conduct a systematic comparative study of different pre-training objectives on these strong source data. Our experiments suggest that data quality and coverage are the primary drivers of performance, while the choice of objective dictates downstream task specialization.

Xuanru Zhou, Yiwen Shao, Wei-Cheng Tseng +1

UT-Austin, Austin, USA · Tencent AI Lab, Bellevue, USA

Audio Representation LearningAudio Classification
AskPDF
Mar 20, 2026

BioDCASE 2026 Challenge Baseline for Cross-Domain Mosquito Species Classification

Mosquito-borne diseases affect more than one billion people each year and cause close to one million deaths. Traditional surveillance methods rely on traps and manual identification that are slow, labor-intensive, and difficult to scale. Audio-based mosquito monitoring offers a non-destructive, lower-cost, and more scalable complement to trap-based surveillance, but reliable species classification remains difficult under real-world recording conditions. Mosquito flight tones are narrow-band, often low in signal-to-noise ratio, and easily masked by background noise, and recordings for several epidemiologically relevant species remain limited, creating pronounced class imbalance. Variation across devices, environments, and collection protocols further increases the difficulty of robust classification. Such variation can cause models to rely on domain-specific recording artefacts rather than species-relevant acoustic cues, which makes transfer to new acquisition settings difficult. The BioDCASE 2026 Cross-Domain Mosquito Species Classification (CD-MSC) challenge is designed around this deployment problem by evaluating performance on both seen and unseen domains. This paper presents the official baseline system and evaluation pipeline as a simple, fully reproducible reference for the CD-MSC challenge task. The baseline uses log-mel features and a multitemporal resolution convolutional neural network (MTRCNN) with species and auxiliary domain outputs, together with complete training and test scripts. The baseline system performs strongly on seen domains but degrades markedly on unseen domains, showing that cross-domain generalisation, rather than within-domain recognition, is the central challenge for practical mosquito species classification from multi-source bioacoustic recordings.

Yuanbo Hou, Vanja Zdravkovic, Marianne Sinka +5

Machine Learning Research Group, University of Oxford · Oxford Long-Term Ecology Laboratory, University of Oxford · King’s College London +1

BioacousticsAudio ClassificationCross-Domain Transfer Learning
AskPDF
Mar 16, 2026

Music Genre Classification: A Comparative Analysis of Classical Machine Learning and Deep Learning Approaches

Automatic music genre classification is a long-standing challenge in Music Information Retrieval (MIR); work on non-Western music traditions remains scarce. Nepali music encompasses culturally rich and acoustically diverse genres--from the call-and-response duets of Lok Dohori to the rhythmic poetry of Deuda and the distinctive melodies of Tamang Selo--that have not been addressed by existing classification systems. In this paper, we construct a novel dataset of approximately 8,000 labeled 30-second audio clips spanning eight Nepali music genres and conduct a systematic comparison of nine classification models across two paradigms. Five classical machine learning classifiers (Logistic Regression, SVM, KNN, Random Forest, and XGBoost) are trained on 51 hand-crafted audio features extracted via Librosa, while four deep learning architectures (CNN, RNN, parallel CNN-RNN, and sequential CNN followed by RNN) operate on Mel spectrograms of dimension 640 x 128. Our experiments reveal that the sequential Convolutional Recurrent Neural Network (CRNN)--in which convolutional layers feed into an LSTM--achieves the highest accuracy of 84%, substantially outperforming both the best classical models (Logistic Regression and XGBoost, both at 71%) and all other deep architectures. We provide per-class precision, recall, F1-score, confusion matrices, and ROC analysis for every model, and offer a culturally grounded interpretation of misclassification patterns that reflects genuine overlaps in Nepal's musical traditions.

Sachin Prajuli, Abhishek Karna, OmPrakash Dhakl

Computer Science and Engineering, University of texas at Arlington, Arlington, USA · Electronics and Computer Engineering, Tribhuvan University, IOE, kathmandu · Electronics and Computer Engineering, Tribhuvan University, IOE, Dharan

Audio ClassificationMusic Information RetrievalConvolutional RNNs
AskPDF
Mar 11, 2026

Geo-ATBench: A Benchmark for Geospatial Audio Tagging with Geospatial Semantic Context

Environmental sound understanding in computational auditory scene analysis (CASA) is often formulated as an audio-only recognition problem. This formulation leaves a persistent drawback in multi-label audio tagging (AT): acoustic similarity can make certain events difficult to separate from waveforms alone. In such cases, disambiguating cues often lie outside the waveform. Geospatial semantic context (GSC), derived from geographic information system data, e.g., points of interest (POI), provides location-tied environmental priors that can help reduce this ambiguity. A systematic study of this direction is enabled through the proposed geospatial audio tagging (Geo-AT) task, which conditions multi-label sound event tagging on GSC alongside audio. To benchmark Geo-AT, Geo-ATBench is introduced as a polyphonic audio benchmark with geographical annotations, containing 10.71 hours of audio across 28 event categories; each clip is paired with a GSC representation from 11 semantic context categories. GeoFusion-AT is proposed as a unified geo-audio fusion framework that evaluates feature-, representation-, and decision-level fusion on representative audio backbones, with audio- and GSC-only baselines. Results show that incorporating GSC improves AT performance, especially on acoustically confounded labels, indicating geospatial semantics provide effective priors beyond audio alone. A crowdsourced listening study with 10 participants on 579 samples shows that there is no significant difference in performance between models on Geo-ATBench labels and aggregated human labels, supporting Geo-ATBench as a human-aligned benchmark. The Geo-AT task, benchmark Geo-ATBench, and reproducible geo-audio fusion framework GeoFusion-AT provide a foundation for studying AT with geospatial semantic context within the CASA community. Dataset, code, models are on homepage (https://github.com/WuYanru2002/Geo-ATBench).

Yuanbo Hou, Yanru Wu, Qiaoqiao Ren +3

Machine Learning Research Group, Engineering Science, University of Oxford, UK · Department of Intelligent Science, Xi’an Jiaotong-Liverpool University, China · EECS, KTH Royal Institute of Technology, Sweden +1

Audio ClassificationSound Event DetectionMultimodal Fusion
AskPDF
Mar 11, 2026

Abjad-Kids: An Arabic Speech Classification Dataset for Primary Education

Speech-based AI educational applications have gained significant interest in recent years, particularly for children. However, children speech research remains limited due to the lack of publicly available datasets, especially for low-resource languages such as Arabic.This paper presents Abjad-Kids, an Arabic speech dataset designed for kindergarten and primary education, focusing on fundamental learning of alphabets, numbers, and colors. The dataset consists of 46397 audio samples collected from children aged 3 - 12 years, covering 141 classes. All samples were recorded under controlled specifications to ensure consistency in duration, sampling rate, and format. To address high intra-class similarity among Arabic phonemes and the limited samples per class, we propose a hierarchical audio classification based on CNN-LSTM architectures. Our proposed methodology decomposes alphabet recognition into a two-stage process: an initial grouping classification model followed by specialized classifiers for each group. Both strategies: static linguistic-based grouping and dynamic clustering-based grouping, were evaluated. Experimental results demonstrate that static linguistic-based grouping achieves superior performance. Comparisons between traditional machine learning with deep learning approaches, highlight the effectiveness of CNN-LSTM models combined with data augmentation. Despite achieving promising results, most of our experiments indicate a challenge with overfitting, which is likely due to the limited number of samples, even after data augmentation and model regularization. Thus, future work may focus on collecting additional data to address this issue. Abjad-Kids will be publicly available. We hope that Abjad-Kids enrich children representation in speech dataset, and be a good resource for future research in Arabic speech classification for kids.

Abdul Aziz Snoubara, Baraa Al_Maradni, Haya Al_Naal +4

Arab International University, Daraa, Syria.

Speech ProcessingAudio ClassificationHierarchical Classification
AskPDF
Mar 9, 2026

Quantifying Cross-Lingual Transfer in Paralinguistic Speech Tasks

Paralinguistic speech tasks are often considered relatively language-agnostic, as they rely on extralinguistic acoustic cues rather than lexical content. However, prior studies report performance degradation under cross-lingual conditions, indicating non-negligible language dependence. Still, these studies typically focus on isolated language pairs or task-specific settings, limiting comparability and preventing a systematic assessment of task-level language dependence. We introduce the Cross-Lingual Transfer Matrix (CLTM), a systematic method to quantify cross-lingual interactions between pairs of languages within a given task. We apply the CLTM to two paralinguistic tasks, gender identification and speaker verification, using a multilingual HuBERT-based encoder, to analyze how donor-language data affects target-language performance during fine-tuning. Our results reveal distinct transfer patterns across tasks and languages, reflecting systematic, language-dependent effects.

Pol Buitrago, Oriol Pareras, Federico Costa +1

Barcelona Supercomputing Center (BSC), Spain · Universitat Polit`ecnica de Catalunya (UPC), Spain

Speaker VerificationSpeech ProcessingAudio ClassificationCross-Lingual Transfer
AskPDF
Mar 9, 2026

Soundscapes in Spectrograms: Pioneering Multilabel Classification for South Asian Sounds

Environmental sound classification is a field of growing importance for urban monitoring and cultural soundscape analysis, especially within the acoustically rich environments of South Asia. These regions present a unique challenge as multiple natural, human, and cultural sounds often overlap, straining traditional methods that frequently rely on Mel Frequency Cepstral Coefficients (MFCC). This study introduces a novel spectrogram-based methodology with a superior ability to capture these complex auditory patterns. A Convolutional Neural Network (CNN) architecture is implemented to solve a demanding multilabel, multiclass classification problem on the SAS-KIIT dataset. To demonstrate robustness and comparability, the approach is also validated using the renowned UrbanSound8K dataset. The results confirm that the proposed spectrogram-based method significantly outperforms existing MFCC-based techniques, achieving higher classification accuracy across both datasets. This improvement lays the groundwork for more robust and accurate audio classification systems in real-world applications.

Sudip Chakrabarty, Pappu Bishwas, Rajdeep Chatterjee +3

School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India · School of Computation, Information and Technology, Technical University of Munich, Germany · School of Computer Engineering, American International University, Dhaka, Bangladesh

Audio ClassificationMulti-Label Classification
AskPDF
Mar 7, 2026

Towards Objective Gastrointestinal Auscultation: Automated Segmentation and Annotation of Bowel Sound Patterns

Bowel sounds (BS) are typically momentary and have low amplitude, making them difficult to detect accurately through manual auscultation. This leads to significant variability in clinical assessment. Digital acoustic sensors allow the acquisition of high-quality BS and enable automated signal analysis, offering the potential to provide clinicians with both objective and quantitative feedback on bowel activity. This study presents an automated pipeline for bowel sound segmentation and classification using a wearable acoustic SonicGuard sensor. BS signals from 83 subjects were recorded using a SonicGuard sensor. Data from 40 subjects were manually annotated by clinical experts and used to train an automatic annotation algorithm, while the remaining subjects were used for further model evaluation. An energy-based event detection algorithm was developed to detect BS events. Detected sound segments were then classified into BS patterns using a pretrained Audio Spectrogram Transformer (AST) model. Model performance was evaluated separately for healthy individuals and patients. The best configuration used two specialized models, one trained on healthy subjects and one on patients, achieving (accuracy: 0.97, AUROC: 0.98) for healthy group and (accuracy: 0.96, AUROC: 0.98) for patient group. The auto-annotation method reduced manual labeling time by approximately 70%, and expert review showed that less than 12% of automatically detected segments required correction. The proposed automated segmentation and classification system enables quantitative assessment of bowel activity, providing clinicians with an objective diagnostic tool that may improve the diagnostic of gastrointestinal function and support the annotation of large-scale datasets.

Zahra Mansour, Verena Uslar, Dirk Weyhe +2

Division AI4Health, Department for Health Services Research, Faculty of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg, 26129 Oldenburg, Germany · Fraunhofer IDMT, Institute Part HSA, 26129 Oldenburg, Germany · University Clinic for Visceral Surgery, Faculty of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg, 26121 Oldenburg, Germany

Sound Event DetectionAudio Classification
AskPDF
Mar 7, 2026

Adaptive Discovery of Interpretable Audio Attributes with Multimodal LLMs for Low-Resource Classification

In predictive modeling for low-resource audio classification, extracting high-accuracy and interpretable attributes is critical. Particularly in high-reliability applications, interpretable audio attributes are indispensable. While human-driven attribute discovery is effective, its low throughput becomes a bottleneck. We propose a method for adaptively discovering interpretable audio attributes using Multimodal Large Language Models (MLLMs). By replacing humans in the AdaFlock framework with MLLMs, our method achieves significantly faster attribute discovery. Our method dynamically identifies salient acoustic characteristics via prompting and constructs an attribute-based ensemble classifier. Experimental results across various audio tasks demonstrate that our method outperforms direct MLLM prediction in the majority of evaluated cases. The entire training completes within 11 minutes, proving it a practical, adaptive solution that surpasses conventional human-reliant approaches.

Kosuke Yoshimura, Hisashi Kashima

Kyoto University, Kyoto, Japan

Audio Classification
AskPDF
Feb 24, 2026

LMU-Based Sequential Learning and Posterior Ensemble Fusion for Cross-Domain Infant Cry Classification

Decoding infant cry causes remains challenging for healthcare monitoring due to short nonstationary signals, limited annotations, and strong domain shifts across infants and datasets. We propose a compact acoustic framework that fuses mel-frequency cepstral coefficients (MFCCs), short-time Fourier transform (STFT) features, and fundamental-frequency (F0) contours within a multi-branch convolutional neural network (CNN) encoder, and models temporal dynamics using an enhanced Legendre Memory Unit (LMU). Compared to LSTMs, the LMU backbone provides stable sequence modeling with substantially fewer recurrent parameters, supporting efficient deployment. To improve cross-dataset generalization, we introduce calibrated posterior ensemble fusion with entropy-gated weighting to preserve domain-specific expertise while mitigating dataset bias. Experiments on Baby2020 and Baby Crying demonstrate improved macro-F1 under cross-domain evaluation, along with leakage aware splits and real-time feasibility for on-device monitoring.

Niloofar Jazaeri, Hilmi R. Dajani, Marco Janeczek +1

University of Ottawa, Ottawa, Canada · Crynostics Inc., Canada

Audio ClassificationEnsemble Learning
AskPDF
Feb 20, 2026

Fairness-Aware Partial-label Domain Adaptation for Voice Classification of Parkinson's and ALS

Voice-based digital biomarkers can enable scalable, non-invasive screening and monitoring of Parkinson's disease (PD) and Amyotrophic Lateral Sclerosis (ALS). However, models trained on one cohort or device often fail on new acquisition settings due to cross-device and cross-cohort domain shift. This challenge is amplified in real-world scenarios with partial-label mismatch, where datasets may contain different disease labels and only partially overlap in class space. In addition, voice-based models may exploit demographic cues, raising concerns about gender-related unfairness, particularly when deployed across heterogeneous cohorts. To tackle these challenges, we propose a hybrid framework for unified three-class (healthy/PD/ALS) cross-domain voice classification from partially overlapping cohorts. The method combines style-based domain generalization with conditional adversarial alignment tailored to partial-label settings, reducing negative transfer. An additional adversarial gender branch promotes gender-invariant representations. We conduct a comprehensive evaluation across four heterogeneous sustained-vowel datasets, spanning distinct acquisition settings and devices, under both domain generalization and unsupervised domain adaptation protocols. The proposed approach is compared against twelve state-of-the-art machine learning and deep learning methods, and further evaluated through three targeted ablations, providing the first cross-cohort benchmark and end-to-end domain-adaptive framework for unified healthy/PD/ALS voice classification under partial-label mismatch and fairness constraints. Across all experimental settings, our method consistently achieves the best external generalization over the considered evaluation metrics, while maintaining reduced gender disparities. Notably, no competing method shows statistically significant gains in external performance.

Arianna Francesconi, Zhixiang Dai, Arthur Stefano Moscheni +7

School of Computer and Communication Sciences, EPFL (École polytechnique fédérale de Lausanne), Lausanne, Switzerland · Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico di Roma, Rome, Italy · Eustema S.p.A., Research and Development Centre, Naples, Italy +2

Parkinson's Disease Detection from SpeechDomain AdaptationAudio ClassificationClinical Speech ProcessingDomain GeneralizationAlgorithmic Fairness
AskPDF
Feb 19, 2026

Investigating Target Class Influence on Neural Network Compressibility for Energy-Autonomous Avian Monitoring

Biodiversity loss poses a significant threat to humanity, making wildlife monitoring essential for assessing ecosystem health. Avian species are ideal subjects for this due to their popularity and the ease of identifying them through their distinctive songs. Traditionalavian monitoring methods require manual counting and are therefore costly and inefficient. In passive acoustic monitoring, soundscapes are recorded over long periods of time. The recordings are analyzed to identify bird species afterwards. Machine learning methods have greatly expedited this process in a wide range of species and environments, however, existing solutions require complex models and substantial computational resources. Instead, we propose running machine learning models on inexpensive microcontroller units (MCUs) directly in the field. Due to the resulting hardware and energy constraints, efficient artificial intelligence (AI) architecture is required. In this paper, we present our method for avian monitoring on MCUs. We trained and compressed models for various numbers of target classes to assess the detection of multiple bird species on edge devices and evaluate the influence of the number of species on the compressibility of neural networks. Our results demonstrate significant compression rates with minimal performance loss. We also provide benchmarking results for different hardware platforms and evaluate the feasibility of deploying energy-autonomous devices.

Nina Brolich, Simon Geis, Maximilian Kasper +3

Fraunhofer Institute for Integrated Circuits IIS, Germany · Fachhochschule Erfurt, Germany · Universität Erfurt, Germany +2

Bird Vocalization ClassificationEnergy-Efficient MLTiny MLNeural Network CompressionAudio ClassificationPassive Acoustic Monitoring
AskPDF
Feb 19, 2026

Detection and Classification of Cetacean Echolocation Clicks using Image-based Object Detection Methods applied to Advanced Wavelet-based Transformations

A challenge in marine bioacoustic analysis is the detection of animal signals, like calls, whistles and clicks, for behavioral studies. Manual labeling is too time-consuming to process sufficient data to get reasonable results. Thus, an automatic solution to overcome the time-consuming data analysis is necessary. Basic mathematical models can detect events in simple environments, but they struggle with complex scenarios, like differentiating signals with a low signal-to-noise ratio or distinguishing clicks from echoes. Deep Learning Neural Networks, such as ANIMAL-SPOT, are better suited for such tasks. DNNs process audio signals as image representations, often using spectrograms created by Short-Time Fourier Transform. However, spectrograms have limitations due to the uncertainty principle, which creates a tradeoff between time and frequency resolution. Alternatives like the wavelet, which provides better time resolution for high frequencies and improved frequency resolution for low frequencies, may offer advantages for feature extraction in complex bioacoustic environments. This thesis shows the efficacy of CLICK-SPOT on Norwegian Killer whale underwater recordings provided by the cetacean biologist Dr. Vester. Keywords: Bioacoustics, Deep Learning, Wavelet Transformation

Christopher Hauer

Sound Event DetectionBioacousticsAudio Classification
AskPDF
Feb 18, 2026

Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA

As the volume of data recorded by embedded edge sensors increases, particularly from neuromorphic devices producing discrete event streams, there is a growing need for hardware-aware neural architectures that enable efficient, low-latency, and energy-conscious local processing. We present an FPGA implementation of event-graph neural networks for audio processing. We utilise an artificial cochlea that converts time-series signals into sparse event data, reducing memory and computation costs. Our architecture was implemented on a SoC FPGA and evaluated on two open-source datasets. For classification task, our baseline floating-point model achieves 92.7% accuracy on SHD dataset - only 2.4% below the state of the art - while requiring over 10x and 67x fewer parameters. On SSC, our models achieve 66.9-71.0% accuracy. Compared to FPGA-based spiking neural networks, our quantised model reaches 92.3% accuracy, outperforming them by up to 19.3% while reducing resource usage and latency. For SSC, we report the first hardware-accelerated evaluation. We further demonstrate the first end-to-end FPGA implementation of event-audio keyword spotting, combining graph convolutional layers with recurrent sequence modelling. The system achieves up to 95% word-end detection accuracy, with only 10.53 microsecond latency and 1.18 W power consumption, establishing a strong benchmark for energy-efficient event-driven KWS.

Kamil Jeziorek, Piotr Wzorek, Krzysztof Blachut +6

The AGH University of Krakow, Poland · Graduate School of Science and Technology, Keio University, Japan · CEA-List, Université Grenoble Alpes, France

Audio ClassificationKeyword SpottingGraph Neural NetworksFPGA-Based Neural Network AccelerationNeuromorphic Computing
AskPDF
Jan 18, 2026

Embryonic Exposure to VPA Influences Chick Vocalisations: A Computational Study

In young animals like poultry chicks (Gallus gallus), vocalisations convey information about affective and behavioural states. Traditional approaches to vocalisation analysis, relying on manual annotation and predefined categories, introduce biases, limit scalability, and fail to capture the full complexity of vocal repertoires. We introduce a computational framework for the automated detection, acoustic feature extraction, and unsupervised learning of chick vocalisations. Applying this framework to a dataset of newly hatched chicks, we identified two primary vocal clusters. We then tested our computational framework on an independent dataset of chicks exposed during embryonic development to vehicle or Valproic Acid (VPA), a compound that disrupts neural development and is linked to autistic-like symptoms. Clustering analysis on the experimental dataset confirmed two primary vocal clusters and revealed systematic differences between groups. VPA-exposed chicks showed an altered repertoire, with a relative increase in softer calls. VPA differentially affected call clusters, modulating temporal, frequency, and energy domain features. Overall, VPA-exposed chicks produced vocalisations with shorter duration, reduced pitch variability, and modified energy profiles, with the strongest alterations observed in louder calls. This study provides a computational framework for analysing chick vocalisations, advancing knowledge of early-life communication in typical and atypical vocal development.

Antonella M. C. Torrisi, Inês Nolasco, Paola Sgadò +2

Centre for Digital Music, Queen Mary University of London, London, UK · Centre for Mind/Brain Sciences, University of Trento, Rovereto, Italy · School of Biological and Behavioural Sciences, Centre for Brain and Behaviour, Queen Mary University of London, London, UK

Audio Classification
AskPDF
Nov 23, 2025

DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation

Existing Test-time Adaptation (TTA) studies rely heavily on static and homogeneous corruption protocols, such as ImageNet-C and CIFAR-10-C/100-C, leading to inconsistent evaluation settings and potentially inflated robustness estimates that are compared with real-world situations. TTA lacks a standardized evaluation infrastructure capable of modeling realistic heterogeneous acoustic degradation. We introduce DHAuDS, a standardized benchmark suite for evaluating audio classification TTA robustness under dynamic corruption severity and heterogeneous noise mixtures. Rather than proposing a new TTA algorithm, DHAuDS focuses on exposing robustness limitations that remain hidden under conventional fixed-noise evaluation protocols.

Weichuang Shao, Iman Yi Liao, Tomas Henrique Bode Maul +1

School of Computer and Mathematical Sciences, University of Nottingham Malaysia, Semenyih, Malaysia

Test-Time AdaptationNeural Network RobustnessNoise RobustnessAudio Classification
AskPDF
Sep 18, 2025

BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings

Child-centered daylong recordings are essential for studying early language development, but existing speech models trained on clean adult data perform poorly due to acoustic and linguistic differences. We introduce BabyHuBERT, a self-supervised speech model trained on 13,000 hours of multilingual child-centered recordings from 40+ languages. Evaluated on voice type classification, the task of identifying who produces speech and when in child-centered recordings (key child, other children, male, and female adults), BabyHuBERT-VTC achieves F1-scores from 55.0% to 76.1% across six corpora, consistently outperforming W2V2-LL4300 and HuBERT (pretrained on English daylongs and clean adult speech, respectively). Notable gains include 14.0 and 18.3 absolute F1 points over HuBERT on Vanuatu and Solomon Islands, demonstrating effectiveness on underrepresented languages. We share code and models to support researchers working with child-centered recordings across diverse linguistic contexts.

Théo Charlot, Tarek Kunze, Maxime Poli +3

LSCP, DEC, ENS, EHESS, CNRS, PSL University, France · Laboratoire d’Informatique et Syst`emes, Universit´e Aix-Marseille, CNRS, France

Speech Foundation ModelsSpeech ProcessingAudio UnderstandingSelf-Supervised Speech Representation LearningAudio ClassificationSpeaker Diarization
AskPDF
Sep 15, 2025

Scaling to Multimodal and Multichannel Heart Sound Classification with Synthetic and Augmented Biosignals

Cardiovascular diseases (CVDs) are the leading cause of death worldwide, accounting for approximately 17.9 million deaths each year. Early detection is critical, creating a demand for accurate and inexpensive pre-screening methods. Deep learning has recently been applied to classify abnormal heart sounds indicative of CVDs using synchronised phonocardiogram (PCG) and electrocardiogram (ECG) signals, as well as multichannel PCG (mPCG). However, state-of-the-art architectures remain underutilised due to the limited availability of synchronised and multichannel datasets. Augmented datasets and pre-trained models provide a pathway to overcome these limitations, enabling transformer-based architectures to be trained effectively. This work combines traditional signal processing with denoising diffusion models, WaveGrad and DiffWave, to create an augmented dataset to fine-tune a Wav2Vec 2.0-based classifier on multimodal and multichannel heart sound datasets. The approach achieves state-of-the-art performance. On the Computing in Cardiology (CinC) 2016 dataset of single channel PCG, accuracy, unweighted average recall (UAR), sensitivity, specificity and Matthew's correlation coefficient (MCC) reach 92.48%, 93.05%, 93.63%, 92.48%, 94.93% and 0.8283, respectively. Using the synchronised PCG and ECG signals of the training-a dataset from CinC, 93.14%, 92.21%, 94.35%, 90.10%, 95.12% and 0.8380 are achieved for accuracy, UAR, sensitivity, specificity and MCC, respectively. Using a wearable vest dataset consisting of mPCG data, the model achieves 77.13% accuracy, 74.25% UAR, 86.47% sensitivity, 62.04% specificity, and 0.5082 MCC. These results demonstrate the effectiveness of transformer-based models for CVD detection when supported by augmented datasets, highlighting their potential to advance multimodal and multichannel heart sound classification.

Milan Marocchi, Matthew Fynn, Kayapanda Mandana +1

School of Electrical Engineering, Computing, and Mathematical Sciences (EECMS), Faculty of Science and Engineering, Curtin University, Bentley, WA 6102, Australia · Department of Cardiology, Fortis Healthcare, Kolkata, 7007107, West Bengal, India

Synthetic Data AugmentationMultimodal ClassificationAudio Classification
AskPDF
Feb 19, 2025

Localized time-frequency representation learning for bioacoustic classification in complex soundscapes

Prevailing bioacoustic classifiers assign species labels to fixed time-frequency windows rather than to individual vocalizations. When multiple vocalizations occur within the same window, predictions cannot be unambiguously linked to specific calls, which limits analyses at the level of individual vocalizations. This work introduces a framework for time-frequency localized bird classification. A Local-Context Classifier (LCC) identifies species from localized time-frequency events (TFEs), while a Dual-Context Classifier (DCC) combines local and global acoustic context through a fine-tuned bioacoustic foundation model. On an in-distribution dataset from Singapore comprising 306 vocalization classes from 103 bird species, the LCC achieves an F1-micro score of 79.3%, while combining local and global context through the DCC yields the highest overall performance (94.6%). To reduce labeled data requirements, the LCC is pre-trained via self-supervised contrastive learning, achieving an 18.8% relative gain on an out-of-distribution dataset. A focused evaluation on continuous soundscape recordings further demonstrates the potential of the framework for long-term monitoring applications. By preserving the time-frequency localization of individual vocalizations, the proposed framework supports both ecological monitoring and vocalization-level studies of animal acoustic behavior.

Simen Hexeberg, Mandar Chitre, Matthias Hoffmann-Kuhnt +1

ARL, Tropical Marine Science Institute, National University of Singapore · Department of Electrical & Computer Engineering, National University of Singapore · National Parks Board, Singapore

BioacousticsAudio ClassificationBird Vocalization Classification
AskPDF
Oct 22, 2024

Classifying bioacoustic data without individual call annotations using temporal convolutional networks and feature extractors

Bioacoustic data from Passive Acoustic Monitoring (PAM) generates large datasets where obtaining detailed auditing and labelling is often impractical, resulting in weak annotations (e.g., presence/absence of species over several minutes of recording). In order to effectively capture the complex temporal patterns and key features of long audio segments, we propose a framework comprising dataset standardisation, feature extraction, and classification via Temporal Convolutional Networks (TCN). This approach eliminates the necessity for setting heuristic decision rules or creating time-consuming strong labels. To demonstrate the effectiveness of our approach, we use sperm whale (Physeter macrocephalus) click trains in 4-minute recordings as a case study, from a dataset comprising diverse sources and deployment conditions to maximise generalisability. Our TCN classifiers achieve recall rates exceeding 0.83 at a 0.13 false positive rate, comparable to agreement rates between expert annotators. We compare two methods of feature extraction, Variational AutoEncoders (VAEs) and traditional handpicking of features, and found them to yield similar performance results, with the VAE-based classifiers seeing a more stable performance across datasets and recording conditions. These results offer a way forward in leveraging numerous existing annotated bioacoustic datasets to train automatic classification models, effectively overcoming previous limitations associated with weak labels.

Laia Garrobé Fonollosa, Douglas Gillespie, Lina Stankovic +2

Sea Mammal Research Unit, School of Biology, University of St. Andrews, KY16 9TH, St. Andrews, Scotland · Electrical and Electronic Engineering, University of Strathclyde, Montrose Street, Glasgow, G1 1XJ, Scotland · Electrical and Electronic Engineering, University of Strathclyde, George Street, Glasgow, G1 1XW, Scotland

BioacousticsPassive Acoustic MonitoringAudio Classification
AskPDF
Nov 26, 2023

Topology-enhanced machine learning for speech signal processing

In artificial-intelligence-aided signal processing, existing deep learning models often exhibit a black-box structure. Here, conceptually beyond spectral analysis, we demonstrate that topological methods not only effectively capture intrinsic and complex structural information but can also enhance neural networks. We provide a transparent methodology, TopCap, to capture topological features inherent in time series for basic machine learning. Compared to prior approaches, we obtain descriptors that probe finer information such as the vibration of a time series. Notably, in classifying voiced and voiceless consonants, TopCap achieves an accuracy consistently standing in comparison with neural network models. Moreover, by integrating TopCap features into those neural networks, our approach improves upon state-of-the-art methods in terms of robustness against noise, as well as accuracy, stability, convergence of loss function, and interpretability.

Pingyao Feng, Qingrui Qu, Haiyu Zhang +4

Department of Mathematics, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, North Carolina State University, Raleigh, NC, USA. · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen), Shenzhen, China. +1

Interpretable MLSpeech ProcessingAudio ClassificationTopological Data Analysis
AskPDF
Oct 17, 2023

Robust Wake-Up Word Detection by Two-stage Multi-resolution Ensembles

Voice-based interfaces rely on a wake-up word mechanism to initiate communication with devices. However, achieving a robust, energy-efficient, and fast detection remains a challenge. This paper addresses these real production needs by enhancing data with temporal alignments and using detection based on two phases with multi-resolution. It employs two models: a lightweight on-device model for real-time processing of the audio stream and a verification model on the server-side, which is an ensemble of heterogeneous architectures that refine detection. This scheme allows the optimization of two operating points. To protect privacy, audio features are sent to the cloud instead of raw audio. The study investigated different parametric configurations for feature extraction to select one for on-device detection and another for the verification model. Furthermore, thirteen different audio classifiers were compared in terms of performance and inference time. The proposed ensemble outperforms our stronger classifier in every noise condition.

Fernando López, Jordi Luque, Carlos Segura +1

Telefónica I+D, Digital Life Disruption Lab, Spain · Universidad Autónoma de Madrid · Telefónica I+D, Research, Spain

Keyword SpottingEnsemble LearningAudio ClassificationOn-Device Inference
AskPDF