Audio Representation Learning

Ask about this topic

Overview

Audio representation learning studies how to encode waveforms or spectrograms into features that preserve information useful for tasks such as speech recognition, sound classification, music retrieval, and generation. Common approaches include self-supervised masked prediction in wav2vec 2.0, HuBERT, and BEST-RQ; contrastive audio-language learning such as CLAP; and distillation or autoencoder-based continuous and discrete audio tokenizers. Researchers assess transfer with linear probes and fine-tuning on benchmarks including SUPERB, ESC-50, MARBLE, and X-ARES, while also probing which acoustic, semantic, spatial, or speaker attributes the features retain. A central tension is how to make representations broadly transferable and compact while preserving fine acoustic detail, disentangling nuisance factors, and supporting both understanding and high-fidelity generation.

Latest papers

CardsList

211 papers30 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

  1. Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)

    May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Audio-Language Model EvaluationLLM Evaluation

    Google USA & Germany

  2. PHALAR: Phasors for Learned Musical Audio Representations

    May 5, 2026Davide Marincione, Michele Mancusi, Giorgio Strano +4Audio Representation LearningRepresentation Learning

    Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.

  3. Shao: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation

    May 3, 2026Jiafeng Liu, Yuanliang Dong, Hongjia Liu +8Text-to-Music GenerationAudio Representation Learning

    1Central Conservatory of Music · 2Tsinghua University

  4. LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation

    May 1, 2026Venkata Pushpak Teja MentaAdversarial TrainingAdversarial Representation Learning

    Praxel Ventures

  5. Alethia: A Foundational Encoder for Voice Deepfakes

    Apr 30, 2026Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram +1Audio Representation LearningAudio Deepfake Detection

    Reality Defender Inc.

  6. From Birdsong to Rumbles: Classifying Elephant Calls with Out-of-Species Embeddings

    Apr 30, 2026Christiaan M. Geldenhuys, Thomas R. NieslerAudio Representation LearningAudio Understanding

    Department of Electrical and Electronic Engineering University of Stellenbosch

  7. Beyond the Baseband: Adaptive Multi-Band Encoding for Full-Spectrum Bioacoustics Classification

    Apr 30, 2026Eklavya Sarkar, Marius Miron, David Robinson +6Audio Representation LearningFrequency-Domain Feature Learning

    Earth Species Project

  8. Multimodal LLMs are not all you need for Pediatric Speech Language Pathology

    Apr 29, 2026Darren Fürst, Sebastian Steindl, Ulrich SchäferAudio Representation LearningHealthcare

    Ostbayerische Technische Hochschule Amberg-Weiden

  9. S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models

    Apr 27, 2026Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2Model CompressionAudio Representation Learning

    LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France

  10. Enhancing Speaker Verification with Whispered Speech via Post-Processing

    Apr 22, 2026Magdalena Gołębiowska, Piotr SygaAudio Representation LearningSpeaker Verification

    Department of Artificial Intelligence, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego 27, Wroclaw, 50-370, Poland

  11. Audio Spoof Detection with GaborNet

    Apr 21, 2026Waldek MaciejkoAudio Representation LearningAudio Deepfake Detection

  12. On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation

    Apr 14, 2026Changhao Cheng, Wei Wang, Wangyou Zhang +4Variational AutoencodersAudio Representation Learning

    School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · Auditory Cognition and Computational Acoustics Lab, MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · ByteDance Seed, China

  13. Cross-Cultural Bias in Mel-Scale Representations: Evidence and Alternatives from Speech and Music

    Apr 12, 2026Shivam Chauhan, Ajay PundhirAudio Representation LearningAudio Classification

    Presight AI, Abu Dhabi, United Arab Emirates

  14. Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

    Apr 12, 2026Jielin Qiu, Ming Zhu, Wenting Zhao +11Audio Representation LearningDomain Adaptation

    Salesforce AI Research

  15. Masked Contrastive Pre-Training Improves Music Audio Key Detection

    Apr 11, 2026Ori Yonay, Tracy Hammond, Tianbao YangMusic Information RetrievalAudio Representation Learning

    Department of Computer Science and Engineering, Texas A&M University

  16. Unmixing The Crowd: Learning Persistent Speaker Representations from Mixture-Derived Multi-Speaker Embeddings

    Apr 3, 2026Sidharth Sidharth, Meysam Asgari, Hao-Wen Dong +1Audio Representation LearningBiometric Identification

    University of Michigan, Ann Arbor, MI, USA · Oregon Health & Science University, Portland, OR, USA

  17. FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining

    Apr 1, 2026Xiquan Li, Xuenan Xu, Ziyang Ma +4Audio-Language ModelsAudio Representation Learning

    X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1

  18. Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

    Mar 31, 2026Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov +3TTS SynthesisSpeech Prosody

    MTUCI, Moscow, Russia

  19. Can pre-trained Deep Learning models predict groove ratings?

    Mar 28, 2026Axel Marmoret, Nicolas Farrugia, Jan Alexander StupacherMusic Information RetrievalAudio Representation Learning

    IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France · Center for Music In the Brain, Aarhus University, Århus, Denmark

  20. Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

    Mar 28, 2026Axel MarmoretAudio Representation LearningMusic Information Retrieval

    BRAIN team, MEE, IMT Atlantique, Brest, France

  21. A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning

    Mar 27, 2026Harunori Kawano, Takeshi SasakiAudio Representation LearningSelf-Supervised Pre-Training

    Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology

  22. Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods

    Mar 26, 2026Xuanru Zhou, Yiwen Shao, Wei-Cheng Tseng +1Audio Representation LearningAudio Classification

    UT-Austin, Austin, USA · Tencent AI Lab, Bellevue, USA

  23. Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

    Mar 25, 2026Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda +3Audio Representation LearningSelf-Supervised Pre-Training

    Tokyo Metropolitan University, Tokyo, Japan · NTT Inc., Atsugi, Japan

  24. The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models

    Mar 24, 2026Heinrich Dinkel, Jiahao Zhou, Guanbo Wang +8Audio-Language Model EvaluationAudio Representation Learning

    MiLM Plus, Xiaomi Inc., Beijing, China · DataoceanAI Inc., USA · Centre for Vision Speech and Signal Processing, University of Surrey, Guildford, UK +1

  25. AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference

    Mar 23, 2026Risa Shinoda, Kaede Shiohara, Nakamasa Inoue +2Audio-Language ModelsBioacoustics

    The University of Osaka · The University of Tokyo · Institute of Science Tokyo

  26. HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit

    Mar 22, 2026Khushiyant, Param ThakkarAudio Representation LearningHybrid Mamba-Transformer Architectures

    Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany · Department of Computer Engineering and Information Technology, Veermata Jijabai Technological Institute, Mumbai, India

  27. Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models

    Mar 17, 2026Joseph Cameron, Alan BlackwellVariational AutoencodersAudio Representation Learning

    Department of Computer Science & Technology, University of Cambridge, Cambridge, United Kingdom

  28. Spectrogram features for audio and speech analysis

    Mar 16, 2026Ian McLoughlin, Lam Pham, Yan Song +7Audio Representation LearningFrequency-Domain Feature Learning

    Singapore Institute of Technology · Austrian Institute of Technology · The University of Science and Technology of China +1

  29. Probing neural audio codecs for distinctions among English nuclear tunes

    Mar 14, 2026Juan Pablo Vigneaux, Jennifer ColeSpeech ProsodyNeural Audio Codecs

    Northwestern University

  30. Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track

    Mar 14, 2026Jiawen Huang, Chenxi Huang, Zhuofan Wen +7Emotion RecognitionMultimodal Emotion Recognition

    School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tianjin Normal University