Audio Representation Learning

Ask about this topic

Overview

Audio representation learning studies how to encode waveforms or spectrograms into features that preserve information useful for tasks such as speech recognition, sound classification, music retrieval, and generation. Common approaches include self-supervised masked prediction in wav2vec 2.0, HuBERT, and BEST-RQ; contrastive audio-language learning such as CLAP; and distillation or autoencoder-based continuous and discrete audio tokenizers. Researchers assess transfer with linear probes and fine-tuning on benchmarks including SUPERB, ESC-50, MARBLE, and X-ARES, while also probing which acoustic, semantic, spatial, or speaker attributes the features retain. A central tension is how to make representations broadly transferable and compact while preserving fine acoustic detail, disentangling nuisance factors, and supporting both understanding and high-fidelity generation.

Latest papers

CardsList

211 papers30 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

  1. Evaluating Compositional Structure in Audio Representations

    Mar 14, 2026Chuyang Chen, Bea Steers, Brian McFee +1Compositional Representation LearningAudio Representation Learning

    Music and Audio Research Laboratory, New York University, USA

  2. Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics

    Mar 11, 2026Yangyang Qu, Massimiliano Todisco, Chiara Galdi +1Speaker VerificationAlgorithmic Fairness

    EURECOM Sophia Antipolis, France

  3. Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

    Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker EmbeddingSpeaker Verification

    Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

  4. Speech Codec Probing from Semantic and Phonetic Perspectives

    Mar 11, 2026Xuan Shi, Chang Zeng, Tiantian Feng +3Audio Representation LearningNeural Audio Codecs

    University of Southern California, USA · Dolby Laboratories, USA

  5. ALARM: Audio-Language Alignment for Reasoning Models

    Mar 10, 2026Petr Grinberg, Hassan ShahmohammadiAudio-Language ModelsAudio Reasoning

    EPFL, Switzerland · Sony Europe Ltd., Germany

  6. TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

    Mar 10, 2026Shihao He, Yihan Xia, Fang Liu +2Music Information RetrievalAudio Representation Learning

    College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China

  7. Universal Speech Content Factorization

    Mar 9, 2026Henry Li Xinyuan, Zexin Cai, Lin Zhang +5Voice ConversionSpeaker Disentanglement

    Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA · Center for Language and Speech Processing, Johns Hopkins University, USA

  8. Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

    Mar 9, 2026Maryem Bouziane, Salima Mdhaffar, Yannick EstèveSpeech ProcessingAudio Representation Learning

    LIA - Avignon Université, France

  9. SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval

    Mar 9, 2026Ruixiang Zhao, Zhihao Xu, Bangxiang Lan +3Audio-Visual Representation LearningVideo Representation Learning

    Renmin University of China

  10. Koopman Regularized Deep Speech Disentanglement for Speaker Verification

    Mar 5, 2026Nikos Chazaridis, Mohammad Belal, Rafael Mestre +2Speaker VerificationSpeaker Disentanglement

    School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, U.K. · Marine Geosciences, National Oceanography Centre, SO14 3ZH Southampton, U.K.

  11. Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features

    Mar 3, 2026Kyle Janse van Rensburg, Benjamin van Niekerk, Herman KamperSelf-Supervised Speech Representation LearningAudio Representation Learning

    Department of Electrical and Electronic Engineering, Stellenbosch University, South Africa · Concordia University, Canada

  12. AG-REPA: Causal Layer Selection for Representation Alignment in Audio Flow Matching

    Mar 1, 2026Pengfei Zhang, Tianxin Xie, Minghao Yang +1Flow-Based Generative ModelingFlow Matching

    AI Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou).

  13. An Empirical Analysis of Task-Induced Encoder Bias in Fréchet Audio Distance

    Feb 27, 2026Wonwoo JeongText-to-Audio GenerationAudio Representation Learning

    Dept. of Computer Science and Engineering, Sogang University, South Korea

  14. DashengTokenizer: One layer is enough for unified audio understanding and generation

    Feb 27, 2026Heinrich Dinkel, Xingwei Sun, Gang Li +8Audio TokenizationAudio Representation Learning

    MiLM Plus, Xiaomi Inc., Beijing, China

  15. SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents

    Feb 26, 2026Zeyu Xie, Chenxing Li, Qiao Jin +6Neural VocodersText-to-Audio Generation

    Peking University, Shenzhen · Tencent AI Lab, Shenzhen · Shanghai AI Lab, Shanghai +1

  16. UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

    Feb 25, 2026Yuxuan Chen, Peize He, Haoyuan Yu +1Audio Representation LearningAudio Understanding

    Jilin University · University of Electronic Science and Technology of China · Hunan University +1

  17. Musical Training, but not Mere Exposure to Music, Drives the Emergence of Chroma Equivalence in Artificial Neural Networks

    Feb 20, 2026Lukas Grasse, Matthew S. TataAudio Representation Learning

    Canadian Centre for Behavioural Neuroscience, Department of Neuroscience, University of Lethbridge, AB, Canada, T1K 3Y4

  18. BAT: Better Audio Transformer Guided by Convex Gated Probing

    Feb 18, 2026Houtan Ghaffari, Lukas Rauch, Christoph Scholz +1Audio Representation LearningSelf-Supervised Pre-Training

    Ghent University · University of Kassel · Now at Earth Species Project +1

  19. A Generative-First Neural Audio Autoencoder

    Feb 17, 2026Jonah Casebeer, Ge Zhu, Zhepei Wang +1AutoencodersAudio Representation Learning

    Adobe Research

  20. What Do Neurons Listen To? A Neuron-level Dissection of a General-purpose Audio Model

    Feb 17, 2026Takao Kawamura, Daisuke Niizumi, Nobutaka OnoAudio Representation LearningNeural Network Interpretability

    Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan

  21. Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings

    Jan 16, 2026Joanne Affolter, Benjamin Martin, Elena V. Epure +2Audio Representation LearningMusic Information Retrieval

    Deezer Research - 24 Rue de Calais, 75009 Paris · EPFL DHLAB - Rte Cantonale, 1015 Lausanne

  22. SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages

    Jan 14, 2026Tianyi Xu, Xuan Ouyang, Binwei Yao +4Audio Representation LearningSelf-Supervised Speech Representation Learning

    University of Wisconsin–Madison, Madison, WI, USA

  23. Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings

    Jan 13, 2026Hilde I. Hummel, Sandjai Bhulai, Rob D. van der Mei +1Audio Representation LearningLinear Probing

    Centre of Mathematics and Computer Science, Stochastics group, Science Park 123, Amsterdam, 1098 XG, Netherlands · Vrije Universiteit, Department of Mathematics, De Boelelaan 1111, Amsterdam, 1081 HV, Netherlands · Naturalis Biodiversity Center, Darwinweg 2, Leiden, 2333 CR, Netherlands

  24. Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition

    Jan 11, 2026Nathan Roll, Pranav Bhalerao, Martijn Bartelds +7Audio Representation LearningHybrid CNN-Transformer Architectures

    Stanford University

  25. The Affective Bridge: Preserving Speech Representations while Enhancing Deepfake Detection via emotional Constraints

    Dec 12, 2025Yupei Li, Chenyang Lyu, Longyue Wang +3Audio Deepfake DetectionAudio Representation Learning

  26. Harmonic-Percussive Disentangled Neural Audio Codec for Bandwidth Extension

    Nov 26, 2025Benoît Giniès, Xiaoyu Bie, Olivier Fercoq +1Audio Representation LearningNeural Audio Codecs

    LTCI, Télécom Paris, Institut Polytechnique de Paris Palaiseau, France

  27. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model EvaluationAudio Representation Learning

    University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1

  28. Perceptually Aligning Representations of Music via Noise-Augmented Autoencoders

    Nov 7, 2025Mathias Rose Bjare, Giorgia Cantisani, Marco Pasini +2Audio Representation LearningDenoising Autoencoders

    Johannes Kepler University, Linz, AT · STMS, CNRS, IRCAM, Sorbonne Universit´e, Paris, FR · Queen Mary University, London, UK +1

  29. OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder

    Jul 18, 2025Shikhar Bharadwaj, Samuele Cornell, Kwanghee Choi +4Audio Representation LearningAudio QA

    Carnegie Mellon University, USA · National Institute of Advanced Industrial Science and Technology (AIST), Japan

  30. Structured-Noise Masked Modeling for Video, Audio and Beyond

    Mar 20, 2025Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker +2Audio Representation LearningVideo Representation Learning

    University of Amsterdam · King Abdullah University of Science and Technology