Audio Representation Learning

Overview

Audio representation learning studies how to encode waveforms or spectrograms into features that preserve information useful for tasks such as speech recognition, sound classification, music retrieval, and generation. Common approaches include self-supervised masked prediction in wav2vec 2.0, HuBERT, and BEST-RQ; contrastive audio-language learning such as CLAP; and distillation or autoencoder-based continuous and discrete audio tokenizers. Researchers assess transfer with linear probes and fine-tuning on benchmarks including SUPERB, ESC-50, MARBLE, and X-ARES, while also probing which acoustic, semantic, spatial, or speaker attributes the features retain. A central tension is how to make representations broadly transferable and compact while preserving fine acoustic detail, disentangling nuisance factors, and supporting both understanding and high-fidelity generation.

Latest papers

CardsList

224 papers13 papers in the last four weeks, down 67% on the four weeks before. 0.1% of all new papers.

  1. Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy

    Nov 18, 2026Anurag Singh, Prajwal Singh, Shanmuganathan RamanCompositional Image GenerationMultimodal Diffusion Models

  2. 21Codec: Decoder-side Speaker Control for Text-to-Speech

    Nov 18, 2026David Lindevelt, Suzan Verberne, Joost BroekensText-to-SpeechNeural Audio Codecs

  3. Distributionally Robust BEST-RQ For Speech Representation Learning

    Nov 10, 2026Muhammad Khan, Xiaodong Cui, Brian Kingsbury +1Audio Representation LearningSelf-Supervised Speech Representation Learning

  4. Exploring Post-Hoc Disentanglement of Audio Foundation Models for Music Similarity

    Nov 10, 2026Lucas de Souza Brandão, Gabriel da Costa Merlin, Diego Furtado SilvaRepresentation DisentanglementDisentangled Representation Learning

  5. AudioCodex: Streaming Autoencoders for Live Audio

    Nov 9, 2026Marco Pasini, Javier Nistal, Cyran Aouameur +2Neural Audio CodecsAutoencoders

  6. AudioSphere: Towards Self-Supervised Spatial Audio Representation Models

    Nov 6, 2026Goksenin Yuksel, Marcel van Gerven, Kiki van der HeijdenAudio Representation LearningSpatial Audio

  7. Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval

    Nov 6, 2026Chen-An Li, Cheng-Han Chiang, Hung-yi LeeAudio-Text RetrievalAudio Representation Learning

  8. GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

    Oct 31, 2026Yosuke Higuchi, Tom Fizycki, Mahi Luthra +14Speech Language ModelsDiscrete Speech Representation Learning

  9. Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

    Oct 30, 2026Parthasaarathy Sudarsanam, Tuomas VirtanenAudio Representation LearningAudio Source Separation

  10. TAPE: Learning General Audio Effect Representations

    Oct 29, 2026Phillip Long, Yongyi Zang, Qihui Yang +5Differentiable Digital Signal ProcessingAudio Representation Learning

  11. AudioFM: Towards General Purpose Audio Representation Learning

    Oct 25, 2026Yamin Li, Ruidong Zhang, Anran Wang +19Audio Representation LearningTemporal Representation Learning

  12. Dual-Channel Contrastive Encoding of Sperm Whale Coda

    Oct 21, 2026João Victor Quintanilha, Emma Virnelli, Shuo ZhangAudio Representation LearningBioacoustics

  13. MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

    Oct 20, 2026Kohei Yamamoto, Kosuke OkusaAudio Representation LearningSelf-Supervised Pre-Training

  14. ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

    Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language ModelsSpeech Language Models

  15. From Signals to Evidence: Interpretable Underwater Acoustic Target Sensing with Large Language Models - Part I: Framework and Recognition

    Oct 15, 2026Yuge Zhang, Jie Chen, Zhengqiao Zhao +4Underwater AcousticsAudio Representation Learning

  16. Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

    Oct 14, 2026Jihwan Lee, Tiantian Feng, Minu Kim +6Neural Audio CodecsSpeech Processing

  17. ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition

    Oct 14, 2026Zhe Cao, Jichao Zhang, Xiao-Lei ZhangClassificationAudio Representation Learning

  18. MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces

    Oct 12, 2026Guobin Ma, Yuepeng Jiang, Dake Guo +15Audio Representation LearningCompositional Representation Learning

  19. Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection

    Oct 12, 2026Jianqiao Cui, Haoxian Zhu, Hanbiao Meng +1Audio Deepfake DetectionAudio Deepfake Detection Robustness

  20. SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations

    Oct 10, 2026Kecan Mao, Nan Xu, Zhaoyang Meng +4Neural Audio CodecsAudio Representation Learning

  21. How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders

    Oct 8, 2026Marios Glytsos, Brian McFeeAudio Representation LearningMusic Information Retrieval

  22. Input Representation, Not Model Size or Pretraining Scale, Shapes Speaker Verification Robustness to Temporal Degradation

    Oct 8, 2026Erdem Baha Topbas, Srikanth Madikeri, Elisa Pellegrino +1Speaker VerificationAudio Representation Learning

  23. Mosquito acoustic identification at unseen recording sites: practical limits of common transfer remedies for vector surveillance

    Oct 7, 2026Yucheng Gong, Rui Zhou, Binbin Zeng +2Cross-Domain GeneralizationAudio Classification

  24. Feature Encoding in VAE-based Audio Decoders: Effects of Input, Depth and Distribution

    Oct 6, 2026Louis McCallum, Mick GriersonVariational AutoencodersAudio Representation Learning

  25. Neural Representations, Natural Connections: What Transfers From Human Speech Foundation Models to Animal Vocalizations?

    Oct 5, 2026Tomás Arias-Vergara, Christopher Hauer, Héloïse Brotier +3Audio Representation LearningSpeech Foundation Models

  26. Do Speech Representations Preserve Regional Accent Across Read and Spontaneous Speech?

    Oct 5, 2026Paula A. Perez-Toro, Tomas Arias-Vergara, Annette Schwarz +4Audio Representation LearningSelf-Supervised Speech Representation Learning

  27. TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

    Oct 4, 2026Junjie Li, Zheng Liang, Zhe Li +2Speaker VerificationAudio-Language Models