Self-Supervised Speech Representation Learning

Overview

This field studies how models can learn useful speech features from unlabeled audio, reducing dependence on costly transcripts and annotations. Common approaches include contrastive prediction, masked-frame prediction with clustered or quantized targets, and newer predictive and reconstruction objectives; models are evaluated with phonetic ABX tests and benchmarks such as SUPERB, then used for speech recognition, speaker analysis, and generation. Multilingual and domain-adapted pretraining extends these methods to low-resource languages, clinical and child speech, streaming audio, and animal vocalizations. A central challenge is choosing what information to preserve: suppressing speaker, channel, or language variation may help some tasks but remove tone, accent, identity, or acoustic detail needed for others, while strong in-domain results do not guarantee fair cross-domain performance.

Latest papers

CardsList

108 papers9 papers in the last four weeks, down 62% on the four weeks before. 0.1% of all new papers.

  1. Factorized Self-Supervised Speech Tokenization

    Nov 13, 2026Benjamin Van Niekerk, Jean-Philippe Letendre, Nicol Visser +3Audio TokenizationDiscrete Speech Representation Learning

  2. What Do Multilingual Self-Supervised Speech Encoders Know About Sentence Meaning?

    Nov 13, 2026Kevin Krahn, Eric Fosler-LussierSelf-Supervised Speech Representation LearningSemantic Representation Learning

  3. Distributionally Robust BEST-RQ For Speech Representation Learning

    Nov 10, 2026Muhammad Khan, Xiaodong Cui, Brian Kingsbury +1Audio Representation LearningSelf-Supervised Speech Representation Learning

  4. Understanding When and Where Self-Supervised Speech Models Learn Phonemes and Prosodic Cues through Layer-Wise and Time-Resolved Probing

    Nov 8, 2026Masaru Tanibata, Shun Takahashi, Hiroki Ouchi +1Self-Supervised Speech Representation LearningSpeech Prosody

  5. Drowning in Noise: On Signal Saliency in Self-Supervised Learning for Ocean Acoustics

    Nov 4, 2026Shikhar Bharadwaj, Samuele Cornell, Shinji Watanabe +1Training Data SelectionBioacoustics

  6. GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

    Oct 31, 2026Yosuke Higuchi, Tom Fizycki, Mahi Luthra +14Speech Language ModelsDiscrete Speech Representation Learning

  7. S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

    Oct 26, 2026Jincheng Wang, Li Li, Weixin Zhu +4Speaker DiarizationSpeaker Embedding

  8. When Do Self-Supervised Representations Help Speech Restoration?

    Oct 22, 2026Yongjoon Lee, Younghoo Kwon, Jung-Woo ChoiSelf-Supervised LearningSpeech Enhancement

  9. CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

    Oct 18, 2026Jaehong Myung, Seungho Eum, Taegon Kim +2Audio-Driven Facial AnimationAudio-Visual Representation Learning

  10. Toward a Universal Local Speech Feature Extractor Through Distillation

    Oct 17, 2026Jessica Yang, Marcelo Beramendi Caballero, Ju-Chieh Chou +1Speech ProcessingSelf-Supervised Speech Representation Learning

  11. Interpreting SSL Representations for Spoof Detection: a WavLM Study

    Oct 16, 2026Mohamed Mallat, Michele Panariello, Massimiliano Todisco +2Audio Deepfake DetectionSpeech Processing

  12. Learning Speech Attribute Representations via Multi-Teacher Distillation

    Oct 15, 2026Maryem Bouziane, Salima Mdhaffar, Yannick EstèveRepresentation LearningMulti-Teacher Knowledge Distillation

  13. Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectory

    Oct 12, 2026Burin Naowarat, Hao Tang, Sharon GoldwaterNeural Representation GeometrySelf-Supervised Speech Representation Learning

  14. From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

    Oct 9, 2026Yaroslav Getman, Tamás Grósz, Tommi Lehtonen +1ConformerSpeech Processing

  15. Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders

    Oct 7, 2026Beimnet Bekele Guta, Xiaoyu Yang, Guangzhi Sun +1Disentangled Representation LearningSelf-Supervised Speech Representation Learning

  16. Mitigating Accent-Language Confusion in Self-Supervised Speech Representations for Language Identification

    Oct 7, 2026Minu Kim, Jihwan Lee, David R. Mortensen +1Language IdentificationAlgorithmic Bias

  17. Do Speech Representations Preserve Regional Accent Across Read and Spontaneous Speech?

    Oct 5, 2026Paula A. Perez-Toro, Tomas Arias-Vergara, Annette Schwarz +4Audio Representation LearningSelf-Supervised Speech Representation Learning

  18. SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation

    Oct 3, 2026Séverin Baroudi, Hervé Bredin, Ricard MarxerSelf-Supervised Speech Representation LearningSpeech Separation

  19. What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

    Oct 2, 2026Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong ChngSelf-Supervised Knowledge DistillationKnowledge Distillation

  20. Beyond Decodability: Do Acoustic Factors Drive Predictions in Speech-Based Alzheimer's Assessment?

    Oct 1, 2026Serli Kopar, Alkis Koudounas, Roshan P. Rane +3Audio Representation LearningSpeech-Based Dementia Detection

  21. GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

    Sep 29, 2026Gaspard Botté, Séverin Baroudi, Samir Sadok +5Joint-Embedding Predictive ArchitecturePredictive Representation Learning

  22. GLAD: Global-Local Adaptive Detector for Robust Speech Deepfake Detection

    Sep 28, 2026Zelin Zhao, Guanjie Huang, Danny Hin Kwok Tsang +1Audio Deepfake DetectionDomain Generalization

  23. OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations

    Sep 28, 2026Faadil Mustun, Chiara Semenzin, Roberto Dessi +7Sound Event DetectionBioacoustics

  24. SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

    Sep 28, 2026Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos +3Representation LearningSelf-Supervised Speech Representation Learning

  25. What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection

    Sep 27, 2026Jiajun Xu, Menglu Li, Xiao-Ping ZhangAudio Representation LearningAudio Deepfake Detection

  26. Do Audio Language Models Hear and Read Distinctive Features Alike?

    Sep 24, 2026Yuanhao Chen, Peter ChinAudio-Text AlignmentAudio-Language Models