Self-Supervised Speech Representation Learning

Ask about this topic

Overview

This field studies how models can learn useful speech features from unlabeled audio, reducing dependence on costly transcripts and annotations. Common approaches include contrastive prediction, masked-frame prediction with clustered or quantized targets, and newer predictive and reconstruction objectives; models are evaluated with phonetic ABX tests and benchmarks such as SUPERB, then used for speech recognition, speaker analysis, and generation. Multilingual and domain-adapted pretraining extends these methods to low-resource languages, clinical and child speech, streaming audio, and animal vocalizations. A central challenge is choosing what information to preserve: suppressing speaker, channel, or language variation may help some tasks but remove tone, accent, identity, or acoustic detail needed for others, while strong in-domain results do not guarantee fair cross-domain performance.

Latest papers

CardsList

101 papers18 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

  1. Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR

    Mar 6, 2026Ajinkya Kulkarni, Sandipana Dowerah, Atharva Kulkarni +2Audio Deepfake DetectionAudio Deepfake Detection Robustness

    Idiap Research Institute, Switzerland · Tallinn University of Technology, Estonia · MBZUAI, UAE

  2. Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features

    Mar 3, 2026Kyle Janse van Rensburg, Benjamin van Niekerk, Herman KamperSelf-Supervised Speech Representation LearningAudio Representation Learning

    Department of Electrical and Electronic Engineering, Stellenbosch University, South Africa · Concordia University, Canada

  3. Bias and Fairness in Self-Supervised Acoustic Representations for Cognitive Impairment Detection

    Mar 3, 2026Kashaf Gulzar, Korbinian Riedhammer, Elmar Nöth +2Speech-Based Dementia DetectionClinical Speech Processing

    Zentrum für Künstliche Intelligenz (KIZ), Technische Hochschule Nürnberg, Nürnberg, Germany · Pattern Recognition Lab, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany

  4. A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection

    Mar 2, 2026Hashim Ali, Nithin Sai Adupa, Surya Subramani +1Audio Deepfake DetectionDeepfake Detection

    Electrical and Computer Engineering, University of Michigan, 4901 Evergreen Rd, Dearborn, MI 48128, USA

  5. Polynomial Mixing for Efficient Self-supervised Speech Encoders

    Feb 28, 2026Eva Feillet, Ryan Whetten, David Picard +1Speech ProcessingSelf-Supervised Speech Representation Learning

    Universit´e Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Num´erique · Miles team, Universit´e Paris-Dauphine-PSL · Laboratoire Informatique d’Avignon, Avignon Universit´e +1

  6. Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context

    Feb 27, 2026Keita Goto, Takashi Maekaku, Jin Sakuma +3Self-Supervised Speech Representation LearningPredictive Representation Learning

    LY Corporation, Tokyo, Japan · Carnegie Mellon University, PA, USA

  7. [b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic

    Feb 21, 2026Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2Self-Supervised Speech Representation Learning

    UT Austin · UC Berkeley · CMU

  8. SA-SSL-MOS: Self-supervised Learning MOS Prediction with Spectral Augmentation for Generalized Multi-Rate Speech Assessment

    Feb 16, 2026Fengyuan Cao, Xinyu Liang, Fredrik Cumlin +4Speech Quality AssessmentSelf-Supervised Speech Representation Learning

    KTH Royal Institute of Technology, Stockholm, Sweden · Google LLC

  9. SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages

    Jan 14, 2026Tianyi Xu, Xuan Ouyang, Binwei Yao +4Audio Representation LearningSelf-Supervised Speech Representation Learning

    University of Wisconsin–Madison, Madison, WI, USA

  10. MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

    Dec 22, 2025Angelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun +2Speech ProcessingSelf-Supervised Speech Representation Learning

    ENS, PSL Research University, EHESS, CNRS, Paris, France · Meta AI Research, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France

  11. BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings

    Sep 18, 2025Théo Charlot, Tarek Kunze, Maxime Poli +3Speech Foundation ModelsSpeech Processing

    LSCP, DEC, ENS, EHESS, CNRS, PSL University, France · Laboratoire d’Informatique et Syst`emes, Universit´e Aix-Marseille, CNRS, France