Self Supervised Speech Representation

Self-supervised speech representation learning aims to create powerful speech embeddings from vast amounts of unlabeled audio data, improving downstream tasks like speech recognition and enhancement without relying heavily on transcribed data. Current research focuses on refining model architectures like Wav2Vec 2.0, HuBERT, and XLSR, investigating the properties of these representations (e.g., orthogonality of speaker and phonetic information), and addressing biases in performance across different language varieties. This field is significant because it enables advancements in speech technology for low-resource languages and diverse speaker populations, while also providing insights into the fundamental nature of speech representation itself.

Papers

August 26, 2024

Self-supervised Speech Representations Still Struggle with African American Vernacular English
Kalvin Chang, Yi-Hui Chou, Jiatong Shi, Hsuan-Ming Chen, Nicole Holliday, Odette Scharenborg, David R. Mortensen
Zero Shot Automatic Speech Recognition System Speech Model Automatic Speech Recognition Performance Self Supervised Speech Representation Speech Supervised Learning Model African American

July 18, 2024

Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement
Robert Sutherland, George Close, Thomas Hain, Stefan Goetze, Jon Barker
Loss Function Speech Enhancement Speech Intelligibility Self Supervised Speech Representation Speech Foundation Model Speech Enhancement Model Self Supervised Speech Representation Model

June 30, 2024

Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations
Salah Zaiem, Titouan Parcollet, Slim Essid
Language Model Fine Tuning Self Supervised Speech Recognition Domain Generalization Better Generalization Self Supervised Speech Representation Downstream Fine Tuning

June 13, 2024

Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations
Mukhtar Mohamed, Oli Danyi Liu, Hao Tang, Sharon Goldwater
Speaker Diarization Representation Space Self Supervised Speech Representation Post Hoc Orthogonalization Speech Driven Phonetic Information Speech Application

June 12, 2024

Self-Supervised Speech Representations are More Phonetic than Semantic
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, Shinji Watanabe
Speech Model Semantic Description Intent Classification Self Supervised Speech Representation Language Encoders

June 7, 2024

Towards objective and interpretable speech disorder assessment: a comparative analysis of CNN and transformer-based models
Malo Maisonneuve, Corinne Fredouille, Muriel Lalain, Alain Ghio, Virginie Woisard
Convolutional Neural Network Comparative Study Transformer Based Model Self Supervised Speech Representation Language Disorder Wav2Vec2 Model

April 2, 2024

Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context
Antoine Caubrière, Elodie Gauthier
Automatic Speech Recognition African Language Self Supervised Speech Representation Unlabeled Speech Multilingual Speech Representation

March 13, 2024

Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations
Amit Meghanani, Thomas Hain
Speech Model Self Supervised Speech Representation Speech Supervised Learning Model Acoustic Word Embeddings Based Correspondence

February 2, 2024

Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations
Panos Kakoulidis, Nikolaos Ellinas, Georgios Vamvoukakis, Myrsini Christidou, Alexandra Vioni, Georgia Maniati, Junkwang Oh, Gunu Jho, Inchul Hwang, Pirros Tsiakoulis, Aimilios Chalamandaris
Self Supervised Speech Representation Singing Voice Synthesis High Fidelity Vocoder Multi Speaker Virtual Singer

January 10, 2024

Self-supervised speech representation and contextual text embedding for match-mismatch classification with EEG recording
Bo Wang, Xiran Xu, Zechen Zhang, Haolin Zhu, YuJie Yan, Xihong Wu, Jing Chen
Context Information Speech Representation Self Supervised Speech Representation Electroencephalography Recording EEG Signal Analysis Electroencephalogram Data Mismatch Classification

January 8, 2024

LUPET: Incorporating Hierarchical Information Path into Multilingual ASR
Wei Liu, Jingyong Hou, Dong Yang, Muyong Cao, Tan Lee
Speech Recognition Multilingual Automatic Speech Recognition Self Supervised Speech Representation Hierarchical Information Language Specific Module

December 15, 2023

Fine-Tuned Self-Supervised Speech Representations for Language Diarization in Multilingual Code-Switched Speech
Geoffrey Frost, Emily Morris, Joshua Jansen van Vüren, Thomas Niesler
Self Supervised Speech Representation Code Switched Diarization Error Rate Language Diarization

December 14, 2023

Acoustic models of Brazilian Portuguese Speech based on Neural Transformers
Marcelo Matheus Gauy, Marcelo Finger
Transformer Based Acoustic Model Self Supervised Speech Representation Deep Transformer Brazilian Portuguese Speech Biomarkers

November 25, 2023

Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching
Tolúlopé Ògúnrèmí, Christopher D. Manning, Dan Jurafsky
Self Supervised Speech Recognition African Language Self Supervised Speech Representation Gram Language Model Code Switched Data

October 5, 2023

Evaluating Self-Supervised Speech Representations for Indigenous American Languages
Chih-Chen Chen, William Chen, Rodolfo Zevallos, John E. Ortega
Low Resource Speech Representation Self Supervision Speech Corpus Monolingual Model Self Supervised Speech Representation Indigenous Language

September 22, 2023

VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks
Heitor R. Guimarães, Arthur Pimentel, Anderson Avila, Tiago H. Falk
Adversarial Attack Knowledge Distillation Adversarial Robustness Self Supervised Speech Representation Google Speech Command Keyword Enrollment Latent Distillation

August 29, 2023

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos
Ji-Hoon Kim, Jaehun Kim, Joon Son Chung
Sound Design Self Supervised Speech Representation High Quality Speech Lip to Speech Synthesis Silent Video Lip to Speech

August 28, 2023

August 2, 2023

SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis
Ramanan Sivaguru, Vasista Sai Lodagala, S Umesh
Synthesized Speech Speech Quality Self Supervised Speech Representation Text to Speech Synthesis Fastspeech2 Architecture

Self Supervised Speech Representation

Papers

Self-supervised Speech Representations Still Struggle with African American Vernacular English

Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement

Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations

Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations

Self-Supervised Speech Representations are More Phonetic than Semantic

Towards objective and interpretable speech disorder assessment: a comparative analysis of CNN and transformer-based models

Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context

Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations

Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations

Self-supervised speech representation and contextual text embedding for match-mismatch classification with EEG recording

LUPET: Incorporating Hierarchical Information Path into Multilingual ASR

Fine-Tuned Self-Supervised Speech Representations for Language Diarization in Multilingual Code-Switched Speech

Acoustic models of Brazilian Portuguese Speech based on Neural Transformers

Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching

Evaluating Self-Supervised Speech Representations for Indigenous American Languages

VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos

Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads

Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Tasks

SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis