Discrete Speech Representation Learning

Overview

The field studies how to encode speech as discrete units that preserve useful information while making it easier for models to learn and generate speech. Common approaches cluster features from self-supervised speech encoders, learn vector-quantized codec tokens, or align units with phonemes, syllables, or text; some combine discrete tokens with continuous features. These representations are used in tasks such as speech recognition, synthesis, retrieval, translation, and emotion analysis, with language modeling and codec reconstruction serving as common tests. A central challenge is balancing compact, predictable tokens with acoustic fidelity and the preservation of speaker identity, emotion, and prosody, especially across languages and in streaming settings.

Latest papers

CardsList

52 papers5 papers in the last four weeks, down 44% on the four weeks before. 0.0% of all new papers.

  1. Factorized Self-Supervised Speech Tokenization

    Nov 13, 2026Benjamin Van Niekerk, Jean-Philippe Letendre, Nicol Visser +3Audio TokenizationDiscrete Speech Representation Learning

  2. Distributionally Robust BEST-RQ For Speech Representation Learning

    Nov 10, 2026Muhammad Khan, Xiaodong Cui, Brian Kingsbury +1Audio Representation LearningSelf-Supervised Speech Representation Learning

  3. GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

    Oct 31, 2026Yosuke Higuchi, Tom Fizycki, Mahi Luthra +14Speech Language ModelsDiscrete Speech Representation Learning

  4. Probing the phon* representations of the Mimi codebooks

    Oct 26, 2026Artem Saloev, Nicolas Ballier, Erin PacquetetNeural Audio CodecsDiscrete Speech Representation Learning

  5. Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics

    Oct 14, 2026Jihwan Lee, Tiantian Feng, Minu Kim +6Neural Audio CodecsSpeech Processing

  6. Revise-TTS: A Non-Masking Refinement Framework for Masked Diffusion Models in Text-to-Speech

    Oct 14, 2026Wanli Sun, Dan Su, Anton RagniTTS SynthesisZero-Shot TTS

  7. SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations

    Oct 10, 2026Kecan Mao, Nan Xu, Zhaoyang Meng +4Neural Audio CodecsAudio Representation Learning

  8. SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation

    Oct 3, 2026Séverin Baroudi, Hervé Bredin, Ricard MarxerSelf-Supervised Speech Representation LearningSpeech Separation

  9. Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries

    Sep 29, 2026Han Wang, Jiaqi Li, Yingda Shen +2Neural Audio CodecsAudio Tokenization

  10. SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

    Sep 28, 2026Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos +3Representation LearningSelf-Supervised Speech Representation Learning

  11. A Comprehensive Study of Content Representations for Speech Synthesis

    Sep 25, 2026Diego Torres, Axel Roebel, Nicolas ObinAudio Representation LearningDiscrete Speech Representation Learning

  12. Beyond Encoder Fusion: Multi-View Discrete Token Augmentation for LLM-Based ASR

    Sep 20, 2026Paul Moïse Gangbadja, Mickael Rouvier, Fabrice LefèvreMulti-View LearningAutomatic Speech Recognition

  13. Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning

    Sep 11, 2026Hayato Futami, Hassan Shahmohammadi, Tushar Dhyani +5Speech TranslationSpeech Language Models

  14. StreamAlign: Streaming Text-Aligned Speech Tokenization

    Sep 9, 2026Kang-wook Kim, Jinyoung Park, Jinsoo Kim +3Audio TokenizationSpeech Language Models

  15. BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization via Bidirectional State-Space Modeling

    Sep 1, 2026Xin Zhang, Lin Li, Chuanbo Liu +2Neural Audio CodecsVector Quantization

  16. Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPA

    Aug 19, 2026Wenxuan He, Yunpeng Li, Zewei Li +4Soft-Label LearningUnsupervised Clustering

  17. Using the Mimi codec for metalinguistic representations

    Aug 16, 2026Artem Saloev, Erin Pacquetet, Nicolas BallierSpeech ProcessingDiscrete Speech Representation Learning

  18. ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure

    Aug 8, 2026Zixiang Wan, Xusheng Yang, Zheng Wang +1Neural Audio CodecsAudio Tokenization

  19. Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

    Jul 28, 2026Daigo Takizawa, Tomohiko Nakamura, Samuele Cornell +3Audio Representation LearningNeural Audio Codecs

  20. Content is What Remains: Invariant Speech Tokenization from Parallel Utterances

    Jul 21, 2026Laurin Wagner, Bernhard Thallinger, Miroslav Stankovic +1Audio TokenizationSelf-Supervised Speech Representation Learning

  21. AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis

    Jul 17, 2026Zhenqi Jia, Yuan Zhao, Aruukhan +2TTS SynthesisEmotional Speech Synthesis

  22. UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling

    Jun 30, 2026Chuanbo Zhu, Wuyou Zhou, Rongxiu Zhong +4Audio EditingSpeech Editing

  23. HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

    Jun 26, 2026Artem Ploujnikov, Francesco Verdini, Samir Sadok +1Neural Audio CodecsMultimodal Token Compression

  24. wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval

    Jun 25, 2026Adhiraj Banerjee, Vipul AroraAudio Representation LearningSpeech Processing

  25. On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models

    Jun 22, 2026Shunsuke Kando, Wataru Nakata, Shinnosuke Takamichi +1Speech ProcessingSpeech Language Models

  26. Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal

    Jun 18, 2026Syeda Faiza Ahmed Sara, Shammur Absar ChowdhurySpeech ProcessingAutomated Spoken Language Assessment