Discrete Speech Representation Learning

The field studies how to encode speech as discrete units that preserve useful information while making it easier for models to learn and generate speech. Common approaches cluster features from self-supervised speech encoders, learn vector-quantized codec tokens, or align units with phonemes, syllables, or text; some combine discrete tokens with continuous features. These representations are used in tasks such as speech recognition, synthesis, retrieval, translation, and emotion analysis, with language modeling and codec reconstruction serving as common tests. A central challenge is balancing compact, predictable tokens with acoustic fidelity and the preservation of speaker identity, emotion, and prosody, especially across languages and in streaming settings.