Factorized Self-Supervised Speech Tokenization
Organizations: Ubisoft La Forge · Concordia University, Mila · Stellenbosch University
Abstract
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
Figures & tables
| Method | Params | Codebook | WER | SIM | Spoof -EER | MOS |
|---|---|---|---|---|---|---|
| 12.5 Hz | ||||||
| FocalCodec | 145 | 0 8192 | 5.9 | 0.90 | 31.1 | 4.35 |
| Kanade | 207 | 12800 | 3.5 | 0.93 | 41.2 | 4.35 |
| Facto | 131 | 0 8192 | 5.6 | 0.93 | 42.3 | 4.37 |
| Facto | 131 | 16384 | 5.0 | 0.93 | 42.6 | 4.37 |
| Facto (cont.) | 131 | – | 2.4 | 0.94 | 44.9 | 4.37 |
| Method | Codebook | WER | SIM | Spoof -EER | MOS | Nat. | Ref.Sim. |
| 12.5 Hz | |||||||
| FocalCodec | 0 8192 | 3.6 | 0.81 | 11.7 | 4.15 | ||
| Kanade | 12800 | 4.5 | 0.92 | 36.4 | 4.26 | ||
| Facto | 0 8192 | 2.7 | 0.94 | 46.4 | 4.36 | ||
| Facto | 16384 | 2.3 | 0.94 | 45.8 | 4.36 | ||
| 25 Hz | |||||||
| Method | Codebook | WER | SIM | Spoof -EER | MOS |
|---|---|---|---|---|---|
| 12.5 Hz | |||||
| FocalCodec | 0 8192 | 15.3 | 0.74 | 19.4 | 4.02 |
| Kanade | 12800 | 18.9 | 0.86 | 43.2 | 4.11 |
| Facto | 0 8192 | 0 9.4 | 0.90 | 53.1 | 4.31 |
| Facto | 16384 | 0 9.8 | 0.89 | 51.1 | 4.28 |
| 25 Hz | |||||
| Method | Codebook | WER | SIM | Spoof -EER | MOS |
|---|---|---|---|---|---|
| 25 Hz | |||||
| Kanade | 12800 | 0.97 | 0.93 | 34.9 | 4.37 |
| Facto | 0 8192 | 1.83 | 0.94 | 42.0 | 4.40 |
| Facto | 16384 | 1.66 | 0.94 | 41.2 | 4.40 |
| Facto (cont.) | – | 0.85 | 0.93 | 35.4 | 4.40 |
| CosyVoice2 | 0 6561 | 1.30 | 0.86 | 24.5 | 4.33 |
| sWUGGY | ||||||
| Method | Frame Rate | Codebook | All | In Vocab | sBLIMP | tSC |
| 6k hours | ||||||
| FocalCodec | 12.5 | 0 8192 | 60.0 | 64.1 | 49.8 | 61.5 |
| Kanade | 12.5 | 12800 | 61.6 | 66.3 | 50.4 | 63.4 |
| Facto | 12.5 | 0 8192 | 61.0 | 65.5 | 50.9 | 65.2 |
| Facto | 12.5 | 16384 | 60.8 | 65.0 | 51.2 | 64.5 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | X | C | S |
|---|---|---|---|
| Semantic content | |||
| CER ( | 10.87 | 8.96 | – |
| Prosody – frame level | |||
| normalized log- | 0.824 | 0.630 | – |
| normalized log-energy | 0.987 | 0.937 | – |
| Speaker timbre attributes – utterance level | |||
| Parameter | Pretraining | Fine-tuning |
|---|---|---|
| Steps | 2M | 500K |
| Learning Rate | ||
| LR Scheduler | Inverse Power | Inverse Power |
| Warmup (geometric) | 0.999 | 0.999 |
| Decay ( ) | 50000 | 50000 |
| Decay (power) | 0.5 | 0.5 |
| Method | Codebook | White | Pink | Brown | Bit Crush | Real | Avg |
|---|---|---|---|---|---|---|---|
| (SNR 25) | (SNR 22) | (SNR 16) | (BitDepth 10) | (SNR 16) | |||
| 12.5Hz | |||||||
| Kanade | 12800 | 54.71 | 48.63 | 49.26 | 33.23 | 66.42 | 50.45 |
| FocalCodec | 0 8192 | 57.30 | 47.76 | 46.17 | 32.52 | 56.83 | 48.12 |
| Facto | 0 8192 | 27.12 | 18.93 | 17.11 | 0 9.42 | 28.56 | 20.23 |
| Facto | 16384 | 30.30 | 21.57 | 19.36 | 10.71 | 31.51 | 22.69 |
| Dims | White | Pink | Brown | Bit Crush | Real | Avg |
|---|---|---|---|---|---|---|
| (SNR 25) | (SNR 22) | (SNR 16) | (BitDepth 10) | (SNR 16) | ||
| WavLM → PCA → -means | ||||||
| 0 12 | 40.26 | 33.06 | 27.27 | 15.86 | 39.00 | 31.09 |
| 0 64 | 33.10 | 24.98 | 23.89 | 13.27 | 33.21 | 25.69 |
| 256 | 33.42 | 24.15 | 23.34 | 12.81 | 32.72 | 25.29 |
| WavLM → Factorization → -means | ||||||
| Parameter | Value |
|---|---|
| Steps | 100 000 |
| Learning Rate | 1.0e-3 |
| Minimum Learning Rate | |
| LR Scheduler | Warmup-Stable-Decay |
| Warmup Steps | 5000 (linear) |
| Decay Steps | 16000 (linear) |
| Parameter | 12.5 Hz | 25 Hz |
|---|---|---|
| max new tokens | 375 | 750 |
| min new tokens | 12 | 25 |
| temperature | 0.5 | 0.5 |
| top- | 40 | 40 |
| top- | 0.9 | 0.9 |
| repetition penalty | 1.1 | 1.1 |
| Method | Codebook | WER | SIM | Spoof -EER | MOS |
|---|---|---|---|---|---|
| 12.5 Hz | |||||
| FocalCodec | 0 8192 | 3.4 | 0.84 | 15.9 | 4.29 |
| Kanade | 12800 | 1.8 | 0.94 | 42.3 | 4.38 |
| Facto | 0 8192 | 1.6 | 0.95 | 49.1 | 4.41 |
| Facto | 16384 | 1.1 | 0.95 | 49.9 | 4.41 |
| 25 Hz | |||||
| Method | Codebook | WER@50k | WER@75k | WER@100k |
|---|---|---|---|---|
| 12.5 Hz | ||||
| FocalCodec | 0 8192 | 0 3.7 | 2.9 | 3.4 |
| Kanade | 12800 | 10.5 | 8.5 | 1.8 |
| Facto | 16384 | 0 1.2 | 1.1 | 1.1 |
| 25 Hz | ||||
| FocalCodec | 0 8192 | 0 3.3 | 2.6 | 1.4 |
| Method | Codebook | WER | SIM | Spoof -EER | MOS |
|---|---|---|---|---|---|
| 12.5 Hz | |||||
| Kanade | 12800 | 1.93 | 0.94 | 40.8 | 4.38 |
| Facto | 0 8192 | 4.61 | 0.94 | 43.7 | 4.40 |
| Facto | 16384 | 3.88 | 0.94 | 43.3 | 4.41 |
| Facto (cont.) | – | 0.96 | 0.93 | 35.4 | 4.40 |
| 25 Hz | |||||
| Parameter | Value |
|---|---|
| Learning Rate | 2.0e-4 |
| Initial Learning Rate | 1.0e-7 |
| Final Learning Rate | 2.0e-5 |
| LR Scheduler | Linear warmup, cosine decay |
| Warmup steps | 8% |
| Decay steps | 92% |
| sWUGGY | ||||||
|---|---|---|---|---|---|---|
| Method | Frame Rate | Codebook | All | In Vocab | sBLIMP | tSC |
| 6k hours | ||||||
| FocalCodec | 25 | 0 8192 | 58.0 | 61.2 | 49.9 | 62.3 |
| Kanade | 25 | 12800 | 59.4 | 63.4 | 51.0 | 62.8 |
| Facto | 25 | 0 8192 | 59.7 | 64.0 | 50.8 | 60.1 |
| Facto | 25 | 16384 | 59.0 | 62.6 | 50.3 | 61.7 |
Audio Tokenization
Audio tokenization converts speech, music, and other sounds into compact units for machine learning, generation, and understanding.
See all 44 papers