Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
Figures & tables
Figure 1: (a) Factorization : To separate speaker and content information, we jointly factorize self-supervised features across a fixed set of K training speakers. We solve an optimization problem to decompose X1,…,XK into the product of a shared content representation C and speaker-specific transformations Sk . (b) Content estimation and clustering : To project unseen speech into the content space, we fit a least-squares estimator to predict C directly from the self-supervised features. Then, we use k -means to cluster the content space. (c) Decoding : Finally, we train a decoder to reconstruct source speech from the content tokens and a learned speaker embedding.
Figure 2: Analysis of the content space. UMAP plots of the content space colored by (a) sound type, and (b) pitch. The content space is organized globally by phonetic information and locally by prosodic structure. (c) Factorizations trained on disjoint speaker subsets converge at a rate of O(1/K) , validating Theorem 1 .
Figure 3: (a) Relative reconstruction error Ek[∥C−C^k∥F]/∥C∥F for the different content estimators discussed in Section 2.2 . (b) Relative reconstruction error versus the number of speakers used to fit the least-squares estimator. The error decreases at a rate of O(1/K) as predicted by Corollary 1 . The dashed line denotes the predicted asymptotic error as K→∞ . (c) Token stability under small noise perturbations. Higher Unit Edit Distance (UED) indicates greater sensitivity to noise.
Method
Params
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
12.5 Hz
FocalCodec
145
0 8192
5.9
0.90
31.1
4.35
Kanade
207
12800
3.5
0.93
41.2
4.35
Facto
131
0 8192
5.6
0.93
42.3
4.37
Facto
131
16384
5.0
0.93
42.6
4.37
Facto (cont.)
131
–
2.4
0.94
44.9
4.37
Table 1: Speech resynthesis results on the LibriSpeech test-clean set.
Method
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
Nat. ↑
Ref.Sim. ↑
12.5 Hz
FocalCodec
0 8192
3.6
0.81
11.7
4.15
62.1±2.5
16.8±2.0
Kanade
12800
4.5
0.92
36.4
4.26
65.1±2.3
67.4±2.4
Facto
0 8192
2.7
0.94
46.4
4.36
65.3±2.2
78.7±2.3
Facto
16384
2.3
0.94
45.8
4.36
65.5±2.2
78.9±2.2
25 Hz
Table 2: Text-to-speech results on the Seed-TTS test-en set.
Method
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
12.5 Hz
FocalCodec
0 8192
15.3
0.74
19.4
4.02
Kanade
12800
18.9
0.86
43.2
4.11
Facto
0 8192
0 9.4
0.90
53.1
4.31
Facto
16384
0 9.8
0.89
51.1
4.28
25 Hz
Table 3: Text-to-speech results on expressive and spontaneous speech from the Expresso test set.
Method
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
25 Hz
Kanade
12800
0.97
0.93
34.9
4.37
Facto
0 8192
1.83
0.94
42.0
4.40
Facto
16384
1.66
0.94
41.2
4.40
Facto (cont.)
–
0.85
0.93
35.4
4.40
CosyVoice2
0 6561
1.30
0.86
24.5
4.33
Table 4: Voice conversion results on the LibriSpeech test-clean set.
sWUGGY ↑
Method
Frame Rate
Codebook
All
In Vocab
sBLIMP ↑
tSC ↑
6k hours
FocalCodec
12.5
0 8192
60.0
64.1
49.8
61.5
Kanade
12.5
12800
61.6
66.3
50.4
63.4
Facto
12.5
0 8192
61.0
65.5
50.9
65.2
Facto
12.5
16384
60.8
65.0
51.2
64.5
Table 5: Spoken language modeling results on sWUGGY, sBLIMP, and Topic StoryCloze.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: To separate speaker and content information, we jointly factorize self-supervised features across K speakers. The factorization method consists of two steps. (a) Alignment : We randomly choose frames from the combined pool of speakers to construct an anchor set. For every anchor frame, we find a matching feature vector for each speaker using nearest neighbors. (b) Decomposition : we solve an optimization problem to decompose X1,…,XK into the product of a shared content representation C and speaker-specific transformations Sk .
Figure 5: (a) Content estimation : to project new utterances into the content space, we fit a least-squares estimator to predict C from the aligned self-supervised features Xk . (b) Clustering : we stack neighboring frames into fixed-length patches. We apply PCA to decorrelate the patches and cluster the resulting space with k -means. (c) Decoding : then we train the decoder to reconstruct source speech from the content tokens and a learned speaker embedding.
Metric
X
C
S
Semantic content
CER ( %)
10.87
8.96
–
Prosody – frame level
ρ normalized log- F0
0.824
0.630
–
ρ normalized log-energy
0.987
0.937
–
Speaker timbre attributes – utterance level
Appendix
Table 6: Information captured by the original WavLM features ( X ), content representation ( C ), and speaker embedding ( S ). We report CER for semantic content, frame-level Pearson correlation ρ for prosodic features, and utterance-level coefficient of determination R2 for timbre.
Figure 6: Effect of the content space rank R on information retained by the content representation. (a) Character error rate (CER) of a CTC probe trained on content tokens, measuring semantic content retention. (b) Frame-level correlation between predicted and ground-truth prosodic features, measuring retention of pitch and energy variations.
Parameter
Pretraining
Fine-tuning
Steps
2M
500K
Learning Rate
2e−4
1e−4
LR Scheduler
Inverse Power
Inverse Power
Warmup (geometric)
0.999
0.999
Decay ( γ )
50000
50000
Decay (power)
0.5
0.5
Appendix
Table 7: Decoder hyperparameters.
Method
Codebook
White
Pink
Brown
Bit Crush
Real
Avg
(SNR 25)
(SNR 22)
(SNR 16)
(BitDepth 10)
(SNR 16)
12.5Hz
Kanade
12800
54.71
48.63
49.26
33.23
66.42
50.45
FocalCodec
0 8192
57.30
47.76
46.17
32.52
56.83
48.12
Facto
0 8192
27.12
18.93
17.11
0 9.42
28.56
20.23
Facto
16384
30.30
21.57
19.36
10.71
31.51
22.69
Appendix
Table 8: Robustness to noise. Unit Edit Distance (UED ↓ ) under different noise profiles.
Dims
White
Pink
Brown
Bit Crush
Real
Avg
(SNR 25)
(SNR 22)
(SNR 16)
(BitDepth 10)
(SNR 16)
WavLM → PCA → k -means
0 12
40.26
33.06
27.27
15.86
39.00
31.09
0 64
33.10
24.98
23.89
13.27
33.21
25.69
256
33.42
24.15
23.34
12.81
32.72
25.29
WavLM → Factorization → k -means
Appendix
Table 9: Effect of factorization on noise robustness (reported as Unit Edit Distance (UED ↓ )).
Parameter
Value
Steps
100 000
Learning Rate
1.0e-3
Minimum Learning Rate
LR Scheduler
Warmup-Stable-Decay
Warmup Steps
5000 (linear)
Decay Steps
16000 (linear)
Appendix
Table 10: Training hyperparameters.
Parameter
12.5 Hz
25 Hz
max new tokens
375
750
min new tokens
12
25
temperature
0.5
0.5
top- k
40
40
top- p
0.9
0.9
repetition penalty
1.1
1.1
Appendix
Table 11: Generation hyperparameters
Method
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
12.5 Hz
FocalCodec
0 8192
3.4
0.84
15.9
4.29
Kanade
12800
1.8
0.94
42.3
4.38
Facto
0 8192
1.6
0.95
49.1
4.41
Facto
16384
1.1
0.95
49.9
4.41
25 Hz
Appendix
Table 12: Text-to-speech results on LibriSpeech test-clean .
Method
Codebook
WER@50k
WER@75k
WER@100k
12.5 Hz
FocalCodec
0 8192
0 3.7
2.9
3.4
Kanade
12800
10.5
8.5
1.8
Facto
16384
0 1.2
1.1
1.1
25 Hz
FocalCodec
0 8192
0 3.3
2.6
1.4
Appendix
Table 13: Intermediate evaluations during training on LibriSpeech test-clean .
Method
Codebook
WER ↓
SIM ↑
Spoof -EER ↑
MOS ↑
12.5 Hz
Kanade
12800
1.93
0.94
40.8
4.38
Facto
0 8192
4.61
0.94
43.7
4.40
Facto
16384
3.88
0.94
43.3
4.41
Facto (cont.)
–
0.96
0.93
35.4
4.40
25 Hz
Appendix
Table 14: Voice conversion results on the LibriSpeech test-clean set.
Parameter
Value
Learning Rate
2.0e-4
Initial Learning Rate
1.0e-7
Final Learning Rate
2.0e-5
LR Scheduler
Linear warmup, cosine decay
Warmup steps
8%
Decay steps
92%
Appendix
Table 15: SLM Training Hyperparameters
sWUGGY ↑
Method
Frame Rate
Codebook
All
In Vocab
sBLIMP ↑
tSC ↑
6k hours
FocalCodec
25
0 8192
58.0
61.2
49.9
62.3
Kanade
25
12800
59.4
63.4
51.0
62.8
Facto
25
0 8192
59.7
64.0
50.8
60.1
Facto
25
16384
59.0
62.6
50.3
61.7
Appendix
Table 16: Spoken Language Modeling using 25 Hz tokenizers.
Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive evaluations show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks, effectively serving as a unified audio interface. We publicly release all our code, including training and inference scripts, together with the model checkpoints at https://github.com/Tencent/Universal_Audio_Tokenizer.
Yuhan Song, Linhao Zhang, Aiwei Liu +6
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.
In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.
Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets
Dinghao Zhou, Xingchen Song, Di Wu +3
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China