We propose 21Codec, a streaming codec model with reduced recoverable speaker information from the token stream and decoder-side speaker control. Existing codec models infer speaker identity from token stream. Making them streamable is not sufficiently robust to speaker drift, because each chunk re-infers the identity from the local chunk context alone. Our adversarial speaker classifier and NANSY-style formant-shift data augmentation pushes the model to rely on the provided speaker embedding while reducing the recoverable speaker information from the token stream. We show that 21Codec can control speaker identity through the conditioning vector, and that the speaker identity is maintained across chunks during streaming. With the ability to control speaker identity even in mid-stream, 21Codec is a promising candidate for streaming TTS systems for better controllability of speaker identity.
Figures & tables
Figure 1: 21Codec Architecture. The encoder is non-causal, the decoder chunk-causal.
Corpus
Lang
Hours
Spk.
Weight
In-house (audiobooks)
en
862.08
44
.30
LibriTTS-R
en
449.68
2277
.18
Common Voice 23.0
54
1865.16
36382
.15
MLS
7
5870.32
518
.15
In-house (acted dialogue)
2
532.37
2135
.15
HUI
de
322.88
121
.05
Table 1: Training mixture. Weights are per-batch sampling probabilities, not proportions of the available hours.
Table 3
Chunk
Spk. sim.
WER/CER
DNSMOS
NISQA
↑
↓
↑
↑
1 † ( 0 20 ms)
.854
.107 / .062
3.28
4.58
4 † ( 0 80 ms)
.872
.099 / .055
3.32
4.64
8 † (160 ms)
.875
.095 / .052
3.34
4.68
16 † (320 ms)
.878
.093 / .052
3.34
4.69
25 0 (500 ms)
.881
.087 / .051
3.35
4.68
Table 4: Decoder chunk size vs. quality for 21Codec. † marks chunk sizes seen during training
Figure 2: Speaker drift under un-cached chunked decoding. Markers are means over 2 s windows and files, bars the within-utterance standard deviation. The gray band is natural audio through the same windowing ( 0.735±0.063 ), the zero-drift floor.
System
Sim. src
DNSMOS
NISQA
WER / CER
↑
↑
↑
↓
Ground Truth
–
3.390
4.754
–
NeuCodec
.889
3.309
4.735
.027 / .013
WavTokenizer
.801
3.408
4.698
.037 / .019
21Codec
.903
3.415
4.618
.095 / .052
21Codec-U
.818
3.378
4.569
.047 / .025
Table 5: Reconstruction on EN-80, n=160 . All systems run at 0.80 kbps except WavTokenizer ( 0.90 ).
Text-to-speech converts written language into spoken audio, improving intelligibility, naturalness, voice control, multilingual coverage, and efficiency.