We propose 21Codec, a streaming codec model with reduced recoverable speaker information from the token stream and decoder-side speaker control. Existing codec models infer speaker identity from token stream. Making them streamable is not sufficiently robust to speaker drift, because each chunk re-infers the identity from the local chunk context alone. Our adversarial speaker classifier and NANSY-style formant-shift data augmentation pushes the model to rely on the provided speaker embedding while reducing the recoverable speaker information from the token stream. We show that 21Codec can control speaker identity through the conditioning vector, and that the speaker identity is maintained across chunks during streaming. With the ability to control speaker identity even in mid-stream, 21Codec is a promising candidate for streaming TTS systems for better controllability of speaker identity.
Figures & tables
Figure 1: 21Codec Architecture. The encoder is non-causal, the decoder chunk-causal.
Corpus
Lang
Hours
Spk.
Weight
In-house (audiobooks)
en
862.08
44
.30
LibriTTS-R
en
449.68
2277
.18
Common Voice 23.0
54
1865.16
36382
.15
MLS
7
5870.32
518
.15
In-house (acted dialogue)
2
532.37
2135
.15
HUI
de
322.88
121
.05
Table 1: Training mixture. Weights are per-batch sampling probabilities, not proportions of the available hours.
Table 3
Chunk
Spk. sim.
WER/CER
DNSMOS
NISQA
↑
↓
↑
↑
1 † ( 0 20 ms)
.854
.107 / .062
3.28
4.58
4 † ( 0 80 ms)
.872
.099 / .055
3.32
4.64
8 † (160 ms)
.875
.095 / .052
3.34
4.68
16 † (320 ms)
.878
.093 / .052
3.34
4.69
25 0 (500 ms)
.881
.087 / .051
3.35
4.68
Table 4: Decoder chunk size vs. quality for 21Codec. † marks chunk sizes seen during training
Figure 2: Speaker drift under un-cached chunked decoding. Markers are means over 2 s windows and files, bars the within-utterance standard deviation. The gray band is natural audio through the same windowing ( 0.735±0.063 ), the zero-drift floor.
System
Sim. src
DNSMOS
NISQA
WER / CER
↑
↑
↑
↓
Ground Truth
–
3.390
4.754
–
NeuCodec
.889
3.309
4.735
.027 / .013
WavTokenizer
.801
3.408
4.698
.037 / .019
21Codec
.903
3.415
4.618
.095 / .052
21Codec-U
.818
3.378
4.569
.047 / .025
Table 5: Reconstruction on EN-80, n=160 . All systems run at 0.80 kbps except WavTokenizer ( 0.90 ).
Text-to-speech converts written language into spoken audio, improving intelligibility, naturalness, voice control, multilingual coverage, and efficiency.
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality usually comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches about 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. In a blinded study, 40 listeners preferred TontaubeV1's prosody over Fish Audio S2 Pro in 70.25% of 400 comparisons (95% CI: 64.2-75.7%). On an LLM-as-a-judge benchmark, it matches ElevenLabs Flash v2.5 and outperforms the April 2026 Gradium API and Cartesia Sonic 3 on prosody. The weights are released on Hugging Face under the Tontaube Community Model License 1.0.
Full-stream text-to-speech (TTS) for interactive systems must start speaking with minimal delay while remaining controllable as text arrives incrementally. We present VoXtream2, a zero-shot full-stream TTS model with dynamic speaking-rate control that can be updated mid-utterance on the fly. VoXtream2 combines a distribution matching mechanism over duration states with classifier-free guidance across conditioning signals to improve controllability and synthesis quality. Prompt-text masking enables textless audio prompting, removing the need for prompt transcription. Across standard zero-shot benchmarks and a dedicated speaking-rate test set, VoXtream2 achieves competitive objective and subjective results against public baselines despite a smaller model and less training data. In full-stream mode, it runs 4 times faster than real time with 74 ms first-packet latency on a consumer GPU.
Nikita Torgashov, Gustav Eje Henter, Gabriel Skantze
Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden
High linear decodability does not entail causal controllability in continuous-latent text-to-speech (TTS). In VoxCPM2, position-matched probes and equal-norm interventions across 30 speakers and 60 directed dyads expose this separation: reference-audio states at the final residual acoustic language model layer (RALM L7) retain 0.970 probe accuracy, yet aligned edits yield 0/60 target-dominant dyads. Edits at early reference-audio positions produce the opposite pattern. At L0, pair-aligned updates increase target similarity by 0.326 and achieve target dominance in all 60 dyads, against at most 1/60 under any equal-norm direction or position control. Independent speaker encoders, retrieval against held-out speaker prototypes, and double-blind listening confirm that the shifted voice is heard as the intended target. Cross-architecture interventions in dots.tts distinguish backbone leverage from direct acoustic control: backbone edits displace the source without target attraction, while x-vector ablation reduces source similarity from 0.849 to 0.006. These findings identify two requirements for training-free speaker control: geometric specificity and a computational route that carries the intended speaker condition into acoustic synthesis.