We present M5-TTS, a compact text-to-speech system supporting multilingual and Chinese multidialect synthesis, zero-shot voice cloning, speaker-specific multilingual generation, and CPU streaming. Its 50M-parameter language model, excluding the speech codec, combines a temporal backbone with a local multi-token prediction module over 16 codebooks. GRPO post-training combines global speaker-similarity and pause feedback with a token-level CER process advantage. Character-level error comparisons provide local feedback, with negative credit propagated backward within a clause and projected onto audio-token positions. M5-TTS achieves 1.37% CER on Seed-ZH and 2.91% WER on Seed-EN, corresponding to relative error reductions of 62.2% and 35.3% over MOSS-TTS-Nano with half the language-model parameters. It also achieves lower content error rates in five of six evaluated MiniMax languages, while difficult Chinese inputs remain a limitation. On four Intel Xeon Gold 6530 cores, it achieves a real-time factor of 0.653 and a warm first-chunk latency of 225ms. A 10-listener evaluation gives M5-TTS higher speaker-similarity and intelligibility ratings than MOSS-TTS-Nano, with a lower naturalness rating. We will release training, post-training, and inference code, together with voice-cloning and speaker-specific checkpoints, to support reproducible research on compact speech generation for on-device voice interaction.
Figures & tables
Figure 1: Qwen-TTS-Tokenizer-12Hz Hu et al. (2026) adopted by M5-TTS: a semantic VQ branch and an acoustic RVQ branch provide 16 codec tokens per frame for causal waveform reconstruction.
Figure 2: M5-TTS architecture. The temporal backbone predicts codebook 0; the local MTP module generates codebooks 1–15 within each frame, which are then decoded into speech.
Figure 3: GRPO with global feedback and CER-guided local process feedback. Utterance-level speaker-similarity and pause rewards are normalized separately within each candidate group, then weighted and summed to obtain a shared global advantage. ASR edit alignment yields per-character group-relative advantages; negative advantages propagate backward within a clause before temporal projection onto codec frames. The local process advantage is added to the global advantage, and all 16 codec tokens in a frame share the resulting advantage for the policy update.
Metric
Original order A: MOSS; B: M5
Swapped order A: M5; B: MOSS
Evaluated pairs
311
311
M5-TTS wins
264 (84.89%)
269 (86.50%)
MOSS-TTS-Nano wins
47 (15.11%)
42 (13.50%)
Ties
0
0
Mean reward: M5-TTS
0.2586
0.3231
Mean reward: MOSS-TTS-Nano
−0.6542
−0.6234
Table 5: VoiceJudge-based naturalness preferences on the same 311 audio pairs under two presentation orders. Reward values are scalar outputs of VoiceJudge.
Original preference
Swapped preference
Pairs
Percentage
M5-TTS
M5-TTS
255
81.99%
MOSS-TTS-Nano
MOSS-TTS-Nano
33
10.61%
MOSS-TTS-Nano
M5-TTS
14
4.50%
M5-TTS
MOSS-TTS-Nano
9
2.89%
Same preferred system (subtotal)
288
92.60%
Table 6: Joint VoiceJudge preferences after mapping A/B choices to system identities. The first four rows partition the 311 audio pairs; the final row summarizes agreement across orders.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Pre-training
CPT
backbone_flavor
llama-30M
llama-30M
decoder_flavor
llama-20M
llama-20M
tokenizer
llama3
llama3
audio_num_codebooks
16
16
learning_rate
1×10−4
1×10−5
min_lr
1×10−5
1×10−6
Appendix
Table 7: Hyperparameter configurations for pre-training and CPT. Field names follow the training configuration.
Hyperparameter
Value
Rewards and optimization
reward_weight.cer_reward
0.0
reward_weight.sim_reward
1.0
reward_weight.pause_reward
0.5
token_cer.enabled
true
token_cer.mode
char_exp
Appendix
Table 8: Hyperparameters for GRPO post-training. Field names follow the training configuration.
While recent advances in text-to-speech (TTS) synthesis have substantially improved speech quality and modeling flexibility, the development of lightweight and computationally efficient systems remains comparatively underexplored. We present LeanTTS, a bilingual complete synthesis stack that supports two complementary regimes: reference-prompt zero-shot cloning for unseen speakers and token-conditioned customization for recurring voices. LeanTTS combines a compact Mandarin polyphone front end, a structurally compressed flow-matching acoustic model, and a lightweight Vocos decoder with four-step MeanFlow generation. The resulting system contains 18.9M learned parameters, 86.12% fewer than the 136.2M-parameter ZipVoice-Vocos pipeline. On Seed-TTS-Eval, its zero-shot mode obtains 2.02 CER/62.3 SIM on Mandarin and 2.32 WER/48.3 SIM on English, exposing the quality-size trade-off under aggressive compression. With token-conditioned customized voices, the same system reaches 1.57 CER/77.2 SIM on Mandarin and 1.93 WER/76.6 SIM on English, with UTMOS scores of 3.03 and 4.33, respectively. It further achieves mean/P90 CPU RTFs of 0.049/0.061, establishing a sub-20M operating point for bilingual personalization and real-time CPU deployment. Audio demos are available at https://github.com/notlu/leantts.
Yuheng Lu, Qiulin Li, Xi Wang +5
Tianjin University, Tianjin, China · Didichuxing Co. Ltd., Beijing, China · Tsinghua University, Beijing, China
We introduce Voxtral TTS, an expressive multilingual text-to-speech model that generates natural speech from as little as 3 seconds of reference audio. Voxtral TTS adopts a hybrid architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens. These tokens are encoded and decoded with Voxtral Codec, a speech tokenizer trained from scratch with a hybrid VQ-FSQ quantization scheme. In human evaluations conducted by native speakers, Voxtral TTS is preferred for multilingual voice cloning due to its naturalness and expressivity, achieving a 68.4% win rate over ElevenLabs Flash v2.5. We release the model weights under a CC BY-NC license.
Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.
Bowen Li, Shaotong Guo, Zhen Wang +11
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen