M5-TTS: A Mini-Scale Multilingual and Multi-Dialect Text-to-Speech Language Model with Multi-Token Prediction
Organizations: Shenzhen Pimei Technology Co., Ltd. · Moonstep AI · WeNet Open Source Community
Abstract
We present M5-TTS, a compact text-to-speech system supporting multilingual and Chinese multidialect synthesis, zero-shot voice cloning, speaker-specific multilingual generation, and CPU streaming. Its 50M-parameter language model, excluding the speech codec, combines a temporal backbone with a local multi-token prediction module over 16 codebooks. GRPO post-training combines global speaker-similarity and pause feedback with a token-level CER process advantage. Character-level error comparisons provide local feedback, with negative credit propagated backward within a clause and projected onto audio-token positions. M5-TTS achieves 1.37% CER on Seed-ZH and 2.91% WER on Seed-EN, corresponding to relative error reductions of 62.2% and 35.3% over MOSS-TTS-Nano with half the language-model parameters. It also achieves lower content error rates in five of six evaluated MiniMax languages, while difficult Chinese inputs remain a limitation. On four Intel Xeon Gold 6530 cores, it achieves a real-time factor of 0.653 and a warm first-chunk latency of 225ms. A 10-listener evaluation gives M5-TTS higher speaker-similarity and intelligibility ratings than MOSS-TTS-Nano, with a lower naturalness rating. We will release training, post-training, and inference code, together with voice-cloning and speaker-specific checkpoints, to support reproducible research on compact speech generation for on-device voice interaction.
Figures & tables
| Metric | Original order A: MOSS; B: M5 | Swapped order A: M5; B: MOSS |
| Evaluated pairs | 311 | 311 |
| M5-TTS wins | 264 (84.89%) | 269 (86.50%) |
| MOSS-TTS-Nano wins | 47 (15.11%) | 42 (13.50%) |
| Ties | 0 | 0 |
| Mean reward: M5-TTS | 0.2586 | 0.3231 |
| Mean reward: MOSS-TTS-Nano |
| Original preference | Swapped preference | Pairs | Percentage |
| M5-TTS | M5-TTS | 255 | 81.99% |
| MOSS-TTS-Nano | MOSS-TTS-Nano | 33 | 10.61% |
| MOSS-TTS-Nano | M5-TTS | 14 | 4.50% |
| M5-TTS | MOSS-TTS-Nano | 9 | 2.89% |
| Same preferred system (subtotal) | 288 | 92.60% | |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Pre-training | CPT |
| backbone_flavor | llama-30M | llama-30M |
| decoder_flavor | llama-20M | llama-20M |
| tokenizer | llama3 | llama3 |
| audio_num_codebooks | 16 | 16 |
| learning_rate | ||
| min_lr |
| Hyperparameter | Value |
| Rewards and optimization | |
| reward_weight.cer_reward | 0.0 |
| reward_weight.sim_reward | 1.0 |
| reward_weight.pause_reward | 0.5 |
| token_cer.enabled | true |
| token_cer.mode | char_exp |
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 291 papers