Ruby-OPD: Enhancing Pronunciation Controllability and Accuracy for Flow-Matching Text-to-Speech via On-Policy Distillation
Organizations: Baidu Inc., Shenzhen, China
Abstract
Achieving both high pronunciation accuracy and fine-grained controllability remains a central challenge in Japanese text-to-speech (TTS), as Japanese is an unsegmented language mixing logographic Kanji and phonographic Kana with extensive polyphony. Traditional cascaded systems convert text into phoneme sequences via a front-end, ensuring pronunciation control but discarding lexical semantics and prosodic cues in surface orthography. Conversely, recent zero-shot TTS models directly consume raw orthographic text to improve naturalness, yet suffer from polyphone errors and lack explicit pronunciation control. To bridge this gap, we propose a two-stage framework for flow-matching TTS. First, we investigate Ruby text representations with orthography perturbation to jointly preserve surface forms and target pronunciations for reliable control. Second, we introduce On-Policy Distillation (OPD) on large-scale text-only corpora to enhance pronunciation accuracy on raw orthographic text without paired speech. On the Joyo-Kanji-Yomi benchmark, our model achieves 97.13% Accuracy, 2.02% Target PER, 0.70% Sentence PER, and 4.21% Sentence CER under raw text while retaining strong controllability.
Figures & tables
| Orthographic text |
| とある科学の超電磁砲 |
| Implicit Ruby |
| とある科学(かがく)の超電磁砲(レールガン) |
| Explicit Ruby |
| とある|科学《かがく》の|超電磁砲《レールガン》 |
| Model | Pronunciation Accuracy | Speech Quality | Voice Cloning | |||||
|---|---|---|---|---|---|---|---|---|
| Accuracy | Target PER | Sentence PER | Sentence CER | UTMOSv2 | DNSMOS | CAM++ | ERes2Net | |
| Qwen3-TTS 1.7B Base [ 6 ] | 72.10% | 23.68% | 4.38% | 6.52% | 3.50 | 3.23 | 0.757 | 0.779 |
| FireRedTTS3 [ 8 ] | 73.29% | 22.35% | 4.04% | 6.12% | 3.36 | 3.26 | 0.833 | 0.832 |
| OmniVoice [ 12 ] | 83.66% | 13.24% | 2.91% | 5.18% | 3.19 | 3.21 | 0.797 | 0.786 |
| FishAudio S2 Pro [ 7 ] | 89.90% | 8.27% | 1.52% | 4.77% | 3.30 | 3.31 | 0.735 | 0.726 |
| Sarashina2.2-TTS [ 24 ] | 90.42% | 7.69% | 3.15% | 6.26% | 3.42 | 3.33 | 0.819 | 0.825 |
| Model | Input | Accuracy | Target PER | Sentence PER |
|---|---|---|---|---|
| Base | Raw text | 94.11% | 4.68% | 0.97% |
| Base | Target Kana | 97.89% | 1.37% | 0.56% |
| Base | Full Kana | 95.40% | 2.89% | 1.06% |
| SFT | Target Impl. | 97.56% | 1.71% | 0.75% |
| SFT | Target Expl. | 98.12% | 1.26% | 0.72% |
| SFT | Full Impl. | 98.05% | 1.33% | 0.53% |
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 291 papers