ProTTS: End-to-End Prominence-aware Natural and Controllable Text-to-Speech via Automatic Word Stress Discovery
Organizations: Speech Processing Lab, LTRC IIIT Hyderabad Hyderabad, India
Abstract
Natural speech conveys meaning not only through words, but also through the prominence on a few words. Yet most text-to-speech (TTS) systems remain prominence-unaware, existing prominence-aware TTS require external stress annotations, while unsupervised stress detectors operate in isolation, disconnected from synthesis. This creates a fundamental gap no existing system can automatically discover where stress should be placed, realize it as natural-sounding prosody, and offer user control over emphasis placement, all within a single model. To circumvent this, we propose ProTTS, a unified framework where stress detection emerges from the TTS model itself and, in turn, guides it to synthesize more natural and controllable speech. We incorporate a clustering head into the variance adaptor of FastSpeech_2 that discovers word stress directly from aggregated prosodic features. The discovered stress then drives emphasis variance predictors that modify pitch, energy, and duration for stressed words. The framework is trained end-to-end: TTS reconstruction and clustering losses jointly shape a stress-discriminative feature space without requiring stress labels at any stage. At inference, the same architecture also supports user-controllable emphasis by accepting a custom stress mask. We evaluate ProTTS on ISLE (non-native), LibriTTS, VCTK (native training), and three evaluation benchmarks (Tatoeba, TinyStress-15k, EmphAssess). Listeners prefer ProTTS over the baseline with a MOS of 3.96 vs. 3.12 and 83.5% preference rate. For stress detection, ProTTS achieves 0.839/0.861 F1 on ISLE (German/Italian L2), outperforming unsupervised (0.70) and supervised (0.82) baselines, and 0.933 F1 in zero-shot transfer to Tatoeba.
Figures & tables
| Method | GER | ITA | |
|---|---|---|---|
| Sup. † | DNN ( Mondal et al., 2024 ) | 0.83 | 0.74 |
| SupraDoRAL ( Mallela et al., 2025 ) | 0.78 | 0.87 | |
| WhiStress ( Yosha et al., 2025 ) | 0.80 | 0.87 | |
| ProTTS (sup.) | 0.901 | 0.932 | |
| Unsup. | K-Means ( Mondal et al., 2024 ) | 0.76 | 0.71 |
| ProTTS (unsup.) | 0.839 | 0.861 |
| Method | Tat. | Tiny. | Emph. | |
|---|---|---|---|---|
| Sup. † | DNN ( Mondal et al., 2024 ) | 0.891 | — | — |
| WhiStress ( Yosha et al., 2025 ) | — | 0.901 | 0.93 | |
| Unsup. | K-Means ( Mondal et al., 2024 ) | 0.71 | — | — |
| ProTTS | unsup. a | 0.933 | 0.911 | 0.923 |
| unsup. b | 0.921 | 0.90 | 0.915 |
| Configuration | GER | ITA |
|---|---|---|
| (a) Training strategy | ||
| ProTTS (joint) | 0.839 | 0.861 |
| ProTTS (casc.) | 0.803 | 0.831 |
| (b) Feature composition (centroid init.) | ||
| Duration + Pitch (DP) | 0.731 | 0.782 |
| Energy + Duration + Pitch (EDP) | 0.664 | 0.675 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Train | Val | Test | Spkrs | Duration |
| ISLE (GER+ITA) | 3,750 | 3,252 | — | 46 | 5.09 h |
| LibriTTS (clean-100) | 3,000 | 469 | — | 264 | 59.23 h |
| VCTK | 3,000 | 509 | — | 20 | 4.35 h |
| Tatoeba | 3,000 | 398 | 993 | diverse | — |
| TinyStress-15k | — | — | 1,000 | synth. | — |
| EmphAssess | — | — | 3,652 | 4 | 2.08 h |
| Dataset | Words | Stressed | Unstressed | % Str. |
|---|---|---|---|---|
| ISLE GER | 27,136 | 17,082 | 10,054 | 62.9 |
| ISLE ITA | 25,225 | 15,848 | 9,377 | 62.8 |
| Tatoeba | 6,720 | 712 | 6,008 | 10.6 |
| TinyStress-15k | 195,000 | 48,752 | 146,248 | 25.0 |
| EmphAssess | 35,371 | 29,216 | 6,155 | 82.6 |
| Symbol | Definition |
|---|---|
| Input phoneme sequence. | |
| Encoder hidden representations corresponding to the phoneme sequence. | |
| -th word spanning phonemes . | |
| Predicted pitch, energy, and duration from the variance adaptor. | |
| Pitch and energy embeddings associated with phoneme . | |
| Aggregated word-level feature representation used for clustering. |
| Attribute | Distribution |
|---|---|
| Total participants | 35 |
| Gender | Balanced (approx. 50/50) |
| Age range | 20–35 |
| Native language | Mix of English and non-English L1 |
| English proficiency | Advanced to native |
| Headphone use | 90% |
| Test | Total | Unique | Repeat | Source |
|---|---|---|---|---|
| MOS + Pref. | 60 | 47 | 13 | 22 Lib. / 25 VCTK |
| ABX Sim. | 50 | 40 | 10 | 21 Lib. / 19 VCTK |
| Stress Pref. | 60 | 46 | 14 | 15 GER / 15 ITA / 10 Tat. / 6 Gen. |
| Total | 170 | 133 | 37 |
| Dataset | Base FS2 | ProTTS |
|---|---|---|
| MOS (mean 95% CI) | ||
| LibriTTS | 3.21 0.18 | 3.89 0.14 |
| VCTK | 3.04 0.16 | 4.02 0.12 |
| Overall | 3.12 0.14 | 3.96 0.11 |
| Naturalness preference (%) | ||
| LibriTTS | 18.2 | 81.8 |
| Dataset | ProTTS | Both | Neither | Base |
|---|---|---|---|---|
| ISLE GER | 76.8 | 13.2 | 5.5 | 4.5 |
| ISLE ITA | 80.1 | 11.8 | 4.8 | 3.3 |
| Tatoeba | 79.4 | 12.1 | 4.6 | 3.9 |
| Generic (ctrl.) | 75.6 | 13.8 | 6.2 | 4.4 |
| Overall | 78.3 | 12.5 | 5.1 | 4.1 |
| Dataset | ProTTS | Base | Both | Neither |
|---|---|---|---|---|
| LibriTTS | 73.2 | 11.8 | 7.5 | 7.5 |
| VCTK | 70.1 | 13.4 | 6.8 | 9.7 |
| Overall | 71.8 | 12.5 | 7.2 | 8.5 |
| Mode | Stress Pref. (%) | ABX (%) |
|---|---|---|
| Automatic | 79.1 | 72.4 |
| Controllable | 75.6 | 69.8 |
| Sentence | Stress | Implied meaning |
|---|---|---|
| I didn’t say he stole the money | stole | He did something else with it |
| I didn’t say he stole the money | didn’t | Denying the claim was made |
| She always arrives early | always, early | Emphasizing habitual punctuality |
| The photographer captured a magnificent sunset | photographer, magnificent | Highlighting the agent and quality |
| We should never have trusted them | never, trusted | Expressing strong regret |
| Method | Sup. | Det. | Syn. | Joint | Native | Non-nat. | Datasets |
| Stress detection only | |||||||
| K-Means on FS2 emb. ( Mondal et al., 2024 ) | U | ✓ | ✓ | ✓ | ISLE, Tatoeba | ||
| DNN on FS2 emb. ( Mondal et al., 2024 ) | S | ✓ | ✓ | ✓ | ISLE, Tatoeba | ||
| SupraDoRAL ( Mallela et al., 2025 ) | S | ✓ | ✓ | ISLE | |||
| WhiStress ( Yosha et al., 2025 ) | S | ✓ | ✓ | ✓ | TinyStress, EmphAssess, Expresso, ISLE | ||
| Emphasis-controllable synthesis | |||||||
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 294 papers