Assessing the Ability of Neural TTS Systems to Model Consonant-Induced F0 Perturbation
Authors: Tianle Yang, Chengzhe Sun, Phil Rose, Cassandra L. Jacobs, Siwei Lyu
Organizations: University at Buffalo, Department of Linguistics, Buffalo, 14260, NY, United States · University at Buffalo, Department of Computer Science and Engineering, Buffalo, 14260, NY, United States · Australian National University, Emeritus Faculty, Canberra, 0200, ACT, Australia
This study proposes a segmental-level prosodic probing framework to evaluate neural TTS models' ability to reproduce consonant-induced f0 perturbation, a fine-grained segmental-prosodic effect that reflects local articulatory mechanisms. We compare synthetic and natural speech realizations for thousands of words, stratified by lexical frequency, using Tacotron 2 and FastSpeech 2 trained on the same speech corpus (LJ Speech). These controlled analyses are then complemented by a large-scale evaluation spanning multiple advanced TTS systems. Results show accurate reproduction for high-frequency words but poor generalization to low-frequency items, suggesting that the examined TTS architectures rely more on lexical-level memorization than on abstract segmental-prosodic encoding. This finding highlights a limitation in such TTS systems' ability to generalize prosodic detail beyond seen data. The proposed probe offers a linguistically informed diagnostic framework that may inform future TTS evaluation methods, and has implications for interpretability and authenticity assessment in synthetic speech.
Table 1: IPA phones used in the analysis, grouped by segment type. This phone inventory is based on the MFA English (US) dictionary ( McAuliffe and Sonderegger, 2022 ) .
Speech source
Frequency
[+voi] obstruents
[-voi] obstruents
sonorant
LJ speech
High
1,000
1,000
1,000
FastSpeech 2
High
1,000
1,000
1,000
Tacotron 2
High
1,000
1,000
1,000
LJ speech
Low
1,000
1,000
1,000
FastSpeech 2
Low
1,000
1,000
1,000
Tacotron 2
Low
1,000
1,000
1,000
Table 2: Token count per onset type across speech source and frequency bands.
Figure 1: GAMM smooths of f0 trajectories for onset types (leftmost column) and pairwise difference smooths (right three columns) for all high-frequency speech sources. Shaded areas indicate significant intervals.
Figure 2: GAMM smooths of f0 trajectories for onset types (leftmost column) and pairwise difference smooths (right three columns) for all low-frequency speech sources. Shaded areas indicate significant intervals.
Figure 3: GAMM smooths and difference smooths for z -scored f0 trajectories (seen vs unseen).
Figure 4: GAMM smooths and difference smooths for z -scored f0 trajectories (high frequency group).
Figure 5: GAMM smooths and difference smooths for z -scored f0 trajectories (low frequency group).
Neural TTS systems can sound natural across languages, but naturalness does not guarantee the preservation of sound contrasts that distinguish words from their grammatical forms. Standard metrics like MOS do not test for this. We propose a classifier-based framework that audits TTS output against language-specific phonological patterns using human speech as a benchmark. Testing Assamese advanced tongue root (ATR) vowel harmony with Meta's MMS TTS, we show that a classifier trained on human speech transfers to synthesized speech with minimal loss. The faithfulness audit reveals that [+ATR] mid vowels are realized as [-ATR] in 1/3 tokens despite an underlying [+ATR] specification, a bias absent in human speech. At the word level, predicted ATR labels classify harmony more accurately than transcription labels, indicating a gap between intended and produced phonology. The framework offers task-specific diagnostics and generalizes to other phonological contrasts with measurable acoustic cues.
Sneha Ray Barman, Neeraj Kumar Sharma, Shakuntala Mahanta
Centre for Linguistic Science & Technology, IIT Guwahati · Mehta Family School for Data Science & Artificial Intelligence, IIT Guwahati · Department of Humanities & Social Sciences, IIT Guwahati
Zero-shot text-to-speech (TTS) typically conditions on a short speech prompt, which conveys timbre but provides only limited evidence of a speaker's prosodic patterns. We formulate Long-Prompt Zero-Shot TTS, a task that uses long speech prompts at inference time to reproduce speaker-specific prosody without target-speaker fine-tuning. We introduce LP-TTS-Bench, a bilingual benchmark of 37 speakers with 63.95 prompt hours and 2,728 held-out test utterances. Beyond intelligibility and timbre similarity, LP-TTS-Bench evaluates timing, prominence, intonation, and model-based prosody metrics, enabling multidimensional evaluation of prosody reproduction. As a reference system, we develop LPTTS, whose hierarchical Long-Prompt Encoder compresses long speech prompts into a fixed number of continuous prosody vectors. Experiments show improvements in several prosodic dimensions over a short-prompt baseline, with overall gains as prompt duration increases up to approximately 10 minutes and saturation at longer durations.
Dekun Chen, Renjie Xu, Yingda Shen +2
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.
Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.
Masato Takagi, Masaya Kawamura, Reo Shimizu +1
Nagoya Institute of Technology, Japan · LY Corporation, Japan