Assessing the Ability of Neural TTS Systems to Model Consonant-Induced F0 Perturbation

Tianle Yang; Chengzhe Sun; Phil Rose; Cassandra L. Jacobs; Siwei Lyu

cs.CLMar 22, 2026

Assessing the Ability of Neural TTS Systems to Model Consonant-Induced F0 Perturbation

Authors: Tianle Yang, Chengzhe Sun, Phil Rose, Cassandra L. Jacobs, Siwei Lyu

Organizations: University at Buffalo, Department of Linguistics, Buffalo, 14260, NY, United States · University at Buffalo, Department of Computer Science and Engineering, Buffalo, 14260, NY, United States · Australian National University, Emeritus Faculty, Canberra, 0200, ACT, Australia

Abstract

This study proposes a segmental-level prosodic probing framework to evaluate neural TTS models' ability to reproduce consonant-induced f0 perturbation, a fine-grained segmental-prosodic effect that reflects local articulatory mechanisms. We compare synthetic and natural speech realizations for thousands of words, stratified by lexical frequency, using Tacotron 2 and FastSpeech 2 trained on the same speech corpus (LJ Speech). These controlled analyses are then complemented by a large-scale evaluation spanning multiple advanced TTS systems. Results show accurate reproduction for high-frequency words but poor generalization to low-frequency items, suggesting that the examined TTS architectures rely more on lexical-level memorization than on abstract segmental-prosodic encoding. This finding highlights a limitation in such TTS systems' ability to generalize prosodic detail beyond seen data. The proposed probe offers a linguistically informed diagnostic framework that may inform future TTS evaluation methods, and has implications for interpretability and authenticity assessment in synthetic speech.

Figures & tables

Segment Type	Selected Phones
[+voi] obstruent	[ \textipa b], [ \textipa d], [ \textipa g], [ \textipa dZ], [ \textipa Z], [ \textipa v], [ \textipa D], [ \textipa z]
$[-$ voi $]$ obstruent	[ \textipa p], [ \textipa t], [ \textipa k], [ \textipa tS], [ \textipa f], [ \textipa T], [ \textipa s], [ \textipa S], [ \textipa h], [ \textipa p \super h], [ \textipa t \super h], [ \textipa k \super h], [ \textipa c], [ \textipa ç]
sonorant	[ \textipa m], [ \textipa n], [ \textipa N], [ \textltailn ], [ \textipa l], [ \textipa j], [ \textipa w], [ \textipa m \super j], [ \s n]
vowel	[ \textipa i], [ \textipa i:], [ \textipa I], [ \textepsilon ], [ \textipa æ], [ \textipa A], [ \textipa A:], [ \textipa @], [ \textipa 6], [ \textipa u], [ \textipa U], [ \textipa O]

Table 1: IPA phones used in the analysis, grouped by segment type. This phone inventory is based on the MFA English (US) dictionary ( McAuliffe and Sonderegger, 2022 ) .

Speech source	Frequency	[+voi] obstruents	[-voi] obstruents	sonorant
LJ speech	High	1,000	1,000	1,000
FastSpeech 2	High	1,000	1,000	1,000
Tacotron 2	High	1,000	1,000	1,000
LJ speech	Low	1,000	1,000	1,000
FastSpeech 2	Low	1,000	1,000	1,000
Tacotron 2	Low	1,000	1,000	1,000

Table 2: Token count per onset type across speech source and frequency bands.

Figure 1: GAMM smooths of

f_{0}

trajectories for onset types (leftmost column) and pairwise difference smooths (right three columns) for all high-frequency speech sources. Shaded areas indicate significant intervals.

Figure 2: GAMM smooths of

f_{0}

trajectories for onset types (leftmost column) and pairwise difference smooths (right three columns) for all low-frequency speech sources. Shaded areas indicate significant intervals.

Figure 3: GAMM smooths and difference smooths for

z

-scored

f_{0}

trajectories (seen vs unseen).

Figure 4: GAMM smooths and difference smooths for

z

-scored

f_{0}

trajectories (high frequency group).

Figure 5: GAMM smooths and difference smooths for

z

-scored

f_{0}

trajectories (low frequency group).

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Towards a Phonology-Informed Evaluation of Multilingual TTS
Jul 2, 2026Sneha Ray Barman, Neeraj Kumar Sharma, Shakuntala MahantaTTS Synthesis Speech Generation Evaluation
Centre for Linguistic Science & Technology, IIT Guwahati · Mehta Family School for Data Science & Artificial Intelligence, IIT Guwahati · Department of Humanities & Social Sciences, IIT Guwahati
Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning
Oct 14, 2026Dekun Chen, Renjie Xu, Yingda Shen +2Zero-Shot TTS Speech Prosody
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.
Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations
Jun 18, 2026Masato Takagi, Masaya Kawamura, Reo Shimizu +1Human Preference Evaluation Speech Generation Evaluation
Nagoya Institute of Technology, Japan · LY Corporation, Japan

Jul 2, 2026J/K move · Enter open · S save

Towards a Phonology-Informed Evaluation of Multilingual TTS

Sneha Ray Barman, Neeraj Kumar Sharma, Shakuntala Mahanta

Centre for Linguistic Science & Technology, IIT Guwahati · Mehta Family School for Data Science & Artificial Intelligence, IIT Guwahati · Department of Humanities & Social Sciences, IIT Guwahati

TTS Synthesis Speech Generation Evaluation TTS Evaluation Text-to-Speech

Open paper AskPDF arXiv

Neural TTS systems can sound natural across languages, but naturalness does not guarantee the preservation of sound contrasts that distinguish words from their grammatical forms. Standard metrics like MOS do not test for this. We propose a classifier-based framework that audits TTS output against language-specific phonological patterns using human speech as a benchmark. Testing Assamese advanced tongue root (ATR) vowel harmony with Meta's MMS TTS, we show that a classifier trained on human speech transfers to synthesized speech with minimal loss. The faithfulness audit reveals that [+ATR] mid vowels are realized as [-ATR] in 1/3 tokens despite an underlying [+ATR] specification, a bias absent in human speech. At the word level, predicted ATR labels classify harmony more accurately than transcription labels, indicating a gap between intended and produced phonology. The framework offers task-specific diagnostics and generalizes to other phonological contrasts with measurable acoustic cues.

Abstract

TTS Synthesis

Explore similar work

Towards a Phonology-Informed Evaluation of Multilingual TTS

Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations