A Comprehensive Objective Evaluation of Modern Text-to-Speech for Turkish Using Speech Quality Assessment Models

Yunus Emre Ozkose; Alperen Kahraman; Ali Haznedaroglu

cs.SDOct 5, 2026

A Comprehensive Objective Evaluation of Modern Text-to-Speech for Turkish Using Speech Quality Assessment Models

Authors: Yunus Emre Ozkose, Alperen Kahraman, Ali Haznedaroglu

Organizations: Sestek, Ankara, Türkiye

Abstract

Modern text-to-speech (TTS) systems can clone a target speaker from a short reference clip or be fine-tuned on a target voice, yet their behaviour on morphologically rich, lower-resource languages such as Turkish remain under-characterised. We present a systematic benchmark of four contemporary systems (Chatterbox, CosyVoice, OmniVoice, and VoxCPM2) evaluated across fine-tuned and zero-shot configurations, contrasted with a conventional VITS baseline and anchored to natural gold speech. Each configuration is scored with eighteen complementary objective metrics spanning learned naturalness predictors (UTMOS v2, DNSMOS-Pro, SCOREQ, WhisQA, AudioBox-PQ, NatScore, SpeechLMScore), intelligibility and signal-quality estimators (SQUIM PESQ/SI-SDR/STOI, Brouhaha), speaker similarity, distributional fidelity (TTSDS) and low-level acoustic descriptors. We further analyse how quality varies with utterance length and quantify long-form temporal consistency through speaker-identity and naturalness drift over chunked utterances. We release our evaluation code to support reproducible TTS evaluation.

Figures & tables

Metric	CB	CV	OV	OV-Z	VITS	VOX	Gold
audiobox-pq	7.643	7.793	7.785	7.831	7.654	7.793	7.657
brouhaha	50.132	60.249	61.498	58.066	59.507	62.471	55.309
dnsmos_pro	3.564	3.770	3.954	3.808	4.225	3.902	3.601
dnsmos-pro-bvcc	3.347	3.643	3.745	3.522	3.616	3.533	3.698
dnsmos-pro-nisqa	3.564	3.770	3.954	3.808	4.225	3.902	3.601
dnsmos-pro-vcc2018	3.045	2.925	2.722	2.773	3.138	2.886	3.171

Table 1: Overall objective metrics (mean over the full test set). Best synthetic system per row in bold . CosyVoice is represented by its canonical raw-text configuration. Columns: CB = Chatterbox (zero-shot), CV = CosyVoice (raw text), OV = OmniVoice (finetuned), OV-Z = OmniVoice (zero-shot), VITS = VITS, VOX = VoxCPM2 (zero-shot), Gold = natural reference.

Figure 1: Radar comparison of metrics, normalized to a common $[0,100]\%$ scale, with CosyVoice represented by its canonical raw-text configuration. Legend annotations report, per system, the number of headlines

Len.	System	APQ	BRH	DM	SPK	SISDR	UTM	WHQ	NAT
long	Chatterbox	7.735	53.18	3.654	0.889	26.39	3.179	4.578	-1.282
	CosyVoice	7.836	58.36	3.819	0.900	25.35	3.465	4.765	-1.308
	OmniVoice	7.791	58.12	3.948	0.872	26.23	3.135	4.581	-1.027
	OmniVoice (Z)	7.852	55.10	3.770	0.887	28.49	3.165	4.526	-1.064
	VITS	7.724	56.76	4.228	0.894	24.80	3.077	4.582	-1.641
	VoxCPM2	7.820	59.23	3.926	0.899	26.85	2.914	4.640	-0.803

Table 2: Quality by utterance-length group for representative metrics. APQ = audiobox-pq, BRH = brouhaha, DM = dnsmos_pro, SPK = speaker-similarity, SISDR = squim-si-sdr, UTM = utmos-v2, WHQ = whisqa, NAT = natscore.

System	$\bar{n}_{\text{chunks}}$	sim	std	drift	g-drift	$\Delta_{\text{fl}}$
Chatterbox (ZS)	12.08	0.8309	0.0325	-0.0391	-0.0279	-0.0613
CosyVoice	11.14	0.8533	0.0212	-0.0077	-0.0068	-0.0094
OmniVoice	11.73	0.8054	0.0336	-0.0329	-0.0340	-0.0505
OmniVoice (ZS)	11.88	0.8191	0.0282	-0.0184	-0.0110	-0.0276
VITS	10.42	0.8545	0.0226	-0.0125	-0.0207	-0.0175
VoxCPM2 (ZS)	11.89	0.8493	0.0233	-0.0115	- 0.0049	-0.0003

Table 3: Long-form temporal consistency.

\bar{n}_{\text{chunks}}

= mean chunks per utterance; sim = mean_sim_vs_ref; std = consistency_std_mean (lower is more stable); drift = drift_slope_mean; g-drift = gold_drift_slope_mean;

\Delta_{\text{fl}}

= first_last_delta_mean.

System	UTMOS	std	drift	g-drift	$\Delta_{\text{fl}}$	$\Delta_{\text{gold}}$
Chatterbox (ZS)	3.026	0.293	-0.149	0.148	-0.321	- 0.107
CosyVoice (raw)	3.243	0.233	-0.041	0.124	-0.067	- 0.305
OmniVoice (raw)	2.938	0.267	- 0.057	0.105	-0.076	- 0.015
OmniVoice (ZS)	2.941	0.272	- 0.063	0.138	- 0.090	- 0.022
VITS	2.992	0.260	- 0.118	0.175	- 0.147	- 0.046
VoxCPM2 (ZS)	2.708	0.288	- 0.001	0.148	- 0.039	-0.216

Table 4: Long-form temporal consistency of naturalness (UTMOS v2) per system. UTMOS = mean_utmos_mean; std = consistency_std_mean; drift = drift_slope_mean; g-drift = gold_drift_slope_mean;

\Delta_{\text{fl}}

= first_last_delta_mean;

\Delta_{\text{gold}}

= mean_delta_vs_gold_mean. Positive drift means quality rises over the utterance; gold drifts upward for all systems.

Figure 3: Naturalness (UTMOS-v2) drift/consistency quantities per system: consistency_std_mean , drift_slope_mean , gold_drift_slope_mean , first_last_delta_mean and mean_delta_vs_gold_mean .

Metric	CV-R (raw)	CV-P (phon.)	$\Delta_{\text{P$ - $R}}$	Gold
audiobox-pq	7.793	7.757	$-0.036$	7.657
brouhaha	60.249	58.140	$-2.109$	55.309
dnsmos_pro	3.770	3.853	$+0.083$	3.601
dnsmos-pro-bvcc	3.643	3.360	$-0.283$	3.698
dnsmos-pro-nisqa	3.770	3.853	$+0.083$	3.601
dnsmos-pro-vcc2018	2.925	2.928	$+0.003$	3.171

Table 5: CosyVoice front-end ablation: raw text (CV-R) vs. phonemes (CV-P), mean over the full test set.

\Delta_{\text{P

R}}

is the phoneme

-

raw difference; Gold is the natural reference. Differences are small for most metrics and speaker identity is essentially unchanged.

Explore similar work

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study
Aug 3, 2026Ali Jafar, Amal Sarmad, Shifa Yousaf +1Seed-Tts-Eval Benchmark Flow-Matching Text-To-Speech
FAST School of Computing, National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Punjab, Pakistan
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Aug 10, 2026Oluwanifemi Bamgbose, Simon Rosen, Jash Shah +6Seed-Tts-Eval Benchmark Text-To-Speech Synthesis
ServiceNow
An Evaluation Framework for Text-to-Speech Voice Reconstruction
Jun 19, 2026Ariadna Sanchez, Christoph Minixhofer, Korin Richmond +3Seed-Tts-Eval Benchmark Flow-Matching Text-To-Speech
The Centre for Speech Technology Research, University of Edinburgh, UK

Aug 3, 2026, cs.CLJ/K move · Enter open · S save

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

Ali Jafar, Amal Sarmad, Shifa Yousaf, Maryam Bashir

FAST School of Computing, National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Punjab, Pakistan

Seed-Tts-Eval Benchmark Flow-Matching Text-To-Speech Text-To-Speech Synthesis

Open paper AskPDF arXiv

Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages. However, comprehensive evaluation methodologies that jointly assess perceptual quality, speaker similarity, and acoustic fidelity across diverse speech domains remain limited, particularly for low-resource and underrepresented languages. This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems through domain-specific analysis. The proposed framework integrates complementary subjective and objective evaluation protocols and is demonstrated through a comprehensive case study on a representative low-resource language spanning four speech domains: Formal, Conversational, Literary/Storytelling, and Emotional. Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated using MUSHRA listening tests, ABX discrimination tests, speaker similarity scoring with Resemblyzer, and acoustic analyses based on mel-cepstral distortion (MCD) and F0 RMSE over 960 audio pairs. Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge (mean MCD 12.03 dB; mean F0 RMSE 889 cents), while conversational speech achieves the highest overall acoustic fidelity. Beyond the empirical findings, this work provides a reproducible evaluation framework, publicly releasing evaluation scripts, result tables, and executable Colab notebooks to support standardized benchmarking and future research on TTS evaluation for low-resource languages.

Abstract

Explore similar work

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

An Evaluation Framework for Text-to-Speech Voice Reconstruction