SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing

Yizhong Geng; Yanliang Li; Jinghan Yang; Tianhan Jiang; Yingming Gao; Ya Li

cs.SDAug 13, 2026

SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing

Authors: Yizhong Geng, Yanliang Li, Jinghan Yang, Tianhan Jiang, Yingming Gao, Ya Li

Organizations: Beijing University of Posts and Telecommunications Beijing, China · Beijing DeepLogic Intelligence Technology Co., Ltd. Beijing, China · University of California USA

Abstract

Text-to-speech systems are increasingly used for drama dubbing, yet evaluation protocols remain sentence-level, leaving critical scene-level behaviors insufficiently measured. We present SceneTTS-Bench, a benchmark that evaluates TTS along three dimensions: timbre consistency across character turns, emotional expressiveness on high-tension utterances, and rhythm coherence under segmented long-form synthesis. The corpus comprises real-world and generated drama scripts totaling 160 bilingual scenes and approximately 10,300 utterances, with real-world scripts serving as the primary source (100 scenes) and generated scripts as a supplementary source (60 scenes), demonstrating the framework's extensibility through synthetic data augmentation. A backend-agnostic Canonical Intermediate Representation ensures fair cross-system comparison. Three automatic pipelines produce per-utterance diagnostics: Speaker Consistency Score for timbre-drift detection, Under-Acting Ratio for under-acting identification, and Rate Discontinuity Ratio for rate-discontinuity quantification. Experiments on four TTS systems confirm that each system exhibits distinct weaknesses and that scene-level rankings diverge substantially from sentence-level metrics. Benchmark resources are publicly available at https://piedpiperg.github.io/scenetts-bench/ .

Figures & tables

Figure 1. An example of SceneTTS-Bench evaluation showing scene-level diagnostics for a high-tension dialogue. The figure includes metrics for timbre consistency, emotional expressiveness, and rhythm coherence, as well as per-utterance diagnostics.

Figure 2. Overview of SceneTTS-Bench. Top: the backend-agnostic execution protocol converts drama scripts into a Canonical IR and dispatches to four TTS systems under identical conditions. Bottom: three evaluation pipelines assess timbre consistency (SCS), emotional expressiveness (UAR), and rhythm coherence (RDR), each producing per-utterance diagnostics.

	Real-ZH	Real-EN	Gen-ZH	Gen-EN
Scenes	50	50	30	30
Roles / scene	9.3 (1.6)	3.2 (0.7)	3.0 (0.0)	3.0 (0.0)
Turns / scene	59.9 (0.4)	58.0 (3.1)	77.0 (11.7)	70.4 (8.8)
Utt. len (s)	5.7 (4.8)	3.8 (3.5)	4.1 (2.0)	3.6 (1.7)
High-tension %	14.4 (4.3)%	4.1 (2.5)%	21.3 (2.7)%	20.5 (1.9)%
Split-seg. %	7.9 (4.6)%	16.8 (14.3)%	25.3 (6.4)%	32.0 (10.4)%

Table 1. Corpus statistics by subset and language. Values are per-scene means with standard deviations in parentheses.

System	SCS $\uparrow$	UAR(%) $\downarrow$	RDR(%) $\downarrow$
CosyVoice3	$.90{\scriptstyle\pm.02}$	$81.6{\scriptstyle\pm 2.3}$	$27.6{\scriptstyle\pm 2.4}$
Qwen3-TTS	$\mathbf{.96}{\scriptstyle\pm.01}$	$85.5{\scriptstyle\pm 2.1}$	$\mathbf{17.1}{\scriptstyle\pm 1.8}$
IndexTTS2	$.74{\scriptstyle\pm.03}$	$\mathbf{81.1}{\scriptstyle\pm 2.4}$	$21.4{\scriptstyle\pm 2.0}$
Fish-S2	$.71{\scriptstyle\pm.03}$	$84.5{\scriptstyle\pm 2.6}$	$32.1{\scriptstyle\pm 2.5}$

Table 2. Main results with 95% bootstrap confidence intervals. Best results are in bold .

	SCS $\uparrow$		UAR $\downarrow$		RDR $\downarrow$
System	R	G	R	G	R	G
CosyVoice3	0.88	0.93	82.7	79.8	28.5	26.1
Qwen3-TTS	0.95	0.98	86.4	84.0	17.8	15.9
IndexTTS2	0.72	0.78	81.8	79.9	22.1	20.2
Fish-S2	0.68	0.76	85.4	83.0	33.5	29.8

Table 3. Scene-level metrics by script source. R: real-world scripts; G: generated scripts.

	Sentence-Level			Scene-Level
System	WER-zh $\downarrow$	WER-en $\downarrow$	SIM $\uparrow$	SCS $\uparrow$	UAR $\downarrow$	RDR $\downarrow$
CosyVoice3	0.83 3	1.42 3	0.761 4	0.90 2	81.6 2	$27.6_{3}$
Qwen3-TTS	0.77 2	1.24 2	0.794 1	$\mathbf{0.96}_{1}$	85.5 4	$\mathbf{17.1}_{1}$
IndexTTS2	1.06 4	1.52 4	0.772 3	0.74 3	81.1 1	$21.4_{2}$
Fish-S2	0.54 1	0.99 1	0.783 2	0.71 4	84.5 3	$32.1_{4}$

Table 4. Sentence- and scene-level metric ranks.

Figure 3. Normalized sentence-level (grey labels) and scene-level (black labels) profiles; higher is better on every axis.

		Utterance-Level			Sys.
Dimension	Metric	Spear.	Pear.	Kend.	$\rho$
Timbre	SCS	0.782	0.768	0.604	1.00
Emotion	UAR	0.613	0.587	0.451	1.00
Rhythm	RDR	0.724	0.695	0.547	0.80

Table 5. Correlation with human ratings over 800 samples (

p<0.001

); system-level

\rho

uses four backend means.

Long-Form TTS Synthesis

See all 10 papers

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study
Aug 3, 2026Ali Jafar, Amal Sarmad, Shifa Yousaf +1Speech Generation Evaluation TTS Evaluation
FAST School of Computing, National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Punjab, Pakistan
DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events
Sep 11, 2026Qingxiang Guo, Wenke Fan, Shuofeng Zhao +9Spoken Dialogue Systems Synthetic Data Generation for Language Models
Zuoyebang Education Technology, Beijing, China
Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis
Sep 3, 2026Sanyuan Chen, Min-Jae Hwang, Sho Inoue +12TTS Synthesis Audio Diffusion Models
FAIR at Meta

Aug 3, 2026J/K move · Enter open · S save

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

Ali Jafar, Amal Sarmad, Shifa Yousaf, Maryam Bashir

FAST School of Computing, National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Punjab, Pakistan

Speech Generation Evaluation TTS Evaluation Speech Quality Assessment Low-Resource TTS Synthesis Text-to-Speech

Open paper AskPDF arXiv

Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages. However, comprehensive evaluation methodologies that jointly assess perceptual quality, speaker similarity, and acoustic fidelity across diverse speech domains remain limited, particularly for low-resource and underrepresented languages. This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems through domain-specific analysis. The proposed framework integrates complementary subjective and objective evaluation protocols and is demonstrated through a comprehensive case study on a representative low-resource language spanning four speech domains: Formal, Conversational, Literary/Storytelling, and Emotional. Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated using MUSHRA listening tests, ABX discrimination tests, speaker similarity scoring with Resemblyzer, and acoustic analyses based on mel-cepstral distortion (MCD) and F0 RMSE over 960 audio pairs. Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge (mean MCD 12.03 dB; mean F0 RMSE 889 cents), while conversational speech achieves the highest overall acoustic fidelity. Beyond the empirical findings, this work provides a reproducible evaluation framework, publicly releasing evaluation scripts, result tables, and executable Colab notebooks to support standardized benchmarking and future research on TTS evaluation for low-resource languages.

Abstract

Long-Form TTS Synthesis

Explore similar work

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events

Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis