DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Jingbin Hu; Zihang Yang; Xiaoge Peng; Haoyu Zhang; Ziyu Zhang; Qirui Zhan; Huakang Chen; Zhao Guo

eess.ASOct 22, 2026

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Authors: Jingbin Hu, Zihang Yang, Xiaoge Peng, Haoyu Zhang, Ziyu Zhang, Qirui Zhan, Huakang Chen, Zhao Guo,

Organizations: Shenzhen Pimei Technology Co., Ltd. · Yutu Zhineng

Abstract

Recent advances in speech generation are shifting toward context-aware, multi-turn, and multi-speaker synthesis. While these capabilities are in high demand for applications such as audiobook and audio drama production, evaluation has not kept pace with these advances. Beyond traditional zero-shot Text-to-Speech (TTS), modern systems in this domain primarily fall into three categories: instruction-controllable generation models, multi-turn dialogue models, and the audio branches of audio-visual generation models. However, existing benchmarks evaluate these paradigms in isolation, failing to provide a unified assessment of context modeling, instruction following, and multi-speaker generation. Meanwhile, recent progress in audio understanding models has made it feasible to evaluate such complex scenarios using LLM-as-a-Judge approaches. To bridge this gap, we introduce DRAMA-Eval, a comprehensive benchmark for evaluating such systems. DRAMA-Eval utilizes structured script contexts and offers a holistic evaluation framework covering objective, local, and global aspects of generation quality, enabling the assessment of acoustic fidelity, sentence-level execution, and long-range coherence. We benchmark leading open-source and commercial models on a human?verified dataset. The results reveal critical weaknesses in long-range speaker consistency and implicit instruction following that require deep contextual understanding. To facilitate reproducible research, we release a 12-hour bilingual (Chinese-English) test set of manually corrected real-world scripts, accompanied by ground-truth audio, evaluation code, and benchmark results.

Figures & tables

Benchmark	Lang.	Context / conditioning	Evaluation unit	Instr.	Identity	Scene
InstructTTSEval Huang et al. (2025)	EN, ZH	Acoustic, style, and role instructions	Utterance	$\checkmark$	–	–
MINT-Bench Chen et al. (2026)	10	Attribute and style instructions	Utterance	$\checkmark$	–	–
CEAEval Wang et al. (2026)	ZH	Narrative context for a target utterance	Utterance	–	–	–
SwanBench-Speech Pan et al. (2026)	EN, ZH	Long-form texts and speaker-labeled dialogues	Sequence	–	$\checkmark$	$\checkmark$
DRAMA-Eval (ours)	EN, ZH	Scene context, character profiles, and turn-level instructions	Scene	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison with related benchmarks. Instr.: instruction-following evaluation; Identity: cross-utterance speaker consistency; Scene: complete multi-speaker interaction evaluation.

\checkmark

indicates a dedicated evaluation target; “–” indicates otherwise.

Figure 1: Data construction pipeline of DRAMA-Eval, comprising real-world audio collection, automatic annotation, script refinement, and human review.

Figure 2: Distributions of audio sources, scene categories, speaker counts, and turn counts in DRAMA-Eval. All values are percentages.

Model	A1 $\downarrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	B3 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
SeedAudio ByteDance (2026)	2.58	$0.74\pm 0.05$	$\mathbf{62.2}\pm 0.4$	$57.4\pm 0.2$	$\mathbf{64.2}\pm 0.6$	$\mathbf{67.4}\pm 0.7$	$\mathbf{73.2}\pm 0.4$	$\mathbf{75.7}\pm 1.1$
StepAudio 3 Gen Team (2026c)	3.78	$0.76\pm 0.05$	$57.3\pm 0.6$	$\mathbf{58.3}\pm 0.3$	$56.9\pm 0.2$	$60.2\pm 1.7$	$64.7\pm 1.5$	$64.6\pm 1.4$
MiniMax-H3 MiniMax AI (2026)	3.84	$0.58\pm 0.12$	$56.6\pm 0.2$	$57.9\pm 0.2$	$57.6\pm 0.3$	$49.7\pm 0.1$	$42.2\pm 0.8$	$43.2\pm 0.6$
Qwen3-TTS (VD) Team (2026b)	2.73	$0.59\pm 0.08$	$59.7\pm 0.2$	$54.6\pm 0.5$	$60.6\pm 0.5$	$40.0\pm 0.0$	$28.2\pm 0.0$	$27.7\pm 0.0$
FireRedTTS3 Shen et al. (2026a)	2.64	$0.62\pm 0.07$	$52.1\pm 0.3$	$55.1\pm 0.4$	$50.4\pm 0.7$	$39.8\pm 0.7$	$31.9\pm 0.8$	$30.7\pm 1.4$
VoxCPM2 Zhou et al. (2026b)	3.25	$0.67\pm 0.06$	$48.1\pm 0.5$	$48.6\pm 0.3$	$48.0\pm 0.7$	$36.4\pm 0.2$	$29.8\pm 0.5$	$29.1\pm 0.8$

Table 2: DRAMA-Eval results on Chinese. Objective metrics: A1 measures textual fidelity using character error rate (CER, %; lower is better); A2 measures speaker consistency across utterances (higher is better). Local evaluation: B1 assesses instruction following, B2 assesses utterance-level prosody and fluency, and B3 assesses local expressive delivery. Global evaluation: C1 assesses interaction naturalness, C2 assesses overall fluency across turns, and C3 assesses scene immersion. B1–C3 are LALM-evaluated, tie-adjusted win rates against the corresponding real-world audio, reported in percentage points with the % sign omitted; 50 indicates parity with the reference under the evaluation protocol. These scores are averaged over three independent evaluation runs, with accompanying

\pm

values indicating run-to-run variability. Dashes indicate that instruction-following evaluation is not reported for models without an instruction-control interface. Best mean results are highlighted in bold.

Model	A1 $\downarrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	B3 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
SeedAudio ByteDance (2026)	4.45	$0.68\pm 0.07$	$\mathbf{56.5}\pm 0.8$	$\mathbf{50.5}\pm 0.2$	$\mathbf{57.0}\pm 0.7$	$\mathbf{56.8}\pm 1.0$	$\mathbf{54.5}\pm 0.4$	$\mathbf{55.7}\pm 0.8$
StepAudio 3 Gen Team (2026c)	6.00	$0.67\pm 0.10$	$48.5\pm 0.4$	$48.1\pm 0.4$	$45.2\pm 0.2$	$43.4\pm 0.5$	$33.8\pm 0.6$	$33.6\pm 0.5$
MiniMax-H3 MiniMax AI (2026)	10.56	$0.44\pm 0.16$	$50.9\pm 0.3$	$\mathbf{50.5}\pm 0.2$	$51.6\pm 0.6$	$37.8\pm 0.3$	$22.4\pm 0.6$	$23.9\pm 0.9$
Qwen3-TTS (VD) Team (2026b)	4.16	$0.40\pm 0.10$	$46.9\pm 0.3$	$48.3\pm 0.3$	$44.5\pm 0.3$	$34.3\pm 0.0$	$16.1\pm 0.0$	$15.0\pm 0.0$
FireRedTTS3 Shen et al. (2026a)	3.97	$0.47\pm 0.09$	$37.0\pm 0.2$	$46.2\pm 0.4$	$30.7\pm 0.3$	$30.9\pm 1.1$	$13.2\pm 0.6$	$11.5\pm 0.5$
VoxCPM2 Zhou et al. (2026b)	8.77	$0.69\pm 0.06$	$32.5\pm 0.4$	$37.9\pm 0.0$	$28.5\pm 0.7$	$28.9\pm 0.6$	$16.7\pm 0.4$	$14.8\pm 0.6$

Table 3: DRAMA-Eval results on English. A1 reports WER (%); all other metrics and reporting conventions follow Table 2 . Best mean results are highlighted in bold.

Dimension	Individual	Consensus
B1	65.23 (256)	65.91 (66)
B2	66.12 (256)	67.53 (65)
B3	66.02 (256)	71.85 (69)
C1	73.05 (154)	81.58 (38)
C2	69.27 (192)	77.27 (44)
C3	69.27 (192)	76.19 (42)

Table 4: Human–LLM partial-credit agreement (%) for the judge. Exact matches score 1, tie/non-tie mismatches score 0.5, and opposite preferences score 0. Parentheses report the number of annotation-level comparisons for Individual and valid cases for Consensus. Consensus uses the most frequent human judgment, requiring at least two matching votes.

Lang.	Global context	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
ZH	Without	61.1	66.2	64.4
	With	67.4	73.2	75.7
EN	Without	50.8	48.5	45.9
	With	56.8	54.5	55.7

Table 5: Ablation of global context on SeedAudio. C1–C3 are reported.

Speech Generation Evaluation

Speech generation evaluation measures the intelligibility, naturalness, fidelity, and controllability of synthesized and conversational speech.

See all 57 papers

Sep 27, 2026

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.

Kangxiang Xia, Xinfa Zhu, HangRui Hu +8

Independent Researcher

TTS Synthesis Speech Editing Multi-Speaker TTS Controllable Speech Generation

Ask PDF

Aug 12, 2026

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.

Xingwei Sun, Heinrich Dinkel, Gang Li +7

MiLM Plus, Xiaomi Inc., Beijing, China · X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China

Text-to-Audio Generation Autoregressive Generation Audio Generation

Ask PDF

Aug 13, 2026

SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing

Text-to-speech systems are increasingly used for drama dubbing, yet evaluation protocols remain sentence-level, leaving critical scene-level behaviors insufficiently measured. We present SceneTTS-Bench, a benchmark that evaluates TTS along three dimensions: timbre consistency across character turns, emotional expressiveness on high-tension utterances, and rhythm coherence under segmented long-form synthesis. The corpus comprises real-world and generated drama scripts totaling 160 bilingual scenes and approximately 10,300 utterances, with real-world scripts serving as the primary source (100 scenes) and generated scripts as a supplementary source (60 scenes), demonstrating the framework's extensibility through synthetic data augmentation. A backend-agnostic Canonical Intermediate Representation ensures fair cross-system comparison. Three automatic pipelines produce per-utterance diagnostics: Speaker Consistency Score for timbre-drift detection, Under-Acting Ratio for under-acting identification, and Rate Discontinuity Ratio for rate-discontinuity quantification. Experiments on four TTS systems confirm that each system exhibits distinct weaknesses and that scene-level rankings diverge substantially from sentence-level metrics. Benchmark resources are publicly available at https://piedpiperg.github.io/scenetts-bench/ .

Yizhong Geng, Yanliang Li, Jinghan Yang +3

Beijing University of Posts and Telecommunications Beijing, China · Beijing DeepLogic Intelligence Technology Co., Ltd. Beijing, China · University of California USA

Long-Form TTS Synthesis TTS Synthesis TTS Evaluation Emotional Speech Synthesis

Ask PDF

Abstract

Speech Generation Evaluation

Explore similar work

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing