DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Jingbin Hu; Zihang Yang; Xiaoge Peng; Haoyu Zhang; Ziyu Zhang; Qirui Zhan; Huakang Chen; Zhao Guo

eess.ASOct 22, 2026

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Authors: Jingbin Hu, Zihang Yang, Xiaoge Peng, Haoyu Zhang, Ziyu Zhang, Qirui Zhan, Huakang Chen, Zhao Guo,

Organizations: Shenzhen Pimei Technology Co., Ltd. · Yutu Zhineng

Abstract

Recent advances in speech generation are shifting toward context-aware, multi-turn, and multi-speaker synthesis. While these capabilities are in high demand for applications such as audiobook and audio drama production, evaluation has not kept pace with these advances. Beyond traditional zero-shot Text-to-Speech (TTS), modern systems in this domain primarily fall into three categories: instruction-controllable generation models, multi-turn dialogue models, and the audio branches of audio-visual generation models. However, existing benchmarks evaluate these paradigms in isolation, failing to provide a unified assessment of context modeling, instruction following, and multi-speaker generation. Meanwhile, recent progress in audio understanding models has made it feasible to evaluate such complex scenarios using LLM-as-a-Judge approaches. To bridge this gap, we introduce DRAMA-Eval, a comprehensive benchmark for evaluating such systems. DRAMA-Eval utilizes structured script contexts and offers a holistic evaluation framework covering objective, local, and global aspects of generation quality, enabling the assessment of acoustic fidelity, sentence-level execution, and long-range coherence. We benchmark leading open-source and commercial models on a human?verified dataset. The results reveal critical weaknesses in long-range speaker consistency and implicit instruction following that require deep contextual understanding. To facilitate reproducible research, we release a 12-hour bilingual (Chinese-English) test set of manually corrected real-world scripts, accompanied by ground-truth audio, evaluation code, and benchmark results.

Figures & tables

Benchmark	Lang.	Context / conditioning	Evaluation unit	Instr.	Identity	Scene
InstructTTSEval Huang et al. (2025)	EN, ZH	Acoustic, style, and role instructions	Utterance	$\checkmark$	–	–
MINT-Bench Chen et al. (2026)	10	Attribute and style instructions	Utterance	$\checkmark$	–	–
CEAEval Wang et al. (2026)	ZH	Narrative context for a target utterance	Utterance	–	–	–
SwanBench-Speech Pan et al. (2026)	EN, ZH	Long-form texts and speaker-labeled dialogues	Sequence	–	$\checkmark$	$\checkmark$
DRAMA-Eval (ours)	EN, ZH	Scene context, character profiles, and turn-level instructions	Scene	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison with related benchmarks. Instr.: instruction-following evaluation; Identity: cross-utterance speaker consistency; Scene: complete multi-speaker interaction evaluation.

\checkmark

indicates a dedicated evaluation target; “–” indicates otherwise.

Figure 1: Data construction pipeline of DRAMA-Eval, comprising real-world audio collection, automatic annotation, script refinement, and human review.

Figure 2: Distributions of audio sources, scene categories, speaker counts, and turn counts in DRAMA-Eval. All values are percentages.

Model	A1 $\downarrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	B3 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
SeedAudio ByteDance (2026)	2.58	$0.74\pm 0.05$	$\mathbf{62.2}\pm 0.4$	$57.4\pm 0.2$	$\mathbf{64.2}\pm 0.6$	$\mathbf{67.4}\pm 0.7$	$\mathbf{73.2}\pm 0.4$	$\mathbf{75.7}\pm 1.1$
StepAudio 3 Gen Team (2026c)	3.78	$0.76\pm 0.05$	$57.3\pm 0.6$	$\mathbf{58.3}\pm 0.3$	$56.9\pm 0.2$	$60.2\pm 1.7$	$64.7\pm 1.5$	$64.6\pm 1.4$
MiniMax-H3 MiniMax AI (2026)	3.84	$0.58\pm 0.12$	$56.6\pm 0.2$	$57.9\pm 0.2$	$57.6\pm 0.3$	$49.7\pm 0.1$	$42.2\pm 0.8$	$43.2\pm 0.6$
Qwen3-TTS (VD) Team (2026b)	2.73	$0.59\pm 0.08$	$59.7\pm 0.2$	$54.6\pm 0.5$	$60.6\pm 0.5$	$40.0\pm 0.0$	$28.2\pm 0.0$	$27.7\pm 0.0$
FireRedTTS3 Shen et al. (2026a)	2.64	$0.62\pm 0.07$	$52.1\pm 0.3$	$55.1\pm 0.4$	$50.4\pm 0.7$	$39.8\pm 0.7$	$31.9\pm 0.8$	$30.7\pm 1.4$
VoxCPM2 Zhou et al. (2026b)	3.25	$0.67\pm 0.06$	$48.1\pm 0.5$	$48.6\pm 0.3$	$48.0\pm 0.7$	$36.4\pm 0.2$	$29.8\pm 0.5$	$29.1\pm 0.8$

Table 2: DRAMA-Eval results on Chinese. Objective metrics: A1 measures textual fidelity using character error rate (CER, %; lower is better); A2 measures speaker consistency across utterances (higher is better). Local evaluation: B1 assesses instruction following, B2 assesses utterance-level prosody and fluency, and B3 assesses local expressive delivery. Global evaluation: C1 assesses interaction naturalness, C2 assesses overall fluency across turns, and C3 assesses scene immersion. B1–C3 are LALM-evaluated, tie-adjusted win rates against the corresponding real-world audio, reported in percentage points with the % sign omitted; 50 indicates parity with the reference under the evaluation protocol. These scores are averaged over three independent evaluation runs, with accompanying

\pm

values indicating run-to-run variability. Dashes indicate that instruction-following evaluation is not reported for models without an instruction-control interface. Best mean results are highlighted in bold.

Model	A1 $\downarrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	B3 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
SeedAudio ByteDance (2026)	4.45	$0.68\pm 0.07$	$\mathbf{56.5}\pm 0.8$	$\mathbf{50.5}\pm 0.2$	$\mathbf{57.0}\pm 0.7$	$\mathbf{56.8}\pm 1.0$	$\mathbf{54.5}\pm 0.4$	$\mathbf{55.7}\pm 0.8$
StepAudio 3 Gen Team (2026c)	6.00	$0.67\pm 0.10$	$48.5\pm 0.4$	$48.1\pm 0.4$	$45.2\pm 0.2$	$43.4\pm 0.5$	$33.8\pm 0.6$	$33.6\pm 0.5$
MiniMax-H3 MiniMax AI (2026)	10.56	$0.44\pm 0.16$	$50.9\pm 0.3$	$\mathbf{50.5}\pm 0.2$	$51.6\pm 0.6$	$37.8\pm 0.3$	$22.4\pm 0.6$	$23.9\pm 0.9$
Qwen3-TTS (VD) Team (2026b)	4.16	$0.40\pm 0.10$	$46.9\pm 0.3$	$48.3\pm 0.3$	$44.5\pm 0.3$	$34.3\pm 0.0$	$16.1\pm 0.0$	$15.0\pm 0.0$
FireRedTTS3 Shen et al. (2026a)	3.97	$0.47\pm 0.09$	$37.0\pm 0.2$	$46.2\pm 0.4$	$30.7\pm 0.3$	$30.9\pm 1.1$	$13.2\pm 0.6$	$11.5\pm 0.5$
VoxCPM2 Zhou et al. (2026b)	8.77	$0.69\pm 0.06$	$32.5\pm 0.4$	$37.9\pm 0.0$	$28.5\pm 0.7$	$28.9\pm 0.6$	$16.7\pm 0.4$	$14.8\pm 0.6$

Table 3: DRAMA-Eval results on English. A1 reports WER (%); all other metrics and reporting conventions follow Table 2 . Best mean results are highlighted in bold.

Dimension	Individual	Consensus
B1	65.23 (256)	65.91 (66)
B2	66.12 (256)	67.53 (65)
B3	66.02 (256)	71.85 (69)
C1	73.05 (154)	81.58 (38)
C2	69.27 (192)	77.27 (44)
C3	69.27 (192)	76.19 (42)

Table 4: Human–LLM partial-credit agreement (%) for the judge. Exact matches score 1, tie/non-tie mismatches score 0.5, and opposite preferences score 0. Parentheses report the number of annotation-level comparisons for Individual and valid cases for Consensus. Consensus uses the most frequent human judgment, requiring at least two matching votes.

Lang.	Global context	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$
ZH	Without	61.1	66.2	64.4
	With	67.4	73.2	75.7
EN	Without	50.8	48.5	45.9
	With	56.8	54.5	55.7

Table 5: Ablation of global context on SeedAudio. C1–C3 are reported.

Speech Generation Evaluation

Speech generation evaluation measures the intelligibility, naturalness, fidelity, and controllability of synthesized and conversational speech.

See all 57 papers

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Sep 27, 2026Kangxiang Xia, Xinfa Zhu, HangRui Hu +8TTS Synthesis Speech Editing
Independent Researcher
MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
Aug 12, 2026Xingwei Sun, Heinrich Dinkel, Gang Li +7Text-to-Audio Generation Autoregressive Generation
MiLM Plus, Xiaomi Inc., Beijing, China · X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China
SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing
Aug 13, 2026Yizhong Geng, Yanliang Li, Jinghan Yang +3Long-Form TTS Synthesis TTS Synthesis
Beijing University of Posts and Telecommunications Beijing, China · Beijing DeepLogic Intelligence Technology Co., Ltd. Beijing, China · University of California USA

Sep 27, 2026J/K move · Enter open · S save

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

Kangxiang Xia, Xinfa Zhu, HangRui Hu, Kexin Huang+7

Independent Researcher

TTS Synthesis Speech Editing Multi-Speaker TTS Controllable Speech Generation

Open paper AskPDF arXiv

Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.

Abstract

Speech Generation Evaluation

Explore similar work

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing