cs.SDNov 3, 2025

Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play

Authors: Jiatong Shi, Jionghao Han, Yichen Lu, Santiago Pascual, Pengfei Wu, Chenye Cui, Shinji Watanabe, Chao Weng, 

Organizations: Carnegie Mellon University · Anuttacon

Abstract

Evaluating spoken role-play requires judging not only linguistic content but also prosody, emotional expression, character embodiment, and contextual fit. Existing automatic evaluation methods often collapse these aspects into coarse scores and are evaluated primarily on synthesized speech. We introduce Speech-DRAME, a framework comprising Speech-DRAME-EvalBench, a bilingual, human-annotated benchmark with contrastive role-, context-, and delivery-mismatch conditions, and DRAME-Eval, a fine-tuned audio-language evaluator. The benchmark supports two complementary protocols: top-down archetype evaluation and bottom-up realism evaluation, with predominantly synthesized benchmark-distribution data and a held-out stress test consisting entirely of authentic human acting. On the benchmark-distribution sets, DRAME-Eval achieves pooled Pearson correlations of 0.629 for archetype evaluation and 0.602 for realism evaluation, significantly outperforming the evaluated zero- and few-shot audio-language judges; modality controls show that its judgments depend on both speech and textual context. This alignment does not transfer to the authentic-speech test set, where DRAME-Eval obtains an average correlation of -0.142 and a within-speaker correlation of +0.029, compared with matched human single-rater agreement of +0.34. These results identify a substantial synthetic-to-authentic transfer gap and establish Speech-DRAME as a resource for studying and improving evaluation of acting differences among real human performances.

Figures & tables

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing

    Apr 13, 2026Xi Chen, Wei Xue, Yike GuoLarge Language Model-Based Role-Play Simulation

  2. DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

    Oct 22, 2026Jingbin Hu, Zihang Yang, Xiaoge Peng +21Speech Generation EvaluationMulti-Speaker TTS