Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
Organizations: Carnegie Mellon University · Anuttacon
Abstract
Evaluating spoken role-play requires judging not only linguistic content but also prosody, emotional expression, character embodiment, and contextual fit. Existing automatic evaluation methods often collapse these aspects into coarse scores and are evaluated primarily on synthesized speech. We introduce Speech-DRAME, a framework comprising Speech-DRAME-EvalBench, a bilingual, human-annotated benchmark with contrastive role-, context-, and delivery-mismatch conditions, and DRAME-Eval, a fine-tuned audio-language evaluator. The benchmark supports two complementary protocols: top-down archetype evaluation and bottom-up realism evaluation, with predominantly synthesized benchmark-distribution data and a held-out stress test consisting entirely of authentic human acting. On the benchmark-distribution sets, DRAME-Eval achieves pooled Pearson correlations of 0.629 for archetype evaluation and 0.602 for realism evaluation, significantly outperforming the evaluated zero- and few-shot audio-language judges; modality controls show that its judgments depend on both speech and textual context. This alignment does not transfer to the authentic-speech test set, where DRAME-Eval obtains an average correlation of -0.142 and a within-speaker correlation of +0.029, compared with matched human single-rater agreement of +0.34. These results identify a substantial synthetic-to-authentic transfer gap and establish Speech-DRAME as a resource for studying and improving evaluation of acting differences among real human performances.
Figures & tables
| Method / focus | Mod. | src. | Labels | Lang. | Neg. | Strategy | SEval- ready |
|---|---|---|---|---|---|---|---|
| Text-based role-play | |||||||
| RoleLLM ( Wang et al., 2024a ) Persona fidelity | T | – | Judge | EN+ZH | ✗ | — | ✗ |
| CharacterEval ( Tu et al., 2024 ) RPCA, reward model | T | – | Both | ZH | ✗ | — | ✓ |
| RoleEval ( Shen et al., 2023 ) Knowledge / persona | T | – | Judge | EN+ZH | ✗ | — | ✗ |
| RoleMRC ( Lu et al., 2025 ) RP+instruction | T | – | Judge | EN | ✗ | — | ✗ |
| CharacterBox ( Wang et al., 2025d ) Virtual-world RP | T | – | Judge | EN+ZH | ✗ | — | ✗ |
| Aspect | Archetype (top-down) | Realism (bottom-up) |
| Curation operator | (Eq. 3 ) | (Eq. 4 ) |
| Source of | Synthesized by an SFnd | Real human recording (TV/film/recordings) |
| Source of | Stereotypical role keyword + template | Retrieved from media context |
| Source of | Stereotyped scene prompt | LLM-summarized local plot scene |
| Granularity of | 3 dims (AQ/HL context-free; AP context-dependent) (Sec. 4.2 ) | 5 fine-grained dims w/ sub-dims (Sec. 5.2 ) |
| Negative samples ( ) | Not used | Delivery / role / context mismatches |
| Aspect | Archetype (Sec. 4 ) | Realism (Sec. 5 ) |
| Curation operator | (Eq. 3 ) | (Eq. 4 ) |
| Languages | EN + ZH | EN + ZH (Mandarin-majority; real-rec pool & delivery benchmark ZH-only) |
| Source of | Synth. by 7–8 SFnd systems | Mixed: real (TV/film/original) + TTS; only the held-out real-rec test set is all-real ( – of train/basic audio is TTS) |
| # scenarios | 552 (per language) | clip-grounded (bottom-up; no fixed template) |
| # annotated samples (total) | 8,272 | 15,414 (train 10,990 / dist. test 2,000 / real-rec pool 2,164 / delivery 260) |
| Train / Test split | 6,773 / 1,499 | 10,990 / 2,000 (+ 2,164 real-rec pool + 260 delivery) |
| Model | License / Availability | Architecture | Reference |
| English Models (8) | |||
| GPT-4o-realtime | Proprietary | Unknown | ( OpenAI, 2025b ) |
| Doubao | Proprietary | Unknown | ( ByteDance, 2025 ) |
| GPT-4o (Advanced Voice Mode) | Proprietary | Unknown | ( OpenAI, 2025a ) |
| GLM4-Voice | Open-source | End-to-end | ( Zeng et al., 2024 ) |
| KimiAudio | Open-source | End-to-end | ( Ding et al., 2025 ) |
| Dimension | Sub-dimension | Discussion Focus |
| Prosodic Dynamics | Pitch Variation | Contour richness; range; monotony vs. expressiveness |
| Rhythmic Naturalness | Pacing; timing; pause placement; flow | |
| Stress & Emphasis | Accenting; highlight key words; communicative salience | |
| Emotional Expressiveness | Emotion Accuracy | Correct emotion type; scene-appropriate affect |
| Emotion Intensity | Strength calibration; under-/over-shooting | |
| Emotion Transition | Smooth shifts; narrative-motivated changes |
| System | Type | Prompt Type | References |
| OpenAI TTS-1 HD | Proprietary | Speaker Assignment | API Call ( OpenAI, 2025c ) |
| Gemini2.5pro-preview-TTS | Proprietary | Speaker Assignment with Instruction | API Call ( Google Cloud, 2025 ) |
| ElevenLabs Multilingual V2 | Proprietary | Similar Voice Matching | API Call ( ElevenLabs, 2025 ) |
| Proprietary | Voice Design | API Call ( ElevenLabs, 2025 ) | |
| Proprietary | Voice Clone | API Call ( ElevenLabs, 2025 ) | |
| HumeTTS | Proprietary | Voice Design | API Call ( Hume AI, 2025 ) |
| Config. | Model | Content Pass Acc. | Audio Quality | Human Likeness | Appropriateness | Avg. |
| Zeroshot | Gemini2.5Pro | 89.3% | 0.147 | 0.253 | 0.309 | 0.236 |
| GPT4o-audio | 89.9% | 0.094 | 0.409 | 0.196 | 0.233 | |
| KimiAudio | 71.7% | 0.015 | 0.105 | 0.089 | 0.070 | |
| Qwen2Audio † | 86.7% | – | – | – | – | |
| Qwen2.5Omni | 84.1% | -0.029 | 0.004 | 0.003 | -0.008 | |
| Fewshot | Gemini2.5Pro | 91.1% | 0.006 | 0.203 | 0.264 | 0.158 |
| Config. | Model | Prosodic Dynamics | Emotional Expressiveness | Character Consistency | Contextual Relevance | Avg. | ||||||
| Pitch | Rhythm | Stress | Accuracy | Intensity | Transition | Identity | Traits | Local | Global | |||
| Zeroshot | Gemini2.5Pro | 0.285 | 0.317 | 0.292 | 0.378 | 0.228 | 0.203 | 0.525 | 0.505 | 0.339 | 0.459 | 0.353 |
| GPT4o-audio | 0.187 | 0.178 | 0.187 | 0.336 | 0.139 | 0.176 | 0.358 | 0.413 | 0.308 | 0.466 | 0.275 | |
| KimiAudio | 0.049 | 0.041 | 0.073 | 0.025 | 0.105 | 0.095 | 0.082 | 0.019 | 0.016 | 0.033 | 0.054 | |
| Qwen2Audio | 0.031 | 0.035 | 0.055 | -0.003 | 0.080 | 0.060 | 0.012 | -0.022 | 0.063 | -0.002 | 0.031 | |
| Qwen2.5Omni | 0.131 | 0.076 | 0.103 | 0.235 | 0.071 | 0.091 | 0.151 | 0.306 | 0.152 | 0.211 | 0.153 | |
| Config. | Model | Prosodic Dynamics | Emotional Expressiveness | Character Consistency | Contextual Relevance | Avg. | ||||||
| Pitch | Rhythm | Stress | Accuracy | Intensity | Transition | Identity | Traits | Local | Global | |||
| Zeroshot | Gemini2.5Pro | 0.159 | 0.070 | 0.103 | 0.070 | 0.052 | 0.096 | 0.092 | 0.077 | 0.075 | 0.028 | 0.082 |
| GPT4o-audio | 0.067 | -0.013 | 0.035 | 0.013 | 0.042 | 0.036 | -0.004 | 0.130 | 0.065 | 0.018 | 0.039 | |
| KimiAudio | -0.115 | -0.039 | -0.116 | -0.020 | -0.122 | -0.080 | 0.114 | -0.032 | -0.137 | -0.080 | -0.063 | |
| Qwen2.5Omni | 0.078 | 0.071 | 0.043 | 0.074 | 0.047 | 0.045 | -0.082 | 0.076 | 0.102 | 0.098 | 0.055 | |
| Qwen2Audio | 0.055 | 0.057 | 0.088 | 0.072 | 0.077 | 0.057 | -0.006 | -0.040 | 0.050 | 0.111 | 0.052 | |
| Config. | Content Pass Acc. | Audio Quality | Human Likeness | Appropriateness | Avg. |
| Base-FT | 92.1% | 0.688 | 0.648 | 0.551 | 0.629 |
| Base-LoRA | 92.6% | 0.650 | 0.684 | 0.492 | 0.608 |
| Instruct-FT | 92.6% | 0.668 | 0.609 | 0.546 | 0.608 |
| Instruct-LoRA | 93.6% | 0.682 | 0.679 | 0.525 | 0.629 |
| Config. | Prosodic Dynamics | Emotional Expressiveness | Character Consistency | Contextual Relevance | Avg. | ||||||
| Pitch | Rhythm | Stress | Accuracy | Intensity | Transition | Identity | Traits | Local | Global | ||
| Base-FT | 0.643 | 0.633 | 0.655 | 0.657 | 0.600 | 0.599 | 0.689 | 0.668 | 0.611 | 0.707 | 0.646 |
| Base-LoRA | 0.628 | 0.633 | 0.643 | 0.637 | 0.586 | 0.590 | 0.668 | 0.651 | 0.578 | 0.684 | 0.630 |
| Instruct-FT | 0.657 | 0.655 | 0.675 | 0.674 | 0.627 | 0.626 | 0.752 | 0.685 | 0.615 | 0.714 | 0.668 |
| Instruct-LoRA | 0.621 | 0.627 | 0.631 | 0.632 | 0.596 | 0.601 | 0.660 | 0.655 | 0.563 | 0.668 | 0.625 |
| Config. | Prosodic Dynamics | Emotional Expressiveness | Character Consistency | Contextual Relevance | Avg. | ||||||
| Pitch | Rhythm | Stress | Accuracy | Intensity | Transition | Identity | Traits | Local | Global | ||
| Base-FT | -0.422 | -0.119 | -0.333 | -0.261 | -0.330 | -0.351 | 0.059 | 0.538 | -0.3083 | -0.153 | -0.168 |
| Base-LoRA | -0.372 | -0.016 | -0.304 | -0.195 | -0.273 | -0.249 | -0.091 | 0.458 | -0.291 | -0.146 | -0.148 |
| Instruct-FT | -0.269 | -0.031 | -0.202 | -0.208 | -0.222 | -0.242 | 0.076 | 0.543 | -0.219 | -0.094 | -0.087 |
| Instruct-LoRA | -0.357 | -0.051 | -0.228 | -0.161 | -0.333 | -0.291 | -0.041 | 0.419 | -0.262 | -0.118 | |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Model (Mandarin) | Appropriateness | Human Likeness | Audio Quality | Content Pass |
| Doubao | 3.8236 0.6658 | 3.6572 0.6427 | 4.3743 0.6991 | 0.9848 0.1224 |
| G-CosyVoice2 | 3.4243 0.7389 | 3.3156 0.7215 | 4.0406 0.7298 | 0.9725 0.1637 |
| Q-CosyVoice2 | 3.4228 0.7788 | 3.2750 0.7335 | 4.0236 0.7661 | 0.9674 0.1776 |
| KimiAudio | 3.3866 1.1086 | 3.3391 1.0591 | 3.9525 1.2801 | 0.8623 0.3446 |
| GLM4-Voice | 3.2022 0.9597 | 3.0275 0.8687 | 3.9246 1.1105 | 0.8949 0.3067 |
| GPT-4o-realtime | 3.0228 0.8581 | 2.6536 0.7151 | 3.7297 1.0418 | 0.9098 0.2865 |
| Domain | Mandarin | English | ||
| Score (mean std) | Inter-model Var. | Score (mean std) | Inter-model Var. | |
| NamedCharacters | 3.0343 1.1087 | 0.2626 | 1.9896 1.0409 | 0.2421 |
| FantasyOccupation | 3.1629 0.9661 | 0.3439 | 1.9444 0.9809 | 0.2089 |
| Traits | 3.1670 1.2109 | 0.3269 | 2.1518 1.0572 | 0.2966 |
| Events | 3.2535 1.1523 | 0.4358 | 2.2608 1.1280 | 0.3980 |
| DailyOccupation | 3.2760 0.8652 | 0.0555 | 2.5129 1.1781 | 0.2704 |
| Config. | Prosodic Dynamics | Emotional Expressiveness | Character Consistency | Contextual Relevance | Avg. | ||||||
| Pitch | Rhythm | Stress | Accuracy | Intensity | Transition | Identity | Traits | Local | Global | ||
| Zeroshot-1run | 0.277 | 0.325 | 0.303 | 0.355 | 0.286 | 0.295 | 0.480 | 0.475 | 0.308 | 0.424 | 0.353 |
| Zeroshot | 0.319 | 0.352 | 0.336 | 0.391 | 0.316 | 0.336 | 0.529 | 0.509 | 0.349 | 0.465 | 0.390 |
| 3-Shot-1run | 0.316 | 0.335 | 0.323 | 0.358 | 0.279 | 0.294 | 0.530 | 0.476 | 0.335 | 0.428 | 0.367 |
| 3-Shot | 0.400 | 0.450 | 0.432 | 0.414 | 0.344 | 0.353 | 0.584 | 0.505 | 0.374 | 0.477 | 0.433 |
| 6-Shot | 0.419 | 0.431 | 0.427 | 0.455 | 0.393 | 0.391 | 0.606 | 0.540 | 0.404 | 0.519 | 0.459 |
| Config. | Model | Context mismatch | Role mismatch | ||
| AUROC | Gap (pos neg) | AUROC | Gap (pos neg) | ||
| Zeroshot | Gemini2.5Pro | 0.665 | 0.725 | 0.821 | 1.681 |
| GPT4o-audio | 0.815 | 0.560 | 0.847 | 1.483 | |
| KimiAudio | 0.493 | -0.002 | 0.512 | 0.010 | |
| Qwen2Audio | 0.557 | 0.058 | 0.555 | 0.056 | |
| Qwen2.5Omni | 0.851 | 0.845 | 0.782 | 0.683 | |
| Training data | # train | pos/neg AUROC | Avg. Pearson |
| Full (positives both negative branches) | 0.757 | 0.432 | |
| without local-story negatives | 0.696 | 0.400 | |
| without global-story negatives | 0.597 | 0.328 | |
| Positives only (no negatives) | 0.404 | 0.249 |
| Config. | Model | Sys. P | Sys. S | Sys. K | Sample-level Avg. |
| Zeroshot | Gemini2.5Pro | -0.061 | -0.317 | -0.190 | 0.236 |
| GPT4o-audio | -0.153 | 0.040 | 0.119 | 0.233 | |
| KimiAudio | 0.233 | 0.397 | 0.214 | 0.070 | |
| Qwen2Audio † | 0.393 | -0.012 | -0.079 | – | |
| Qwen2.5Omni | -0.110 | -0.103 | -0.048 | -0.008 | |
| Fewshot | Gemini2.5Pro | 0.045 | -0.103 | -0.167 | 0.158 |
| Model | Avg. (mask) [95% CI] | Avg. (impute) |
| DRAME-Eval (Instruct-LoRA) | ||
| Gemini2.5Pro (ZS) | ||
| GPT4o-audio (ZS) | ||
| KimiAudio (ZS) | ||
| Qwen2.5Omni (ZS) | ||
| Gemini2.5Pro (FS) |
| Task | Model (config) | EN Avg | ZH Avg | Macro Avg [95% CI] |
| Realism base | DRAME-Eval (FT) | 0.611 | 0.570 | 0.591 [0.568, 0.613] |
| Gemini2.5Pro (ZS) | 0.292 | 0.431 | 0.362 [0.332, 0.389] | |
| GPT4o-audio (ZS) | 0.220 | 0.348 | 0.284 [0.253, 0.313] | |
| KimiAudio (ZS) | 0.074 | 0.060 | 0.067 [0.034, 0.100] | |
| Qwen2Audio (ZS) | 0.077 | 0.032 [ , 0.066] | ||
| Qwen2.5Omni (ZS) | 0.086 | 0.246 | 0.166 [0.134, 0.196] |
| Realism base | Real recordings | Archetype | ||||
| Dimension | ||||||
| Pitch | 0.580 | 0.584 | 0.446 | 0.452 | — | — |
| Rhythm | 0.565 | 0.577 | 0.287 | 0.301 | — | — |
| Stress | 0.547 | 0.560 | 0.385 | 0.399 | — | — |
| Accuracy | 0.562 | 0.585 | 0.182 | 0.192 | — | — |
| Intensity | 0.476 | 0.518 | 0.674 | 0.638 | — | — |
| Training data | English test | Mandarin test |
| English only | 0.441 | 0.360 |
| Mandarin only | 0.156 | 0.579 |
| Bilingual mixture | 0.353 | 0.563 |
Audio-Language Model Evaluation
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
See all 225 papers