Distributional Metrics for Evaluating Spoken Conversational Systems

Shree Harsha Bokkahalli Satish; Erica Cooper; Patrícia Schmidtová; Maike Züfle; Éva Székely; Nicholas Sanders; Ondřej Klejch

eess.ASSep 21, 2026

Distributional Metrics for Evaluating Spoken Conversational Systems

Authors: Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová, Maike Züfle, Éva Székely, Nicholas Sanders, Ondřej Klejch

Organizations: KTH Royal Institute of Technology, Sweden · NICT, Japan · Charles University, Czech Republic · Karlsruhe Institute of Technology, Germany ·

Abstract

Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.

Figures & tables

Figure 1: Phone-rate CDS scores for different conversation types. CDS scores indicate distributional similarity to the anchors.

Figure 2: Listener ratings of humanness vs. phone rate and Sylber unit duration CDS for AI-partner and human-partner pooled turns per conversation. Each point is one conversation. AI scores cluster near MSC–HSC midpoint, while Behavior-SD–HSC moves AI scores closer to the synthetic distraction anchor.

Figure 3: Effect of caller count and recording duration under HSC–MSC. Subfigures (a,b) show the fraction of subsets recovering at least five of six system comparisons in each estimator’s full-recording, all-caller ordering. Subfigure (c) shows the median correlation between overall conversation CDS differences and listener preferences.

Measurement	$\rho$	95% caller range	Pairs correct
CDS : HSC–MSC
Overall (non-semantic)	$0.222$	$[0.097,0.353]$	5/6
Sylber duration	$0.503$	$[0.412,0.587]$	4/6
Semantic	$0.083$	$[-0.008,0.182]$	4/6
CDS : HSC–Behavior-SD
Overall (non-semantic)	$0.266$	$[0.166,0.368]$	3/6

Table 1: Agreement with listener preferences and rankings. Spearman’s

\rho

relates score differences to average listener preferences for conversation pairs.

Multi-Turn Dialogue Evaluation

Multi-turn dialogue evaluation measures whether AI systems remain accurate, coherent, safe, and useful as conversations evolve.

See all 203 papers

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems
Jun 30, 2026Ashish Hallur, Thomas Thebaud, Georgi Tinchev +2Voice Agent Evaluation Speech Generation Evaluation
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Amazon Inc., Seattle, WA, USA
Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction
Feb 27, 2026Xiang Li, Jiabao Gao, Sipei Lin +5Voice Agent Evaluation Spoken Dialogue Systems
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +2
People Making Conversation: Evaluating How Naturally Speech Models Converse
Oct 8, 2026Arya Bastani, Iman ModarressiSpeech Generation Evaluation Spoken Dialogue Systems
People Make Things

Jun 30, 2026J/K move · Enter open · S save

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

Ashish Hallur, Thomas Thebaud, Georgi Tinchev, Venkatesh Ravichandran+1

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Amazon Inc., Seattle, WA, USA

Voice Agent Evaluation Speech Generation Evaluation Spoken Dialogue Systems Speech Prosody

Open paper AskPDF arXiv

Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because

F_0

, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for

F_0

mean,

F_0

expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th-95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned

F_0

expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation.

Abstract

Multi-Turn Dialogue Evaluation

Explore similar work

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction

People Making Conversation: Evaluating How Naturally Speech Models Converse