CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation

Huan Shen; Yingao Wang; Shangkun Huang; Wei Zou; Yunzhang Chen

cs.SDMar 26, 2026

CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation

Authors: Huan Shen, Yingao Wang, Shangkun Huang, Wei Zou, Yunzhang Chen

Organizations: BRVoice Team, Bairong, Inc., China

Abstract

Turn-taking modeling is fundamental to spoken dialogue systems, yet its evaluation remains fragmented and often limited to binary boundary detection under narrow interaction settings. Such protocols hinder systematic comparison and obscure model weaknesses across conversational conditions. We present CoDeTT, a context-aware decision benchmark for turn-taking evaluation. CoDeTT formulates turn-taking as a structured decision problem and constructs a multi-scenario dataset with fine-grained decision categories and controlled context variations. Under a unified evaluation protocol, we assess representative existing models and observe substantial performance disparities across decision types and interaction scenarios. CoDeTT provides a standardized benchmark for systematic and context-aware evaluation of turn-taking systems. The benchmark dataset and evaluation toolkit are available at https://yingaowang-casia.github.io/CoDeTT.github.io/.

Figures & tables

Figure 1: A conceptual comparison between traditional action-based benchmarks and the proposed CoDeTT intent-based diagnostic benchmark.

System state	Decision strategy	Scenario	Number of Samples	Operational cue (summary)
SystemSpeaking	Maintain	Backchannel	1,000(real) + 1,000(syn)	User produces short non-floor-taking feedback (e.g., “uh-huh”).
		Invalidation	1,000(syn)	Non-speech events (cough, impact, background noise bursts).
		Side-talk	1,000(syn)	Primary user speaks to another person.
		Distraction	1,000(syn)	Background speech unrelated to dialogue topic.
	Stop & Listen	Interruption	1,000(real) + 1,000(syn)	User intends to cut in.
		Dismissal	1,000(syn)	Explicit “stop talking” command directed to system.

Table 1: Hierarchical taxonomy of 14 turn-taking decision scenarios.

Figure 2: The CoDeTT dataset construction pipeline.

Audio-Input Models
		Chinese(ZH)					English(EN)
Model	History	Maintain	Stop & Listen	Takeover	Dismiss	Average	Maintain	Stop & Listen	Takeover	Dismiss	Average
Easy turn [ 7 ]	0	43.98	7.36	80.77	19.35	37.87	N/A	N/A	N/A	N/A	N/A
Smart-Turn-v3 [ 16 ]	0	N/A	N/A	74.53	27.51	51.02	N/A	N/A	58.27	44.45	51.36
FireRedChat [ 8 ]	0	N/A	N/A	86.67	6.83	46.75	N/A	N/A	88.20	42.62	65.41
NAMO-Turn [ 17 ]	0	N/A	N/A	63.07	55.91	59.49	N/A	N/A	78.73	26.12	52.43

Table 2: 4-Action ACC Results: Comparison between Chinese(ZH) and English(EN).

Chinese(ZH) Results
Model	H	Maintain						Stop & Listen					Takeover				Dismiss / Silence
Model	H	BC	Inv.	Side.	Dist.	Avg. $\uparrow$	SMR $\downarrow$	Int.	Dism.	Col.	Avg. $\uparrow$	SMR $\downarrow$	Cmp.	Cop.	Avg. $\uparrow$	SMR $\downarrow$	Inc.	Inv.	Dism.	Exc.	Side.	Avg. $\uparrow$	SMR $\downarrow$
Qwen3-Omni	0	84.40	67.60	45.09	5.60	50.67	24.65	45.40	88.20	18.84	50.81	27.42	44.54	0.00	22.27	41.22	46.04	98.80	82.63	6.97	63.60	59.61	26.73
	1	87.32	22.00	18.24	27.00	38.64	29.05	53.75	78.00	20.04	50.60	30.89	62.25	2.84	32.55	35.17	41.27	78.57	78.44	14.00	48.49	52.15	31.30
	3	82.78	9.02	8.02	12.00	27.96	19.93	62.15	67.00	17.40	48.85	34.51	61.31	0.61	30.96	36.01	45.47	69.56	84.83	22.52	43.69	53.21	30.73
	5	81.36	7.82	7.21	12.00	27.10	19.95	58.03	65.80	15.60	46.68	36.16	58.79	0.80	29.80	35.49	46.78	65.20	80.12	22.86	48.30	52.65	33.17

Table 3: Fine-grained ACC and SMR over 14 scenarios (Chinese(ZH) and English(EN)). Best per model in bold.

Figure 3: Fine-grained semantic confusion matrix of GPT-4o-audio (Chinese, 3-turn history).

Turn-Taking Prediction

Turn-taking prediction estimates when speakers will stop, continue, or enter a conversation, helping dialogue systems respond at appropriate moments.

See all 37 papers

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Aug 25, 2026Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Sesame AI · Mundo AI · Oto +4
TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue
Jul 1, 2026Hao Zhang, Thomas Thebaud, Georgi Tinchev +2Automated Evaluation Full-Duplex Spoken Dialogue Systems
Center for Language and Speech Processing, Johns Hopkins University, USA · Amazon Research, UK · Amazon, USA
Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts
Sep 15, 2026Shuofeng Zhao, Hongwei Cai, Wenke Fan +9Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Zuoyebang Education Technology, Beijing, China

Aug 25, 2026J/K move · Enter open · S save

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri+15

Sesame AI · Mundo AI · Oto+4

Spoken Dialogue Systems Multi-Turn Dialogue Evaluation Turn-Taking Prediction Turn-Taking in Dialogue

Open paper AskPDF arXiv

Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.

Abstract

Turn-Taking Prediction

Explore similar work

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts