MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

Puneet Mathur; Dinesh Manocha

cs.SDOct 14, 2026

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

Authors: Puneet Mathur, Dinesh Manocha

Organizations: University of Maryland College Park, USA

Abstract

Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.

Figures & tables

Figure 1: MultiPartyTurnBench evaluation tracks. The End-of-Turn (EOT) track distinguishes genuine floor relinquishment from mid-turn pauses where the speaker retains the floor. The Interruption (INT) track distinguishes successful floor takeovers from backchannels and non-content vocalizations that leave floor ownership unchanged. Non-floor-taking interruption attempts are excluded from causal INT scoring because their ambiguous outcomes become apparent only through subsequent interaction.

Turn-taking and interruption benchmarks
Benchmarks	Multi- party	EOT event detection	INT event detection	Event-specific negatives	Timestamped event scoring	FPR budget	Controlled Speaker Comparison	Agent behavior scoring
TurnBench ( Jiang and others, 2026 )	✗	✓	✓	✓	✓	✓	✗	✗
Talking Turns ( Arora et al., 2025a )	✗	✗	✗	✗	✗	✗	✗	✓
SID-Bench ( Xia et al., 2026 )	✗	✗	✓	✓	✓	✗	✗	✗
ECHO ( Zhao et al., 2026b )	✗	✗	✗	✓	✗	✗	✗	✓
Multi-party turn-taking evaluation resources

Table 1: Comparison of turn-taking and full-duplex evaluation benchmarks. Existing benchmarks address dyadic turn-taking events, natural multi-party next-speaker predictions, or agent interaction behavior. MultiPartyTurnBench jointly evaluates multi-party EOT and INT event detection with event-specific hard negatives, timestamped causal decisions, FPR-constrained scoring, controlled speaker-count comparisons, and complementary agent behavior validation.

Event	Floor outcome	EOT	INT
Turn : yield	Released	$\checkmark$	–
hold	Retained	$\times$	–
interrupted	Taken over	–	–
Interruption	Transferred	–	$\checkmark$
Non-Floor-Taking Int.	Retained	–	Excl.
Backchannel	Retained	–	$\times$

Table 2: Floor-state event taxonomy in MultiPartyTurnBench . Events are mapped to EOT and INT evaluation targets:

\checkmark

denotes a positive event,

\times

a hard negative, and – an event not scored in that track. Excl. denotes events explicitly excluded from scoring.

Figure 2: MultiPartyTurnBench conversation generation pipeline. Five-stage pipeline transforms event-conditioned dialogue scripts into synchronized multi-speaker audio with exact floor-transition annotations. A discrete-event orchestrator resolves floor ownership and temporal dynamics before speech synthesis, followed by speaker rendering, acoustic augmentation, and channel assembly. The example illustrates overlapping speech, backchannels, successful interruptions, and their event annotations.

Figure 3: MultiPartyTurnBench evaluation protocol. Causal per-speaker predictions are matched to reference events and scored against hard-negative spans. Development-selected thresholds are frozen for test evaluation, with performance measured through recall, false-positive rate, and latency.

	End-of-Turn (EOT)			Interruption (INT)
System	Recall (%) $\uparrow$	FPR (%) $\downarrow$	p50 (ms) $\downarrow$	Recall (%) $\uparrow$	FPR (%) $\downarrow$	p50 (ms) $\downarrow$
Acoustic VAD
RMS Energy	49.8 [44.4, 57.3]	40.9	292	66.2 [61.5, 72.8]	25.0	859
WebRTC VAD ( Google WebRTC Project, )	91.8 [90.8, 92.8]	90.3	332	84.3 [80.6, 89.2]	83.8	128
Silero VAD ( Silero Team, 2024 )	97.2 [96.3, 97.8]	93.5	397	85.4 [81.4, 89.7]	89.2	119
Semantic endpointers

Table 3: EOT and INT detection on MultiPartyTurnBench . Recall is reported with 95% conversation-bootstrap CI; p50 denotes median signed latency (ms). Gray indicates operating points satisfying dev FPR

\leq 10\%

and test FPR

\leq 15\%

; cyan highlights the highest qualified recall in each task. Budget-qualified systems achieve up to 92.6% EOT recall, yet the best INT recall reaches only 56.2%. Acoustic detectors recover more events but incur substantially higher FPR, highlighting the difficulty of distinguishing genuine floor transitions from speech activity.

	START: enter the floor			STOP: yield selectively
Voice agent	Start $\uparrow$	False start $\downarrow$	bACC $\uparrow$	Yield $\uparrow$	False yield $\downarrow$	bACC $\uparrow$	$\mathbf{A}_{\mathrm{Bias}}\rightarrow 0$
GPT-Live-1 ( OpenAI, 2026 )	93.3	1.7	95.8 [90.8, 99.2]	50.0	40.6	54.7 [45.7, 63.5]	-0.09
Gemini-3.8-Live ( Google, 2026 )	88.9	0.0	94.4 [90.0, 97.8]	100.0	97.7	51.1 [50.0, 52.8]	+0.98
MiniCPM-o-4.5 ( Cui et al., 2026 )	86.7	5.6	90.6 [85.0, 95.6]	1.4	0.0	50.7 [50.0, 52.2]	-0.99
Realtime-Venus ( Zhao et al., 2026a )	81.1	8.9	86.1 [80.6, 91.1]	8.1	12.7	47.7 [43.1, 52.4]	-0.79
Step-Audio3-Realtime ( Lin et al., 2026a )	0.0	0.0	50.0 [50.0, 50.0]	9.1	15.4	46.9 [34.6, 60.0]	-0.76

Table 4: Full-duplex agents know when to start, but not when to stop. START distinguishes genuine turn endings from hold pauses; STOP distinguishes interruptions from backchannels. Four agents achieve 86.1–95.8% START balanced accuracy (bACC), yet all five remain near chance on STOP (46.9–54.7%). Gemini yields to 97.7% of backchannels, whereas MiniCPM-o yields to only 1.4% of interruptions, exposing opposite failure behaviors. Brackets denote 95% scenario-clustered bootstrap CIs; cyan highlights above-chance START performance; red marks STOP estimates whose CIs include 50%.

Panel of Figure 4 — Figure 4: Speaker-count sensitivity of EOT and INT detection. Recall by speaker count ( $N=2$ – $5$ ), with 95% conversation-bootstrap CI. EOT is broadly stable except for Kyutai, whereas INT trends differ. At five speakers, no plotted detector reaches 50% INT recall, highlighting the challenge of identifying successful floor takeovers among multiple speech claimants.

Panel of Figure 5 — Figure 5: EOT / INT recall by interaction type on MultiPartyTurnBench test set. Darker means higher recall. † denotes over the FPR budget; its recall reflects indiscriminate firing. EOT recall remains comparatively stable across conversation types. INT recall swings up to 30 points. Multi-party floor-control difficulty depends on conversational dynamics (argumentative for VAP and Mimi EP, collaborative for Kyutai, narrative for MuVAP, and task-oriented for DualTurn), not merely on speaker count.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Values recorded per candidate region
Event category	turn-yield , turn-hold , interruption , non-floor-taking int. , backchannel , noncontent , ambiguous
Floor holder (pre)	Perceived speaker ID immediately before the event ( $1,\ldots,N$ )
Floor holder (post)	Perceived speaker ID immediately after the event ( $1,\ldots,N$ )
Onset / offset	Timestamps (s) on the shared timeline
Stimulus	Noisy speaker channels (24 kHz), solo/mix controls, speaker timeline, reference transcript
Hidden information	Generator label, event log, model predictions, condition metadata

Appendix

Table 5: Human annotation schema. Annotators independently identify event categories, floor holders, and temporal boundaries using the role-relative taxonomy (Table 13 ). Noncontent corresponds to listener laughter in the released corpus.

Event Class	# Events	Consensus (%)	Fleiss’ $\kappa$	Boundary F1	Gen.–Human Match (%)
TURN / Yield	3,422	83.4	0.80	0.94	86.5
TURN / Hold	241	85.6	0.84	0.93	87.3
INTERRUPTION	367	81.4	0.78	0.89	91.2
NON-FLOOR-TAKING INT.	605	88.6	0.82	0.91	92.5
BACKCHANNEL	1,176	93.6	0.92	0.95	97.8
NONCONTENT	305	90.5	0.83	0.90	94.5

Appendix

Table 6: Human validation of MPTB floor-state events. Event counts cover all 240 conversations, with three independent annotator judgments per candidate. Consensus denotes majority categorical agreement; Fleiss’

\kappa

measures chance-corrected inter-annotator agreement; boundary F1 measures temporal agreement within 500 ms; and generator–human match is conditional on consensus (Eq. 1 ). Noncontent denotes listener laughter. Non-floor-taking interruption attempts are included in human validation but excluded from causal INT scoring. Pooled agreement statistics require event-level recomputation and are not estimated by averaging class-level coefficients.

	Corpus		Human Validation
$N$	Conv.	Events	Cons. (%)	Fleiss’ $\kappa$	Bnd. F1
2	60	1,490	89.8	0.84	0.96
3	60	1,527	88.6	0.82	0.93
4	60	1,536	84.3	0.81	0.91
5	60	1,563	82.1	0.76	0.90

Appendix

Table 7: Human agreement across speaker counts. Event totals include the six generator-defined categories in Table 6 . Categorical consensus, Fleiss’

\kappa

, and boundary F1 consistently decrease from two to five speakers.

Interaction Type	# Conv.	Hours	Turns/min	BC/min	INT/min	Overlap (%)	Mean Turn (s)
Casual	40	1.10	12.7	4.24	0.92	11.4	4.81
Task-Oriented	40	1.03	11.4	3.80	0.89	9.9	5.27
Instructional	40	1.17	7.9	3.81	0.70	10.7	7.79
Collaborative	40	1.07	11.4	3.43	1.33	11.0	5.32
Argumentative	40	1.11	9.6	2.95	1.08	10.7	6.29
Narrative	40	1.41	6.6	3.34	0.53	9.1	9.22

Appendix

Table 8: Corpus statistics by interaction type. Turns/min counts floor-holding turn segments; BC/min includes backchannels and laughter; INT/min counts successful interruptions; overlap denotes the fraction of conversation time with simultaneous vocal activity from at least two speakers. Interaction types induce distinct conversational regimes despite balanced conversation counts.

$N$	# Conv.	EOT	INT	BC/min	Overlap (%)
2	60	853	80	3.12	8.9
3	60	838	96	3.61	10.9
4	60	875	94	3.60	10.9
5	60	856	97	4.06	11.1
All	240	3,422	367	3.58	10.4

Appendix

Table 9: Corpus composition by speaker count. Each party-count condition contains 60 conversations. EOT and INT denote positive event anchors; overlap measures the percentage of conversation time containing simultaneous vocal activity.

Split	# Conv.	Hours	# Voices	EOT / INT
Dev	48	1.42	8	706 / 86
Test	192	5.47	16	2,716 / 281
Total	240	6.89	24	3,422 / 367

Appendix

Table 10: MPTB development and test partitions. The partitions use disjoint TTS voices and retain the complete speaker-count

\times

interaction-type design, with two development and eight test conversations per condition. EOT and INT denote positive event counts.

Track	Positive: floor changes	Hard negative: floor unchanged	Prediction anchor
EOT	Current floor holder completes a turn and yields the floor.	Mid-turn pause after which the same speaker resumes while retaining the floor.	End of floor-holding speech
INT	Listener begins speaking while another speaker holds the floor and successfully takes ownership.	Backchannel or non-content listener activity that does not change floor ownership.	Onset of floor-taking speech

Appendix

Table 11: MultiPartyTurnBench evaluation tracks. Each track contrasts a floor-changing event with a conversationally similar event that leaves floor ownership unchanged. This paired formulation prevents systems from succeeding through indiscriminate silence or speech-activity detection.

Split	Conv.	Pos.	Neg.	Pos.	Neg.
		EOT		INT
Dev	48	706	55	86	309
Test	192	2,716	186	281	1,172
Total	240	3,422	241	367	1,481

Appendix

Table 12: Scored denominators (positive anchors and hard-negative spans). Every test-split number in the paper is computed over these anchors. INT negatives comprise 1,176 backchannels and 305 laughter vocalizations (test: 919 and 253); the 605 non-floor-taking claims are excluded intervals.

Event	Operational meaning	Role	Count
turn –yield	Holder releases the floor to another speaker (or the conversation ends).	EOT $+$	3,422
turn –hold	Holder pauses mid-turn and resumes.	EOT $-$	241
turn –interrupted	Holder is cut off by a successful interruption.	EOT excl.	367
interruption	Listener takes the floor before the current turn naturally ends.	INT $+$	367
non-floor-taking int.	Listener attempts a claim but the holder keeps the floor.	INT excl.	605
backchannel	Listener vocalizes ( $\leq$ 3 words) without claiming the floor.	INT $-$	1,176

Appendix

Table 13: Role-relative event taxonomy used by the generator and scorer, with counts over all 240 conversations. Acoustic nuisance (ambient bed, sound effects, distant speech) is rendered in Stage 4 and carries no label.

Interaction Type	Conversation Setting	Expected Floor-Management Characteristics
Casual	Unstructured social conversation with flexible topic progression and informal exchange.	Frequent acknowledgements and backchannels, relatively short turns, topic shifts, and occasional overlap.
Task-Oriented	Goal-directed interaction in which participants jointly complete or resolve a concrete task.	Frequent clarification and confirmation, comparatively structured floor transfer, and short coordination turns.
Instructional	Asymmetric interaction in which one participant primarily explains, teaches, or guides others.	Longer floor holds by the primary speaker, question-driven transitions, and acknowledgement backchannels from listeners.
Collaborative	Shared reasoning or problem solving without a single dominant speaker.	More frequent floor exchange, cooperative overlap, short claims to the floor, and relatively balanced participation.
Argumentative	Structured disagreement or debate in which participants defend competing positions.	Longer contested turns, more competitive floor claims, interruptions, and overlapping speech.
Narrative	One participant primarily recounts an event or story while others act as listeners.	Longer sustained turns, frequent listener backchannels, and comparatively limited floor competition.

Appendix

Table 14: Interaction types used in MultiPartyTurnBench. MPTB adopts the six conversation categories used in TurnBench ( Jiang and others, 2026 ) and instantiates each across

N\in\{2,3,4,5\}

speakers. The final column summarizes the intended floor-management regime used to condition dialogue generation; realized event statistics are measured from the generated corpus rather than assumed from these descriptions.

Type	Style string inserted into the prompt	Turn (s)	BC/min	INT/min	Words/min
		Soft targets (TurnBench dyadic statistics)
Argumentative	a heated debate with frequent disagreement, interruptions, and people talking over each other	10.0	3.09	2.48	206
Casual	a relaxed casual chat with frequent overlaps, backchannels, and light interruptions	8.1	5.20	2.05	211
Collaborative	participants actively building on each other’s ideas, with overlaps and frequent floor contests	7.9	3.97	2.64	207
Instructional	one participant explaining/teaching others who ask clarifying questions; mostly clean turns, few interruptions	10.5	4.30	1.52	202
Narrative	storytelling with personal anecdotes; longer turns, listeners backchanneling, rare interruptions	10.2	4.76	1.50	198

Appendix

Table 15: Generator conditioning per interaction type. For a conversation of target duration

D

(min) with

N

speakers, the prompt requests about

\max(2N,\mathrm{round}(60D/\text{turn}))

floor-holding turns,

\mathrm{round}(\text{INT/min}\cdot D\cdot s_{N})

floor claims and

\mathrm{round}(\text{BC/min}\cdot D\cdot s_{N})

backchannels/laughter, with

s_{N}=1+0.1(N-2)

. Targets are guardrails; realized rates are reported in Tables 20 and 8 .

Panel of Figure 6 — Figure 6: One generated excerpt per interaction type (test split, verbatim script text; tags are the generator’s event labels, highlighted lines are scored anchors). Argumentative and collaborative claims cut into unfinished clauses; casual and task-oriented claims are short coordination moves; instructional claims are listener questions; narrative regions are dominated by stacked listener backchannels that must not be scored as takeovers.

Domains (20). tech support call; study group / exam prep; family logistics planning; product design standup; podcast panel / current-events discussion; board game night; job interview; travel itinerary planning; classroom discussion; relationship/interpersonal conflict; restaurant/group dining decision; sports commentary; book club; home renovation planning; medical appointment scheduling (logistics only, non-clinical); roommate household planning; startup pitch practice; book/movie recommendation swap; volunteer event coordination; group trivia night.

Tones (5; realized count). professional and business-like (57); curious and inquisitive (54); enthusiastic and energetic (53); casual and friendly (42); thoughtful and contemplative (34).

Appendix

Table 16: Scenario conditioning vocabulary. All 20 domains occur in the corpus (most frequent: volunteer event coordination, 19 conversations).

Panel of Figure 7 — Figure 7: Stage-1 system prompt , shared by all 240 conversations. The LLM proposes symbolic acts only; it never assigns timestamps or final floor outcomes.

Panel of Figure 8 — Figure 8: Stage-1 user prompt. The template is filled from the conversation specification and Table 15 ; the middle box shows the instantiated prompt of the narrative excerpt in Figure 6 . On validation failure the repair message is appended and the model is queried again.

Parameter	Value
Lead-in silence	0.3 s
Yield gap, after_pause	$\mathcal{U}(0.20,0.60)$ s
Yield gap, latched	$\mathcal{U}(-0.12,0.06)$ s
Yield gap, long_pause	$\mathcal{U}(0.80,2.00)$ s
Same-holder continuation gap	$\mathcal{U}(0.02,0.18)$ s
Mid-turn pause: probability / min. segment	0.45 / 4.0 s

Appendix

Table 17: Stage-2 orchestrator parameters. Realized timing distributions are given in Table 21 .

ID	Kokoro	G	Style	ID	Kokoro	G	Style
v01	af_alloy	F	clipped/efficient	v13	am_onyx	M	upbeat/expressive
v02	af_aoede	F	energetic/upbeat	v14	af_nicole	F	dry-humor/deadpan
v03	am_adam	M	calm/measured	v15	am_puck	M	laid-back/casual
v04	am_echo	M	forceful/assertive	v16	am_santa	M	deadpan/flat
v05	am_eric	M	warm/friendly	v17	af_nova	F	skeptical/probing
v06	am_fenrir	M	hesitant/soft-spoken	v18	bm_daniel	M	nervous/tentative

Appendix

Table 18: Voice pool and split. Shaded ids form the development pool (4F/4M); the remaining 16 form the test pool. Prefix a / b = American/British English. The style descriptor is passed to the script generator.

Component	Model	Realized (240 conv.)
Ambient bed	Procedural pink noise, scene-specific low-pass (200–2200 Hz), mains hum (100/200 Hz) for office scenes, slow ( $<$ 0.2 Hz) drift; 10 scenes	SNR $\mathcal{U}(20,38)$ dB: range 20.2–37.8, median 29.7
Discrete events	One random MUSAN sound-bible clip ( $\leq$ 4 s) per sampled event slot, 12 dB below active speech, random time; $k\sim\{0,1,1,2\}$ slots	0 / 1 / 2 slots: 49 / 124 / 67 conv.
Distant speech	2–4 faded MUSAN speech snippets of 1.5–3 s at 26 dB below active speech; $p=0.2$	51 conversations

Appendix

Table 19: Stage-4 noise model. SNR is measured against the active-speech RMS pooled over channels. Event-slot labels: keyboard clatter, notification chime, dishes, off-mic laughter, phone ring, door knock, chairs, school bell, alarm beep, paper shuffle (23–30 occurrences each). Scenes: outdoor park 33, living room TV 32, school hallway 27, office HVAC 25, studio room tone 24, cafe 22, office hum 22, dorm 20, quiet office 19, quiet apartment 16.

By speaker count
	Corpus Size		Interaction Dynamics					Benchmark Events
Condition	Dialogues	Duration (h)	Mean Turn (s)	Events/min	Backchannels/min	Interruptions/min	Overlap/dialogue (s)	EOT Pos./Neg.	INT Pos./Neg.
$N=2$	60	1.90	7.07	13.8	3.12	0.70	10.2	853/74	80/356
$N=3$	60	1.66	6.38	16.3	3.61	0.96	10.8	838/64	96/359
$N=4$	60	1.67	6.10	16.3	3.60	0.94	10.9	875/55	94/361
$N=5$	60	1.66	6.26	16.6	4.06	0.97	11.0	856/48	97/405

Appendix

Table 20: Corpus composition and turn-taking dynamics across speaker counts and interaction types. MPTB maintains balanced coverage across

N=2

–

5

while exhibiting relatively stable conversational activity as speaker count increases. Interaction types induce more distinct floor-management patterns: Casual conversations exhibit faster exchanges and shorter turns, Collaborative conversations have the highest interruption rate, and Narrative conversations sustain the longest turns with the fewest interruptions. EOT and INT

+/-

indicate positive anchors and hard-negative spans, respectively.

Quantity (s)	$n$	p10	p50	p90
Floor-holding segment	4,030	2.19	5.18	10.65
Yield gap (next onset $-$ offset)	3,182	$-$ 0.09	0.01	0.48
Hold pause span ( EOT $-$ )	241	0.49	0.83	1.12
Interruption overlap (onset $\rightarrow$ cut-off)	367	0.43	0.74	1.09
Failed claim duration	605	0.80	1.63	3.72
Backchannel duration ( INT $-$ )	1,176	0.44	0.64	1.04

Appendix

Table 21: Realized timing distributions over all 240 conversations. Conversations last 48.2–208.2 s (median 99.9 s). Negative yield gaps are latched hand-overs that begin before the previous holder finishes.

System	Checkpoint / implementation	Input and context	EOT score	INT score
RMS Energy	frame RMS (dBFS)	per channel, 20 ms frames	speech $\rightarrow$ silence offset after 0.3 s silent hangover	speech onset while another channel is voiced, confirmed after 0.1 s
WebRTC VAD	py-webrtcvad	per channel, 16 kHz, 20 ms frames; $\theta$ = aggressiveness	as RMS, on WebRTC masks	as RMS, on WebRTC masks
Silero VAD	Silero VAD	per channel, 16 kHz; $\theta$ = speech prob.	as RMS, on Silero masks	as RMS, on Silero masks
Smart Turn v2	pipecat-ai/smart-turn-v2 (wav2vec2-base, attention pooling)	per channel, 16 kHz, $\leq$ 8 s window ending at each energy-VAD offset ( $-40$ dBFS, 0.3 s hangover)	$P(\text{complete})$	$1-P(\text{complete})$ in TurnBench’s Smart Turn pipeline (Silero VAD, 512-sample chunks)
Kyutai SVAD	kyutai/stt-1b-en_fr-candle	per channel, streaming Mimi, 12.5 Hz	$P(\text{end of turn})$ (VAD head 2)	$1-P(\text{end of turn})$
Mimi Endpointer	viks66/mimi-endpointer ( oto_d1f ), two-stream LSTM over Mimi	$s$ vs. mean of rest, 24 kHz, 25 Hz	$1-P(\text{user})$	$P(\text{user})$

Appendix

Table 22: Offline system configurations. Classes c / na / i / bc / t = continuation / no-activity / interruption / backchannel / turn-shift. Every score is converted to decisions by rising edges above

\theta

with a 2 s refractory period. Mappings follow TurnBench’s reference adapters wherever one exists.

Rule	Threshold
Same-speaker segments merged (contiguous)	gap $<0.15$ s
Mid-turn pause $\rightarrow$ turn –hold ( EOT $-$ )	gap $\geq 0.20$ s
Overlap counted as “during” the holder	$\geq 0.10$ s
Backchannel ( INT $-$ ): duration / words	$\leq 1.20$ s / $\leq 3$
Interruption ( INT $+$ ): interrupter speaks at least	1.00 s
and holder falls silent within	2.50 s of onset

Appendix

Table 23: AMI floor-state mapping from manual segments to MPTB events.

	MPTB (ours)		TurnBench official		TurnBench dev slice		AMI meetings
Audio	synthetic TTS + noise		real, studio		real, studio		real, headset
Speakers	$N=2$ – $5$		$N=2$		$N=2$		$N=4$
Labels	generator script (exact)		3-annotator consensus		3-annotator consensus		derived from manual segments
Threshold	MPTB dev $\rightarrow$ test		TB dev $\rightarrow$ test		tuned in-slice		tuned in-slice
Conversations	192 test		116 test		12		15
System	EOT	INT	EOT	INT	EOT	INT	EOT	INT

Appendix

Table 24: Interruption detection remains challenging in natural multi-party speech. Recall / FPR (%) for systems evaluated on synthetic MPTB, the official TurnBench leaderboard (real dyadic speech; TurnBench’s own runs), our TurnBench dev slice, and real four-party AMI meetings. Gray : within that source’s FPR budget. Sources differ in audio, labels and operating-point selection (header rows), and both real slices tune thresholds on the evaluated data. Absolute numbers are therefore not comparable across columns but only in directions. Every learned system has lower EOT recall on real audio than on MPTB. Interruptions, however, collapse in multi-party speech on both real and synthetic audio. VAP, the one learned system evaluated on every source, detects 94.5% of interruptions in real dyadic TurnBench, but only 35.9% in real four-party AMI, even with in-slice tuning, and 11.7% on MPTB. VAP, Mimi Endpointer and WavLM Anchor are trained on TurnBench data and are in-domain there.

Panel of Figure 10 — Figure 10: Interaction type matters for interruptions, not for turn ends. Recall per conversation type for the five systems within the FPR budget on both tracks (test split, 32 conversations per type). Gray bars span each system’s worst-to-best type and black ticks mark pooled recall. EOT recall varies by at most 5.5 points across types. INT recall swings by up to 30.5 points (Kyutai: 47.0% collaborative vs. 77.5% instructional). The hardest INT type is model-specific: argumentative for VAP and Mimi EP, collaborative for Kyutai, narrative for MuVAP, and task-oriented for DualTurn.

$N$	EOT recall/FPR	INT recall/FPR
2	.93/.04	.07/.03
3	.93/.04	.10/.04
4	.92/.16	.11/.03
5	.92/.08	.18/.02

Appendix

Table 25: VAP test performance by party count. The

N{=}4

EOT FPR exceeds the nominal budget after the dev-selected threshold is frozen.

Acoustic VAD
	EOT recall / FPR (%)				INT recall / FPR (%)
System	$N{=}2$	$N{=}3$	$N{=}4$	$N{=}5$	$N{=}2$	$N{=}3$	$N{=}4$	$N{=}5$
RMS Energy	48.5 / 38.2	61.1 / 62.5	41.9 / 25.0	48.1 / 35.9	55.0 / 22.8	74.0 / 34.4	66.7 / 18.8	67.1 / 23.9
WebRTC VAD	92.1 / 85.5	93.8 / 95.8	90.4 / 86.4	90.9 / 94.9	70.0 / 83.0	82.2 / 83.0	88.9 / 82.7	93.4 / 86.1
Silero VAD	97.8 / 94.5	98.0 / 93.8	96.0 / 95.5	96.9 / 89.7	73.3 / 89.1	82.2 / 89.2	91.7 / 88.4	92.1 / 90.0
Semantic endpointers

Appendix

Table 26: Recall / FPR by party count for every system (test split, pooled operating point). The ESPnet per-channel variant runs the same checkpoint on each isolated channel instead of the mono mix.

System	$\theta$	Dev R / F	Test R / F	$\theta$	Dev R / F	Test R / F
	EOT			INT
RMS Energy	$-50$ dBFS	44.9 / 30.9	49.8 / 40.9	$-50$ dBFS	51.2 / 19.7	66.2 / 25.0
WebRTC VAD	aggr. 3	91.6 / 87.3	91.8 / 90.3	aggr. 3	83.7 / 84.5	84.3 / 83.8
Silero VAD	0.30	97.6 / 90.9	97.2 / 93.5	0.70	83.7 / 86.4	85.4 / 89.2
Smart Turn v2	0.50	78.3 / 14.5	82.5 / 11.8	0.0442	73.3 / 58.9	67.6 / 64.3
Kyutai SVAD	0.95	93.2 / 3.6	92.5 / 4.3	0.9991	65.1 / 6.1	56.2 / 6.6

Appendix

Table 27: Dev-selected operating points. R/F = recall/FPR (%).

\theta

is the score threshold (dBFS for RMS, WebRTC aggressiveness level, probability otherwise). Gray rows are in budget on both tracks. Candidate grids: RMS

\{-50,-45,\dots,-30\}

dBFS; WebRTC

\{0,1,2,3\}

; Silero

\{0.3,\dots,0.7\}

; Kyutai and Smart Turn EOT

\{0.3,\dots,0.9,0.95\}

; VAP and MuVAP

\{0.5,\dots,0.9,0.95\}

; other learned systems

\{0.05,0.10,\dots,0.95,0.97,0.98,0.99,0.995\}

plus dev-score quantiles.

	EOT (ms)			INT (ms)
System	p10	p50	p90	p10	p50	p90
RMS Energy	267	292	1653	$-38$	859	2572
WebRTC VAD	224	332	380	114	128	146
Silero VAD	359	397	435	92	119	198
Smart Turn v2	$-32$	$-18$	614	55	87	446
Kyutai SVAD	666	1829	1960	432	689	1121

Appendix

Table 28: Signed latency

t_{\text{pred}}-t_{\text{gold}}

of matched detections on the test split. ESPnet INT percentiles rest on 3 matched detections.

System $-$ VAP		$\Delta$ Recall	$\Delta$ FPR
EOT	Kyutai SVAD	$-0.1$ [ $-1.5$ , 1.3]	$-3.2$ [ $-7.7$ , 1.1]
	Mimi EP	$-2.1$ [ $-2.9$ , $-1.3$ ]	$-3.8$ [ $-8.1$ , 0.5]
	DualTurn	$-3.2$ [ $-4.0$ , $-2.3$ ]	$+3.2$ [ $-1.6$ , 8.2]
	MuVAP	$-4.5$ [ $-5.6$ , $-3.4$ ]	$+2.7$ [ $-1.8$ , 7.2]
	WavLM	$-2.2$ [ $-3.0$ , $-1.4$ ]	$+15.1$ [7.0, 23.1]
	Turn-1-Mini	$-5.7$ [ $-6.9$ , $-4.5$ ]	$+17.7$ [10.2, 25.5]

Appendix

Table 29: Paired differences to VAP (points, 95% CI;

B=10{,}000

conversation-cluster resamples of the 192 test conversations).

	EOT recall by SNR (dB)				INT recall by SNR (dB)				EOT FPR, distant speech		INT FPR, distant speech
System	20–25	25–30	30–35	35–40	20–25	25–30	30–35	35–40	absent	present	absent	present
Kyutai SVAD	91.9	94.5	92.8	89.6	52.4	56.7	59.3	55.3	3.9	6.1	6.8	5.6
Mimi Endpointer	91.2	91.5	89.2	89.8	12.7	13.3	16.0	12.8	4.6	0.0	3.6	4.8
VAP	92.7	92.5	92.4	92.9	12.7	10.0	13.6	10.6	7.8	6.1	2.8	2.8
MuVAP	88.8	88.7	87.2	87.6	25.4	33.3	40.7	38.3	10.5	9.1	9.3	6.0
DualTurn	89.7	89.6	89.5	88.8	34.9	34.4	39.5	40.4	10.5	12.1	8.8	6.0

Appendix

Table 30: Robustness slices (test split, %, pooled operating points). Signal to Noise Ratio (SNR) bins refer to the ambient bed.

	EOT FPR on holds (%)		Share of
System	contested ( $n{=}28$ )	uncontested ( $n{=}158$ )	false alarms
Kyutai SVAD	7.1	3.8	2 / 8
Mimi Endpointer	25.0	0.0	7 / 7
VAP	39.3	1.9	11 / 14
MuVAP	57.1	1.9	16 / 19
DualTurn	42.9	5.1	12 / 20

Appendix

Table 31: Contested holds. A hold is contested when another speaker’s non-floor-taking claim overlaps the pause span. Last column: contested false alarms / all EOT false alarms on the test split.

	EOT (pre-pause / pre-yield clause ending)					INT FPR on negatives					INT recall by claim length
	FPR on holds			Recall on yields		by word count			by type		words
System	“.”	“?”	“—”	“.”	“?”	1	2–3	4+	BC	Laugh	1–3	4–8	9+
Kyutai SVAD	1.8	9.1	7.7	92.2	93.6	0.2	5.7	72.3	2.4	21.7	37.2	55.6	62.6
Mimi Endpointer	2.7	9.1	2.6	88.9	95.0	1.5	4.5	20.0	2.6	8.3	23.3	12.1	12.2
VAP	9.9	9.1	0.0	90.8	96.9	1.2	3.7	10.8	2.6	3.6	9.3	14.1	10.8
MuVAP	9.9	15.2	7.7	86.2	92.9	6.0	9.4	26.2	7.3	13.4	27.9	37.4	34.5

Appendix

Table 32: Error anatomy (test split, %). Clause ending is the final character of the script utterance before the hold pause or yield (“—” marks a self-interrupted clause). Word counts are taken from the script text of the listener vocalization or the interrupting claim. † Smart Turn EOT from the re-run on released audio at the frozen threshold.

Constant	Value
Scenario families $\times$ size	4 $\times$ 30 (balanced $N{=}2,3,4$ )
Max. scenarios per source conversation	2
Selection seed	20260929
Repeats per scenario and agent	3
Stream audio	mono mix of all participants’ noisy channels, 24 kHz; the agent joins as an additional party
START context before target turn onset	20 s

Appendix

Table 33: Full-duplex protocol constants. All constants are identical across agents; no protocol parameter was adapted to any model.

Panel of Figure 11 — Figure 11: Agent-facing text. The instruction never mentions turn-taking evaluation, interruptions or measurement. Questions elicit a 4–8 s answer that requires no reasoning, so the response content cannot confound the stop decision; only speech onset and cessation are scored.

	true EOT (START $+$ )				hold pause (START $-$ )				true INT (STOP $+$ )				backchannel (STOP $-$ )
Model	$N{=}2$	3	4	Cons.	$N{=}2$	3	4	Cons.	$N{=}2$	3	4	Cons.	$N{=}2$	3	4	Cons.
Gemini-3.8-Live	.93	.80	.93	.77	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	.07	.00	.00	.93
GPT-Live-1	.95	.90	.95	.93	.95	1.00	1.00	.97	.35	.50	.65	.40	.48	.60	.75	.60
MiniCPM-o-4.5	.97	.77	.87	.77	.87	1.00	.97	.87	.04	.00	.00	1.00	1.00	1.00	1.00	1.00
Realtime-Venus	.70	.87	.87	.60	.93	.90	.90	.77	.13	.04	.07	.77	.85	.90	.88	.67
Step-Audio-3	.00	.00	.00	1.00	1.00	1.00	1.00	1.00	.00	.20	.00	1.00	1.00	.80	.80	.50

Appendix

Table 34: Behavioral accuracy by party count and run-to-run consistency (Cons.). Accuracy is over valid trials in each

N

(10 scenarios

\times

up to 3 repeats). Step-Audio-3 STOP cells rest on 2–5 valid trials each.

Model	Family	p10	p50	p90	$n$
Gemini	true EOT	1141	2036	2258	80
Gemini	true INT	200	230	263	90
GPT-Live-1	true EOT	980	1313	1796	56
GPT-Live-1	true INT	1094	1491	2204	60
MiniCPM-o	true EOT	442	870	1223	78
MiniCPM-o	true INT	2000	8600	11000	74

Appendix

Table 35: Selected response latencies (ms) for valid trials with measurable responses.

Model	INT barge-in	BC barge-in	Resume $<2$ s
Gemini	1.000	.955	.022
GPT-Live-1	.000	.000	.617
MiniCPM-o	.000	.000	.122
Venus	.000	.000	.243
Step-Audio-3	.000	.000	.091

Appendix

Table 36: Fraction of STOP events associated with provider barge-in/cancel events and short resumption rate after true interruptions.

	Detector correct		Detector wrong
Probe	beh. ok	beh. wrong	beh. ok	beh. wrong
START	487	122	91	20
STOP	160	114	98	134

Appendix

Table 37: Event-level contingency between offline detector correctness and behavioral correctness on the same scenario (valid trials of the four agents with an offline pass).

Panel of Figure 12 — Figure 12: Interruption track: qualitative cases. (a)–(c) positives, (d) a hard negative. Decisions use each system’s frozen test operating point.

Panel of Figure 13 — Figure 13: Hard negatives: qualitative cases. (a) INT negative; (b)–(c) EOT holds. Learned endpointers reject incomplete clauses but confuse a contested pause with a yield.

Panel of Figure 14 — Figure 14: Full-duplex agents: qualitative cases. Each mark is one repeat of the same scenario. Grey numbers are the time from injection to the end of the agent’s ongoing utterance (ms unless stated).

Component	Released artifact / setting	Specified in
Scenario sampling	240 specifications (N, type, domain, tone, duration, voices, noise profile, seed); 10 per $(N,\text{type})$ cell	App. C.3 , Tab. 16
Script generation	gpt-4.1 , temperature 0.9, 8,000 tokens, $\leq$ 3 attempts; prompts verbatim; validated scripts released	Figs. 7 , 8 , Tab. 15
Orchestration	All timing distributions; per-conversation seed; render sidecar with every clip placement	Tab. 17
Speech synthesis	Kokoro-82M, 24 kHz, speed 1.0, voice binding, content-hashed clip cache	Tab. 18
Noise rendering	Scene, SNR, event slots, distant-speech placements and MUSAN clip ids per conversation; noise seeds	Tab. 19
Gold and anchors	Event logs; deterministic anchor derivation and scorer	App. B.1 , B.2

Appendix

Table 38: Reproducibility map. All non-generative stages are deterministic given the released seeds.

Turn-Taking in Dialogue

Turn-taking in dialogue studies how conversational systems decide when to speak, listen, yield, or interrupt across changing contexts.

See all 38 papers

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Aug 25, 2026Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Sesame AI · Mundo AI · Oto +4
Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts
Sep 15, 2026Shuofeng Zhao, Hongwei Cai, Wenke Fan +9Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Zuoyebang Education Technology, Beijing, China
Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue
Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Peking University · Renmin University of China · Tsinghua University +3

Aug 25, 2026J/K move · Enter open · S save

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri+15

Sesame AI · Mundo AI · Oto+4

Spoken Dialogue Systems Multi-Turn Dialogue Evaluation Turn-Taking Prediction Turn-Taking in Dialogue

Open paper AskPDF arXiv

Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.

Abstract

Turn-Taking in Dialogue

Explore similar work

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue