VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model

Yang Xiao

eess.ASOct 15, 2026

VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model

Authors: Yang Xiao

Organizations: The University of Melbourne

Abstract

A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.

Figures & tables

Figure 1: VoxTurn at a glance. Top: eot-bench [ 1 ] , the share of mid-turn (hold) silences in which a detector takes the turn when it must answer within a 300 ms mean latency budget, in English (left) and averaged over 14 languages (right; only systems evaluated in all 14 languages). Bottom left: TurnBench [ 2 ] test-set end-of-turn recall, scored by the organisers; VoxTurn is applied zero-shot. Bottom right: share of 150 English turns that the agent would cut off, replayed through each framework’s VAD and endpointing with its default turn detector or with VoxTurn. Baseline numbers are official benchmark results or model cards.

Panel of Figure 2 — Figure 2: Architecture. (a) VoxTurn reads the last 8 s of user audio (and optional text) once and answers several questions in one forward pass; a linear head scores the hidden state at each option’s last token. (b) Attention mask (filled: the query row attends to the key column) and position ids: each question block sees only the prefix and itself, at the positions it would have if asked alone.

Question	What it asks	Options	Supervised by
eot	Has the user finished their turn?	yes / no	Smart Turn, pause sources, SemanticVAD
resume	When will the user go on?	within 0.5 s / later / not in this turn	pause timing (auxiliary)
state4	Easy Turn dialogue state	complete / incomplete / backchannel / wait	Easy Turn
intent	Is speech during the agent’s turn an interruption?	interruption / backchannel	SemanticVAD
invalid	Is the sound not valid input (cough, laughter, noise, music)?	yes / no	MUSAN, VocalSound; real speech as negatives

Table 1: The five questions VoxTurn answers, with their option sets and the training sources that supervise them (Table 2 ).

Source	Type	Lang.	Question	Size	Share
Smart Turn v3.2 [ 7 ]	segmented utterances	23 langs	eot	266k	18%
Easy Turn [ 11 ]	segmented utterances	zh	state4	1.19M	25%
MagicData-RAMC [ 21 ]	conversational pauses	zh	eot	181k	11%
SpokenWOZ [ 22 ]	conversational pauses (phone)	en	eot	95k	14%
Seamless Interaction [ 23 ]	conversational pauses (face-to-face)	en	eot	179k	14%
SemanticVAD [ 16 ]	dialogue text	en, zh	eot, intent	15k	4%

Table 2: Training mixture: source, language, main question, size (clips, or pauses for the conversational sources) and sampling share. Each pause sample uses a fresh random decision time. Shaded: real conversational pauses.

Benchmark	What is tested	Languages	Metric
eot-bench [ 1 ]	hold vs. end-of-turn silences in real user turns, scored every 100 ms into each silence	14	false cut-offs at a 300 / 600 ms latency budget; latency at 5 / 10% cut-offs; AUC at 200 ms
TurnBench [ 2 ]	end-of-turn (EOT) and interruption (INT) events in 30 h of two-party conversation	en	recall at an FPR, median latency
Easy Turn test [ 11 ]	four dialogue states from audio	zh	accuracy
TEN test [ 15 ]	finished vs. unfinished turns from text	en, zh	per-class recall
OpenSLR 99 [ 18 ]	nonverbal vocalisations (16 categories) vs. 1,786 valid speech clips	–	flagged / falsely rejected
Framework replay	150 eot-bench English turns through Pipecat [ 26 ] and LiveKit Agents [ 27 ]	en	turns cut off, mean response delay

Table 3: Evaluation suite. “Turns” are single-speaker user turns; “FPR” is the false-positive rate. No evaluation audio is used for training.

			English				Chinese	14 lang.
			cut-offs (%) at		latency (ms) at
System	Open	Params	300 ms	600 ms	5%	10%	@300	mean
LiveKit v1 [ 6 ]	–	–	9.9	4.5	543	295	20.6	14.0
turn-1 [ 10 ]	$\checkmark$	2.8B	10.6	4.5	553	322	–	–
VoxTurn v1	$\checkmark$	0.8B	11.5	6.0	669	335	22.9	19.9
Baton [ 8 ]	–	–	12.3	4.8	577	350	–	–

Table 4: eot-bench. English false cut-offs (%) at a mean-latency budget and mean latency (ms) at a cut-off budget; Chinese and the 14-language mean at 300 ms. Lower is better everywhere. –: not supported or not reported. Sorted by English @300.

Figure 3: eot-bench English. (a) Lowest false cut-off rate at each mean-latency budget, from the eot-harness policy sweep; the shaded area is under the VoxTurn v1 curve. VoxTurn r3 is our earlier Whisper-based model (Table 5 ). (b) Mean latency each system needs to keep false cut-offs at 10% (open circle) and 5% (filled circle); shorter is better.

Figure 4: eot-bench, 14 languages : false cut-offs (%) at a 300 ms latency budget for every system evaluated in all 14 languages, rows sorted by the mean. Outlined: the best open model in each column. Only English and Chinese (purple) have conversational-pause training data.

	English			Chinese	14 lang.
Configuration	@300 (%) $\downarrow$	@5% (ms) $\downarrow$	AUC $\uparrow$	@300 (%) $\downarrow$	mean (%) $\downarrow$
r0: segmented utterances, RAMC at 0.2 s	26.2	921	.896	42.1	33.0
r1: + RAMC decision time within pause	22.6	936	.901	32.7	–
r2: + SpokenWOZ pauses	16.3	850	.944	29.1	–
r3: + Seamless pauses	16.5	808	.940	26.5	20.0
4 $\times$ more training (1.2M)	15.9	833	.948	24.8	21.0

Table 5: The path to v1 (eot-bench, 300k training samples each). Rows r0–r3 use Whisper-small [ 29 ] with Qwen3.5-0.8B; r3 † is r3 re-run on the L40S, the control for the rows below it.

Figure 5: The path to v1 as a waterfall. Floating bars show the change in false cut-offs at 300 ms from the step before; the number above each gives the change (green: fewer cut-offs, orange: more) and the number beside it the new value. Grey bars are absolute values. r3 † re-runs r3 on different hardware and restarts the chain. Data steps (shaded left) bring the largest gains in both languages.

Figure 6: Other questions answered by the same model and pass. (a) Easy Turn test set, four dialogue states from audio alone. (b) TEN text test set, recall of finished and unfinished turns. (c) Invalid-input detection on OpenSLR 99 nonverbal vocalisations and on valid speech.

	EOT			INT
System	R $\uparrow$	FPR	ms $\downarrow$	R $\uparrow$	FPR	ms $\downarrow$
turn-1-mini † [ 30 ]	.975	.075	595	.973	.048	551
Vox Maru v1 †	.960	.072	548	.964	.085	368
Ooma †	.951	.082	388	.968	.076	567
Tavus Sparrow-2 †	.930	.084	615	.958	.126	959
SPX-AI MicroCF †	.906	.079	678	.984	.073	747

Table 6: TurnBench. Recall (R) at a false-positive rate (FPR) and median latency (ms). Top: test set, scored by the organisers (October 2026;

\dagger

: external submission); ranks are among all submissions. Bottom: dev (38 conversations, FPR

\leq 0.1

) and EOT policy ablations re-searched on dev.

Figure 7: TurnBench test set : recall against median latency (upper left is better). Systems operate at different false-positive rates (Table 6 ). Smart Turn v3’s INT recall (0.107) is off the scale of panel (b).

Figure 8: TurnBench dev, EOT decision policy. Recall at FPR

\leq 0.1

as the policy is built up (top three) and when its parts are removed from the re-searched policy (bottom three); the right column gives the median latency.

Call	ms
1 question, unmerged eager model	$\approx$ 38
1 question, serving mode	8.3
4 questions in 4 passes	33.2
4 questions in one pass	9.0
4 clips in one batch, 1 question each	16.0

Table 7: Efficiency and framework replay. Left: latency per call (L40S, bf16, 8 s audio, end to end, p50). Right: 150 eot-bench English turns through each framework’s Silero VAD and endpointing (LiveKit: VAD minimum silence 0.3 s); VoxTurn re-checks every 0.25 s. A turn is cut off if any of its pauses ends it; delay is the mean from the final silence.

Figure 9: Efficiency and deployment. (a) Latency per call; answering four questions in one pass costs about as much as one question. (b, c) Framework replay of 150 English eot-bench turns: share of turns the agent would cut off and mean delay from the start of the final silence, with each framework’s default local detector and with VoxTurn re-checked every 0.25 s.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	ar	de	en	es	fr	hi	id	it	ja	ko	nl	pt	tr	zh	mean
False cut-offs at 300 ms (%)
LiveKit v1 (closed)	16.2	11.8	9.9	15.5	12.7	10.5	22.3	16.9	5.6	5.5	12.5	20.1	15.7	20.6	14.0
VoxTurn v1	21.1	15.4	11.5	21.4	13.4	37.5	30.8	21.0	11.1	12.8	15.9	24.1	20.1	22.9	19.9
LiveKit v1-mini	24.4	24.8	27.8	26.0	22.4	27.4	43.5	33.7	23.5	12.3	32.9	36.9	35.3	27.5	28.4
ultraVAD	45.5	35.5	27.7	41.6	34.4	39.1	45.0	40.4	12.0	33.8	42.3	42.8	44.9	38.8	37.4
Smart Turn v3.2	39.2	35.5	35.2	38.7	38.9	42.6	46.2	44.1	34.9	30.1	42.3	42.7	43.3	42.9	39.7

Appendix

Table 8: eot-bench per language: false cut-offs (%) at a 300 ms mean-latency budget (top) and mean latency (ms) at 5% false cut-offs (bottom); lower is better. Bold: best open model per column.

Setting	Value
Backbone	Qwen3-ASR-0.6B (AuT encoder frozen, Qwen3-0.6B language model)
Trained parameters	projector, LoRA, linear head (11.8M)
LoRA	rank 16, $\alpha=32$ , every linear layer of the language model
Audio context	8 s, left-padded with silence; 13 tokens/s (104 tokens)
Training samples	300k
Batch	128 (4 micro-batches of 32)

Appendix

Table 9: VoxTurn v1 training configuration.

Turn-Taking Prediction

Turn-taking prediction estimates when speakers will stop, continue, or enter a conversation, helping dialogue systems respond at appropriate moments.

See all 37 papers

Sep 10, 2026

Less can be More: What Aspects of Speech Drive End-of-Turn Detection

In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of different modalities remains unclear. We present a controlled ablation of acoustic, prosodic, and semantic signals for streaming end of turn detection using a lightweight trimodal classifier. Under identical training conditions, the acoustic prosodic combination achieves the best balance of accuracy and latency, achieving utterance F1 of 0.93 with 7.8% false alarms at 400ms median latency. Adding text increases premature detections without improving performance. Feature space analysis confirms that prosodic features have the strongest class separability, while text representations overlap substantially. These findings suggest that turn-taking is primarily conveyed through intonation and silence patterns rather than semantic completeness, enabling faster and more reliable systems without expensive text inference.

Rini Sharon, Manickavela A, Kadri Hacioglu +1

Uniphore, India

Speech Prosody Turn-Taking Prediction Turn-Taking in Dialogue

Ask PDF

Sep 24, 2026

BanglaTurn: A Benchmark and Whisper-Based Model for End-of-Turn Detection in Bangla Speech

This paper presents BanglaTurn, a corpus for end-of-turn detection in Bangla conversational speech, and a model trained on it. The corpus holds 35,374 samples of 3 to 15 s of podcast speech, labelled for turn state by combining speaker diarization with an LLM pass, with every label then checked by a human annotator. The model pairs a Whisper encoder with task-specific classification heads. On a class-balanced test set drawn from a held-out podcast, it reaches 84.33% accuracy (95% CI 80.3 to 88.1) against 69.28% for the Smart-Turn v3 baseline, and lowers the false negative rate from 51.57% to 7.55% at the cost of a higher false positive rate. We report what encoder layer fine-tuning, multi-scale pooling and INT8 quantization each contribute, and latency stays within 165 to 191 ms end to end on CPU.

Mizbaul Haque Maruf

Vivasoft Limited, Dhaka, Bangladesh

Turn-Taking Prediction

Ask PDF

Mar 9, 2026

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.

Shangeth Rajaa

Anyreach AI

Speech Language Models Turn-Taking Prediction Speech Foundation Models Full-Duplex Spoken Dialogue Systems

Ask PDF

Abstract

Turn-Taking Prediction

Explore similar work

Less can be More: What Aspects of Speech Drive End-of-Turn Detection

BanglaTurn: A Benchmark and Whisper-Based Model for End-of-Turn Detection in Bangla Speech

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining