Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
Figures & tables
Figure 1: Traditional (a) vs. context-aware turn-taking (b) in Conversational AI.
Dataset
Total
I1
I2
S1
S2
AMI
11,900
1,598
4,407
4,127
1,768
Friends
8,970
1,114
2,632
639
4,585
SPGI
99,290
21,595
28,050
17,441
32,204
Total
120,160
24,307
35,089
22,207
38,557
Table 1: Distribution of datasets in our benchmark.
Figure 2: Overview of the proposed framework.
Model
Dataset
Category-wise Accuracy (%)
Overall (%)
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
gemini-3.1 -pro
AMI
52.53 (-0.63)
39.27 (-0.46)
88.63 (+1.65)
54.14 (+2.55)
60.77 (+0.68)
59.55 (+0.52)
61.03 (+0.70)
Friends
82.52 (-1.94)
80.62 (-0.78)
86.67 (+0.00)
34.10 (+0.41)
56.43 (-0.22)
56.11 (-0.17)
60.54 (-0.36)
SPGI
79.35 (+0.14)
30.31 (-1.58)
91.32 (-0.17)
69.07 (-0.13)
64.57 (-0.48)
63.93 (-0.53)
64.45 (-0.50)
gpt-5.2
AMI
75.95 (-1.27)
55.71 (-0.69)
66.59 (-1.66)
32.48 (-1.91)
59.23 (-0.93)
59.20 (-0.95)
59.21 (-0.95)
Friends
94.17 (+2.92)
84.88 (+1.17)
60.00 (+10.00)
18.71 (-3.12)
49.00 (-0.33)
46.63 (-0.76)
55.41 (+0.00)
Table 2: Zero-shot performance of evaluated LLMs on the context-aware turn-taking benchmark. Values in parentheses denote the change when the system prompt is repeated twice in the input; the base value uses a single system prompt. Bold indicates the best overall model, while bold+italic indicates the best open-source model.
Model
Dataset
Category-wise Accuracy (%)
Overall (%)
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
gpt-oss -20b
AMI
70.89
50.00
62.56
22.93 ( ↓ 12.69)
53.70
58.72
58.74
Friends
93.20
84.11
65.00
20.37
49.89
49.10
56.84
SPGI
76.50
26.34
56.80 ( ↓ 13.38)
29.43 ( ↓ 18.93)
43.74 ( ↓ 11.85)
49.63
49.66
Mistral-7b -instruct
AMI
81.01
58.90 ( ↓ 24.43)
86.49 ( ↑ 71.80 )
61.78 ( ↑ 53.50 )
72.17 ( ↑ 22.72 )
72.05 ( ↑ 30.47 )
72.28 ( ↑ 23.35 )
Friends
98.06
52.33 ( ↓ 32.56)
76.67 ( ↑ 50.00 )
77.75 ( ↑ 59.04 )
72.73 ( ↑ 26.50 )
71.54 ( ↑ 28.24 )
71.50 ( ↑ 18.63 )
Table 3: Performance after supervised fine-tuning. Numbers in parentheses indicate the absolute change in percentage points relative to the zero-shot baseline in Table 2. Arrows are shown when the change exceeds ±10 points. Bold indicates improved performance.
Human
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
H1
84.00
74.00
86.67
29.00
66.39
64.78
64.81
H2
80.00
75.00
65.00
30.00
62.22
59.94
60.31
H3
84.00
88.00
83.33
24.00
68.33
65.80
66.13
Average
82.67
79.00
78.33
27.67
65.65
63.51
63.75
Table 4: Human evaluation with performance metrics (in %).
Training Mode Comparison
Dataset
Mode
Acc
F1 avg
Bal Acc
Friends
Decision-only
63.64
63.63
66.60
Friends
Reasoning with Decision
70.84
68.80
68.46
LoRA Rank Comparison (Reasoning Mode)
Dataset
Rank/Alpha
Acc
F1 avg
Bal Acc
Friends
r=16, α = 32
67.74
65.47
65.23
Table 5: Ablation analysis: training mode, LoRA rank, and combined-dataset generalization. All values are in % .
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
Turn-taking in multi-party spoken conversations remains a fundamental challenge for voice-based agents, particularly under dynamic floor competition and varying user expectations. We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings. The system is built on a speech large language model operating in chunk-wise streaming manner. We further introduce a reasoning-augmented variant that incorporates chain-of-thought reasoning over conversational context and the assigned role. We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles. Experiments on real-world meeting data and RolePlayConv show improved turn-taking precision by over 40% and recall by more than 70%, while substantially reducing false-positive interruptions compared to non-role-conditioned baselines.
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.