Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
Figures & tables
Figure 1: Traditional (a) vs. context-aware turn-taking (b) in Conversational AI.
Dataset
Total
I1
I2
S1
S2
AMI
11,900
1,598
4,407
4,127
1,768
Friends
8,970
1,114
2,632
639
4,585
SPGI
99,290
21,595
28,050
17,441
32,204
Total
120,160
24,307
35,089
22,207
38,557
Table 1: Distribution of datasets in our benchmark.
Figure 2: Overview of the proposed framework.
Model
Dataset
Category-wise Accuracy (%)
Overall (%)
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
gemini-3.1 -pro
AMI
52.53 (-0.63)
39.27 (-0.46)
88.63 (+1.65)
54.14 (+2.55)
60.77 (+0.68)
59.55 (+0.52)
61.03 (+0.70)
Friends
82.52 (-1.94)
80.62 (-0.78)
86.67 (+0.00)
34.10 (+0.41)
56.43 (-0.22)
56.11 (-0.17)
60.54 (-0.36)
SPGI
79.35 (+0.14)
30.31 (-1.58)
91.32 (-0.17)
69.07 (-0.13)
64.57 (-0.48)
63.93 (-0.53)
64.45 (-0.50)
gpt-5.2
AMI
75.95 (-1.27)
55.71 (-0.69)
66.59 (-1.66)
32.48 (-1.91)
59.23 (-0.93)
59.20 (-0.95)
59.21 (-0.95)
Friends
94.17 (+2.92)
84.88 (+1.17)
60.00 (+10.00)
18.71 (-3.12)
49.00 (-0.33)
46.63 (-0.76)
55.41 (+0.00)
Table 2: Zero-shot performance of evaluated LLMs on the context-aware turn-taking benchmark. Values in parentheses denote the change when the system prompt is repeated twice in the input; the base value uses a single system prompt. Bold indicates the best overall model, while bold+italic indicates the best open-source model.
Model
Dataset
Category-wise Accuracy (%)
Overall (%)
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
gpt-oss -20b
AMI
70.89
50.00
62.56
22.93 ( ↓ 12.69)
53.70
58.72
58.74
Friends
93.20
84.11
65.00
20.37
49.89
49.10
56.84
SPGI
76.50
26.34
56.80 ( ↓ 13.38)
29.43 ( ↓ 18.93)
43.74 ( ↓ 11.85)
49.63
49.66
Mistral-7b -instruct
AMI
81.01
58.90 ( ↓ 24.43)
86.49 ( ↑ 71.80 )
61.78 ( ↑ 53.50 )
72.17 ( ↑ 22.72 )
72.05 ( ↑ 30.47 )
72.28 ( ↑ 23.35 )
Friends
98.06
52.33 ( ↓ 32.56)
76.67 ( ↑ 50.00 )
77.75 ( ↑ 59.04 )
72.73 ( ↑ 26.50 )
71.54 ( ↑ 28.24 )
71.50 ( ↑ 18.63 )
Table 3: Performance after supervised fine-tuning. Numbers in parentheses indicate the absolute change in percentage points relative to the zero-shot baseline in Table 2. Arrows are shown when the change exceeds ±10 points. Bold indicates improved performance.
Human
I1
I2
S1
S2
Acc
F1 avg
Bal Acc
H1
84.00
74.00
86.67
29.00
66.39
64.78
64.81
H2
80.00
75.00
65.00
30.00
62.22
59.94
60.31
H3
84.00
88.00
83.33
24.00
68.33
65.80
66.13
Average
82.67
79.00
78.33
27.67
65.65
63.51
63.75
Table 4: Human evaluation with performance metrics (in %).
Training Mode Comparison
Dataset
Mode
Acc
F1 avg
Bal Acc
Friends
Decision-only
63.64
63.63
66.60
Friends
Reasoning with Decision
70.84
68.80
68.46
LoRA Rank Comparison (Reasoning Mode)
Dataset
Rank/Alpha
Acc
F1 avg
Bal Acc
Friends
r=16, α = 32
67.74
65.47
65.23
Table 5: Ablation analysis: training mode, LoRA rank, and combined-dataset generalization. All values are in % .