} Predicting who will speak next, and when, is a prerequisite for conversational agents and social robots that take turns without waiting for silence. Multiparty next-speaker prediction (NSP) has been studied on corpora with per-speaker sensors, each with its own event definitions. No public benchmark covers the single camera and microphone a robot has, and no model has been compared against how well people perform the same prediction. We turn AVCC, a 31-hour audio-visual corpus of two- and three-speaker online conversation recorded with a single static camera and microphone, into a public NSP benchmark. We add a speaker-disjoint seven-video test partition with 865 hold and 865 shift events at mutual-silence gaps, word-level forced-aligned transcripts for the whole corpus, and a human reference benchmark collected at the same events. We score an audio-only Role-Relative VAP model (RoleVAP) and the multimodal MuVAP model. Labels, transcripts, per-response human data, and the scripts that reproduce every number are released.
Figures & tables
Corpus
Annotations
Name
Spk
Hrs
Sns.
Mod.
Trans.
NSP
Hum.
Year
Switchboard [ 13 ]
2
260
M
A
✓
–
–
1992
Fisher [ 5 ]
2
2000
M
A
✓
–
–
2004
AMI [ 4 ]
3–5
100
M
A+V
✓
–
–
2006
NoXi [ 2 ]
2
27
M
A+V
✓
–
–
2017
AVA-AS [ 34 ]
2+
38
S
A+V
–
–
–
2020
Table 1: Corpora relevant to multiparty NSP. Spk: speakers per recording. Sns.: sensors, S = one camera at most and one audio channel, M = per-speaker microphones or several cameras. Mod.: modality, A = audio, V = video. Trans.: transcripts. NSP: hold/shift or next-speaker labels. Hum.: human NSP reference. ∼ = partial.
Figure 1: The study interface. Participants watch a clip of up to 10 s with face labels ( 0 , 1 , 2 for three-speaker clips) drawn on every frame, then choose the next speaker and rate their confidence.
Figure 2: Duration of each of the 29 videos by speaker count and partition (longest 1 h 55 min, shortest 20 min, median 59 min).
Subset
Videos
Duration
2-speaker
18
16h52m
3-speaker
11
14h07m
Training
17
20h39m
Validation
5
2h41m
Test
7
7h41m
Total
29
31h00m
Table 2: AVCC corpus overview. Durations are measured from the video files and rounded to the minute.
Figure 3: A turn event at a mutual-silence gap hold. The dashed line is the prediction point, 0.1 s of padding after the previous speaker’s last word. Up to 10 s of context precede it (shaded). The rest of the gap (at most 3 s) and the following speech (hatched) are withheld.
Balanced accuracy
Per class
NSP
Hold/shift
NSP
Hold/shift
System
2-spk
3-spk
All
2-spk
3-spk
All
hold
shift
hold
shift
All events
Random Baseline
50.0
33.3
41.4
50.0
50.0
50.0
41.4
41.4
50.0
50.0
Human-Ind
63.1
57.0
59.5
63.1
65.1
64.0
63.6
55.4
63.6
64.4
RoleVAP (probe) [ 31 ]
–
–
–
65.2
67.7
66.5
–
–
54.9
78.2
Table 3: Accuracy (%) on the 1,730 AVCC test events (865 hold / 865 shift). Left: balanced accuracy by scoring and speaker count. Right: hold and shift accuracy by class. Human-Ind: mean per-participant accuracy ( N=154 ). Human-Maj: plurality label per event, ties excluded (251 under NSP scoring, 204 under hold/shift); the model rows beneath it are scored on the same retained events. Random Baseline: expected accuracy of a uniform guess over the faces (NSP) or over hold and shift, weighted by the events in each cell. Human-Maj baseline is 41.5. Scoring and the probe readout as in Sec. 3.4 .
Figure 4: Per-participant accuracy by self-reported confidence. Left: hold (blue) vs. shift (orange). Right: two-speaker (green) vs. three-speaker (purple) clips. Bars give the number of participants contributing to each point; dotted lines mark chance (50% and 33%).
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.
Haotian Qi, Gabriel Skantze
Department of Speech Music and Hearing, KTH Stockholm, Sweden
We investigate turn-taking in multimodal multi-party conversations using large language models (LLMs). We construct an evaluation framework for three tasks: addressee detection, turn-change prediction, and next speaker prediction. We compare supervised models trained for these tasks, text-based LLMs, multimodal LLMs (MM-LLMs), and human subjects. Experiments on the AMI corpus showed that LLMs outperformed supervised models and humans in next speaker prediction, despite not being trained on the target domain and without access to audio or visual information. An MM-LLM performed better than text-based LLMs on addressee detection and turn-change prediction but remained below human performance, indicating difficulty leveraging raw audio-visual signals. Ablation analyses revealed that conversational context was critical, particularly for next speaker prediction. We observed that human and LLM prediction patterns were similar, and intervals with frequent turn changes were difficult for both.
Ryo Fukuda, Takatomo Kano, Siddhant Arora +7
NTT, Inc., Japan · Language Technologies Institute, Carnegie Mellon University, USA
Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction.
Antonio Cano, Guillermo Pérez, Luis Merino +1
4i Intelligent Insights, Seville, Spain · Universidad de Sevilla, Seville, Spain · Universidad Pablo de Olavide, Seville, Spain +1