AVCC: Audio-Visual Conversation Corpus for Turn-Taking Prediction with a Human Benchmark
Organizations: Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm
Abstract
} Predicting who will speak next, and when, is a prerequisite for conversational agents and social robots that take turns without waiting for silence. Multiparty next-speaker prediction (NSP) has been studied on corpora with per-speaker sensors, each with its own event definitions. No public benchmark covers the single camera and microphone a robot has, and no model has been compared against how well people perform the same prediction. We turn AVCC, a 31-hour audio-visual corpus of two- and three-speaker online conversation recorded with a single static camera and microphone, into a public NSP benchmark. We add a speaker-disjoint seven-video test partition with 865 hold and 865 shift events at mutual-silence gaps, word-level forced-aligned transcripts for the whole corpus, and a human reference benchmark collected at the same events. We score an audio-only Role-Relative VAP model (RoleVAP) and the multimodal MuVAP model. Labels, transcripts, per-response human data, and the scripts that reproduce every number are released.
Figures & tables
| Corpus | Annotations | |||||||
|---|---|---|---|---|---|---|---|---|
| Name | Spk | Hrs | Sns. | Mod. | Trans. | NSP | Hum. | Year |
| Switchboard [ 13 ] | 2 | 260 | M | A | ✓ | – | – | 1992 |
| Fisher [ 5 ] | 2 | 2000 | M | A | ✓ | – | – | 2004 |
| AMI [ 4 ] | 3–5 | 100 | M | A+V | ✓ | – | – | 2006 |
| NoXi [ 2 ] | 2 | 27 | M | A+V | ✓ | – | – | 2017 |
| AVA-AS [ 34 ] | 2+ | 38 | S | A+V | – | – | – | 2020 |
| Subset | Videos | Duration |
|---|---|---|
| 2-speaker | 18 | 16h52m |
| 3-speaker | 11 | 14h07m |
| Training | 17 | 20h39m |
| Validation | 5 | 2h41m |
| Test | 7 | 7h41m |
| Total | 29 | 31h00m |
| Balanced accuracy | Per class | |||||||||
| NSP | Hold/shift | NSP | Hold/shift | |||||||
| System | 2-spk | 3-spk | All | 2-spk | 3-spk | All | hold | shift | hold | shift |
| All events | ||||||||||
| Random Baseline | 50.0 | 33.3 | 41.4 | 50.0 | 50.0 | 50.0 | 41.4 | 41.4 | 50.0 | 50.0 |
| Human-Ind | 63.1 | 57.0 | 59.5 | 63.1 | 65.1 | 64.0 | 63.6 | 55.4 | 63.6 | 64.4 |
| RoleVAP (probe) [ 31 ] | – | – | – | 65.2 | 67.7 | 66.5 | – | – | 54.9 | 78.2 |
Turn-Taking Prediction
Turn-taking prediction estimates when speakers will stop, continue, or enter a conversation, helping dialogue systems respond at appropriate moments.
See all 42 papers