cs.CLNov 10, 2026

AVCC: Audio-Visual Conversation Corpus for Turn-Taking Prediction with a Human Benchmark

Authors: Haotian Qi, Gabriel Skantze

Organizations: Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm

Abstract

} Predicting who will speak next, and when, is a prerequisite for conversational agents and social robots that take turns without waiting for silence. Multiparty next-speaker prediction (NSP) has been studied on corpora with per-speaker sensors, each with its own event definitions. No public benchmark covers the single camera and microphone a robot has, and no model has been compared against how well people perform the same prediction. We turn AVCC, a 31-hour audio-visual corpus of two- and three-speaker online conversation recorded with a single static camera and microphone, into a public NSP benchmark. We add a speaker-disjoint seven-video test partition with 865 hold and 865 shift events at mutual-silence gaps, word-level forced-aligned transcripts for the whole corpus, and a human reference benchmark collected at the same events. We score an audio-only Role-Relative VAP model (RoleVAP) and the multimodal MuVAP model. Labels, transcripts, per-response human data, and the scripts that reproduce every number are released.

Figures & tables

Explore similar work

CardsList
  1. MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

    Jun 15, 2026Haotian Qi, Gabriel SkantzeMultimodal GroundingAudio-Visual Understanding

  2. Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings

    Jun 16, 2026Ryo Fukuda, Takatomo Kano, Siddhant Arora +7LLM EvaluationMultimodal Large Language Models

  3. Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

    Jul 8, 2026Antonio Cano, Guillermo Pérez, Luis Merino +1Conversational AgentsHuman-Robot Interaction