Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness
Authors: Mark Dourado, Karim Haddad, Henrik G. Hassager, Stefania Serafin
Organizations: Research & Exploration, GN Store Nord, Ballerup, Denmark · Department of Architecture, Design and Media Technology, Aalborg University, Copenhagen, Denmark · Department of Engineering Technology and Didactics, Technical University of Denmark, Kongens Lyngby, Denmark
Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.
Figures & tables
Figure 1: CSC representation of a 4-talker segment. For each participant Pn: Top row shows VAD per participant with IPU on/offsets and output level in Phon. Middle row shows labels, such as within-turn overlaps, gaps, and floor-times as well as detected floor changes (for FTO). Bottom: discretised gaze states showing who Pn is looking at, and relative gaze angle from centre-of-mass. The CSC view operationalises the inputs used for feature extraction and modelling.
Figure 2: Representative bigram chord diagrams for one conversation: P1 (left) and P2–P4 (right). The diagrams illustrate the seven-state gaze space and transition frequencies between gaze targets. Chord thickness indicates transition frequency, while the relative gap size at the ring edge distinguishes the origin and destination of each bigram.
Figure 3: Paired fold-level AUC (AUROC) for gaze-only and multimodal models. Each line represents one cross-validation fold (LOGO/LOCO). The multimodal model consistently outperformed the gaze-only model ( Δ AUC mean = 0.18; p<0.001 ).
Figure 4: Added value of gaze and phonetic features in predicting floor-transfer outcomes. Left: Δ PR–AUC; Right: Δ ROC–AUC. Black squares show mean ± 95% CI across folds.
Figure 5: Receiver operating characteristic (ROC) curves by acoustic condition for gaze-only and multimodal models. Colors indicate acoustic conditions (Quiet, 55 dB, 65 dB, and Varying). Bold lines show pooled performance across outer folds, while faint ”ghost” lines represent individual folds. Multimodal models consistently achieved higher AUCs (0.74–0.77) than gaze-only (0.56–0.59), demonstrating robust generalisation across noise conditions.
Figure 6: Precision–recall (PR) curves by acoustic condition for gaze-only and multimodal models. Colors mark acoustic conditions; bold lines show the pooled performance and faint lines individual folds. Dashed horizontal lines denote the class baseline precision. Gaze features alone provided consistent predictive structure, while adding phon intensity yielded earlier recall gains and higher overall precision across all conditions.
Figure 7: Per-condition performance metrics (Precision, Recall, Accuracy, and F1) for gaze-only and multimodal models, separated by outcome type (gap or overlap). Bars show mean scores across outer folds; error bars indicate standard error. Metrics are reported at the best-F1 threshold per fold.
Figure 8: Fold-level odds ratios (per 1 SD) for stable predictors in the gaze-only (a) and multimodal (b) models. Densities indicate the distribution of effects across folds; values above 1 favour overlap , values below 1 favour gap .
Figure 9: Feature influence for the gaze-only model shown jointly for overlap -positive (blue) and gap -positive (green) fits. Each marker shows the median odds ratio (OR, per 1 SD) with interquartile range across folds, plotted against selection frequency. Values above 1 indicate overlap bias; values below 1 indicate gap bias. Horizontal jitter has been added for visual clarity.
Figure 10: Direction and magnitude of strong predictors across modalities. Bars show median odds ratios per 1 SD with interquartile ranges across folds. Phon intensity decreases overlap likelihood (floor-management), whereas gaze-related features increase it (transition relevance place).