Breakdown of Conversational Involvement: A Joint Multi-Modal Analysis of Conversation Floors and F-formation Detection with Multiple Features

Zonghuan Li; Chirag Raman; Hayley Hung

cs.CLOct 13, 2026

Breakdown of Conversational Involvement: A Joint Multi-Modal Analysis of Conversation Floors and F-formation Detection with Multiple Features

Authors: Zonghuan Li, Chirag Raman, Hayley Hung

Organizations: Intelligent Systems, Delft University of Technology EEMCS, 2628 XE Delft, Zuid-Holland, The Netherlands

Abstract

Interactions happen universally and spontaneously in daily lives. Studying involvement in interactions helps us analyze who can influence who and has a wide potential in applications. Specifically, we consider conversational involvement in free-standing conversations. We study two operationalizations of an FCG: F-formations and conversation floors, and propose that current assumption that F-formations capture single conversations is insufficient. To illustrate how different implementations could impact the detected F-formations, we employ three analyses. First, we propose that speaking status can be used for estimating conversation floors, and leveraged it to analyze floors within F-formations. Second, we analyze spatial characteristics of the detection results by applying different feature sets on chosen baselines, and find that groups resulting from head features are more likely to be subsumed by groups detected using other features. Finally, we do a joint analysis which combines the spatial features and floor patterns in the conversation. We observe that certain pipelines can detect groups with single floors from upper body features, while retaining F-formations with lower body features. We conclude that a more detailed theory and framework is needed to holistically demonstrate conversational involvement within interactions.

Figures & tables

Fig. 1: Depiction of a single F-formation with multiple conversation floors. The darker green regions within dotted lines represent distinct simultaneous conversation floors. Most works representing a conversing group as an F-formation make the simplifying assumption that a single conversation occurs within an F-formation with a joint focus of attention for all members.

Fig. 2: Illustration of gaps, within-overlaps, and between-overlaps for two speakers (S1 and S2) within the same floor. The scheme was originally proposed by Heldner and Edlund [ 20 ] and adopted by Levinson and Torreira in their analysis [ 26 ] .

Fig. 3: Illustration of the experimental design. We start with a dataset with speaking status and F-formation annotations, as well as body spatial information of human bodies. Temporal analysis: we analyze simultaneous speakers within an F-formation to demonstrate there might be more than one speaker; Spatial analysis: We run selected F-formation detection pipelines and analyze spatial patterns between them; Joint analysis: we combine the spatial features with conversation floors to study the relationship between detection results and actual physically embodied floors.

Fig. 4: Illustration of spatial measures

S_{H}

and

S_{F}

. First, we identify the composition of annotation

A_{i}

, detection results

G_{\alpha},G_{\beta}

, and the relative detection

\mathcal{R}_{\alpha}(A_{i})

. For SH, we first calculate

s_{u}

and

s_{p}

, then take the maximum to get

S_{H}

. For SF, it is

1

when there are at least 2 subsets of

g_{\alpha}

G_{\beta}

with at least 2 members, and

0

otherwise.

Fig. 5: Illustration of joint analysis pipeline. First we split the duration of an annotated F-formation into tiles of length

t_{0}

, then compute speaker within each tile. For each frame in each tile where a split event exists, we compute

M_{1}

and

M_{2}

M_{1}

calculates whether the larger group contain at least 2 speakers, and

M_{2}

calculates the proportion of subgroups that contain exactly one speaker. Together, they measure how much the detected split events align with the characteristic that multiple floors exist within one F-formation.

Fig. 6: Simultaneous speakers per cardinality of detection using annotations. Each bar is average speaker number with standard deviation, with respect to a specific window duration. For groups with cardinality 4, more than 1 speaker is observed for 1-5 seconds of window. For groups with cardinality 5, no average speaker number is above 1. For cardinality 6-7, although observed with more than 1 speaker for windows duration up to 2 and 9 seconds respectively, the number of samples also drops substantially, resulting in no standard deviation.

	Coeff ( $\beta$ )	Std Err	$z$	$P>z$
Intercept	-0.7164	0.564	-1.270	0.204
Turn-duration	0.2424	0.076	3.173	0.002
Cardinality	0.2726	0.121	2.244	0.025
Turn-duration:Cardinality	-0.0754	0.018	-4.298	0.000

TABLE I: Generalized model regression results

Cardinality pairs	Intercept ( $\beta_{0}$ )	d( $\beta_{1}$ )	c( $\beta_{2}$ )	d:c( $\beta_{3}$ )
4-5	0.831	0.0005	0.879	0.0003 $*$
4-6	0.426	0.011	0.099	0.002
4-7	0.295	0.592	0.017	0.086
5-6	0.463	0.042	0.285	0.109
5-7	0.387	0.001	0.113	0.006
6-7	0.918	0.048	0.600	0.104

TABLE II: Nominal P-values for six post-hoc GLM regression comparisons

Fig. 7: Number of F-formations at different speaking turn durations.

Panel of Figure 10 — Fig. 8: Spatial analysis results for both pipelines. All figures are interpreted from row to column. For instance, a cell located at shoulder row and hip row of SH is interpreted as $S_{H}(\text{shoulder},\text{hip})$ . 8a : SH and SF for GTCG pipeline. SH is at least $0.96$ which shows high agreement over groupinf patterns. SF shows that it takes highest for the $(\text{shoulder},\text{head})$ pair and is substantially higher when $\beta$ takes head features. 8b : SH and SF for GCFF pipeline. SH score is $0.99$ for every pair of different $(\alpha,\beta)$ , showing high agreement over groupings. SF shows clear asymmetry: $S_{F}(\alpha,\beta)$ is much larger than the opposite when $\alpha$ is physically lower than $\beta$ on the body.

Panel of Figure 11 — Fig. 9: $M_{1}$ and $M_{2}$ for each clue pair for both pipelines. Within the labels, $\alpha\rightarrow\beta$ refers to $M(\alpha,\beta)$ . $M_{1}$ calculates how often there are at least 2 speakers in the larger group when a split event happens, and $M_{2}$ calculates how frequently each subgroup contain exactly one speaker when $M_{1}$ holds. The data bars are grouped in 4 sets: (LBF, UBF), (UBF, LBF), within UBF, and within LBF. 9a : $M_{1}$ and $M_{2}$ for GTCG. 9b : $M_{1}$ and $M_{2}$ for GCFF.

Zonghuan Li received BEng degree in Electrical Engineering from Tsinghua University in China, 2016, and master’s degree in Electrical Engineering from University of California, San Diego in United States, 2023. He is currently a PhD student under the Socially Perceptive Computing Lab, TU Delft, Netherlands since 2024. His research interests include conversational involvement analysis with multimodal signals, multimodal large language model applications and evaluation.

Table 12

Chirag Raman received his master’s degree in entertainment technology from Carnegie Mellon University, USA in 2013 and the Ph.D in Computer Science from TU Delft in 2023. He is currently an Assistant Professor at TU Delft where he leads the Tapri Lab within the Department of Intelligent Systems. His research interests include gnerative modeling, socially intelligent embodied agents, with insights from cognitive science and social psychology.

Table 13

Hayley Hung (Member, IEEE) received the PhD degree in computer vision from the Queen Mary University of London in 2007. She is currently an Associate Professor with the Socially Perceptive Computing Lab, TU Delft, The Netherlands, where she works since 2013. She works towards finding how automated systems best serve people with social awareness. Her research interests include devising multimodel models to automatically interpret face-to-face human social behavior.

Table 14

Fig. 10: Illustration of the position and orientation extraction pipeline. Left: we construct 2D keypoints on a frame in COCO format; Middle: we project the keypoint into 3D space by assuming the height of each keypoint; Right: we construct for sets of features (positions and orientations) to leverage in GTCG/GCFF pipelines.

Keypoint	head	nose	shoulder	hip	ankle	toe
$\gamma$	1	0.95	0.85	0.5	0.02	0.02

TABLE III: Choice of height coefficient

Aug 28, 2026

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.

Mark Dourado, Karim Haddad, Henrik G. Hassager +1

Research & Exploration, GN Store Nord, Ballerup, Denmark · Department of Architecture, Design and Media Technology, Aalborg University, Copenhagen, Denmark · Department of Engineering Technology and Didactics, Technical University of Denmark, Kongens Lyngby, Denmark

Turn-Taking Prediction Turn-Taking in Dialogue

Ask PDF

Oct 14, 2026

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.

Puneet Mathur, Dinesh Manocha

University of Maryland College Park, USA

Turn-Taking in Dialogue Full-Duplex Spoken Dialogue Systems Turn-Taking Prediction

Ask PDF

Jul 20, 2026

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks

Measuring conversational states such as cognitive load and conversational power from multimodal behavior requires characteristic features that are not only predictive but also reliable across task contexts. We present a three-dimensional evaluation framework assessing predictive accuracy, cross-task generalizability, and test-retest reliability, applied to interactional, acoustic, and linguistic features extracted from dyadic conversations during collaborative tasks performed over a video-conferencing platform (AVCAffe dataset; 53 dyads, 9 tasks). Our results show that no single feature family dominates all three dimensions. Linguistic features show the highest predictive accuracy for cognitive load but collapse under cross-task evaluation, revealing sensitivity to task-specific vocabulary. Additionally, acoustic reliability, often reported as evidence of feature stability, degrades once speaker identity is controlled, confirming that standard prosodic features measure vocal characteristics rather than conversational state. Interaction features provide the only genuinely reliable signal, unchanged after speaker normalization. Interestingly, classifying power role remained near chance baseline across all conditions, indicating limitations of task-level aggregated behavior for predicting power role in conversation. Our findings reveal three insights: (1) linguistic features predict best but generalize poorly across task contexts; (2) acoustic reliability collapses to near-zero once speaker identity is controlled, challenging standard evaluation practice; and (3) interaction features provide the only genuinely reliable signal, with floor dominance predicting within-dyad cognitive load asymmetry. These results argue for speaker normalization and multi-dimensional evaluation as prerequisites for context-aware, robust multimodal feature selection in conversational systems.

Tahiya Chowdhury

Colby College Waterville, Maine, United States

Spoken Dialogue Systems Multimodal Model Evaluation

Ask PDF

Abstract

Explore similar work

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks