Breakdown of Conversational Involvement: A Joint Multi-Modal Analysis of Conversation Floors and F-formation Detection with Multiple Features

Zonghuan Li; Chirag Raman; Hayley Hung

cs.CLOct 13, 2026

Breakdown of Conversational Involvement: A Joint Multi-Modal Analysis of Conversation Floors and F-formation Detection with Multiple Features

Authors: Zonghuan Li, Chirag Raman, Hayley Hung

Organizations: Intelligent Systems, Delft University of Technology EEMCS, 2628 XE Delft, Zuid-Holland, The Netherlands

Abstract

Interactions happen universally and spontaneously in daily lives. Studying involvement in interactions helps us analyze who can influence who and has a wide potential in applications. Specifically, we consider conversational involvement in free-standing conversations. We study two operationalizations of an FCG: F-formations and conversation floors, and propose that current assumption that F-formations capture single conversations is insufficient. To illustrate how different implementations could impact the detected F-formations, we employ three analyses. First, we propose that speaking status can be used for estimating conversation floors, and leveraged it to analyze floors within F-formations. Second, we analyze spatial characteristics of the detection results by applying different feature sets on chosen baselines, and find that groups resulting from head features are more likely to be subsumed by groups detected using other features. Finally, we do a joint analysis which combines the spatial features and floor patterns in the conversation. We observe that certain pipelines can detect groups with single floors from upper body features, while retaining F-formations with lower body features. We conclude that a more detailed theory and framework is needed to holistically demonstrate conversational involvement within interactions.

Figures & tables

Fig. 1: Depiction of a single F-formation with multiple conversation floors. The darker green regions within dotted lines represent distinct simultaneous conversation floors. Most works representing a conversing group as an F-formation make the simplifying assumption that a single conversation occurs within an F-formation with a joint focus of attention for all members.

Fig. 2: Illustration of gaps, within-overlaps, and between-overlaps for two speakers (S1 and S2) within the same floor. The scheme was originally proposed by Heldner and Edlund [ 20 ] and adopted by Levinson and Torreira in their analysis [ 26 ] .

Fig. 3: Illustration of the experimental design. We start with a dataset with speaking status and F-formation annotations, as well as body spatial information of human bodies. Temporal analysis: we analyze simultaneous speakers within an F-formation to demonstrate there might be more than one speaker; Spatial analysis: We run selected F-formation detection pipelines and analyze spatial patterns between them; Joint analysis: we combine the spatial features with conversation floors to study the relationship between detection results and actual physically embodied floors.

Fig. 4: Illustration of spatial measures

S_{H}

and

S_{F}

. First, we identify the composition of annotation

A_{i}

, detection results

G_{\alpha},G_{\beta}

, and the relative detection

\mathcal{R}_{\alpha}(A_{i})

. For SH, we first calculate

s_{u}

and

s_{p}

, then take the maximum to get

S_{H}

. For SF, it is

1

when there are at least 2 subsets of

g_{\alpha}

G_{\beta}

with at least 2 members, and

0

otherwise.

Fig. 5: Illustration of joint analysis pipeline. First we split the duration of an annotated F-formation into tiles of length

t_{0}

, then compute speaker within each tile. For each frame in each tile where a split event exists, we compute

M_{1}

and

M_{2}

M_{1}

calculates whether the larger group contain at least 2 speakers, and

M_{2}

calculates the proportion of subgroups that contain exactly one speaker. Together, they measure how much the detected split events align with the characteristic that multiple floors exist within one F-formation.

Fig. 6: Simultaneous speakers per cardinality of detection using annotations. Each bar is average speaker number with standard deviation, with respect to a specific window duration. For groups with cardinality 4, more than 1 speaker is observed for 1-5 seconds of window. For groups with cardinality 5, no average speaker number is above 1. For cardinality 6-7, although observed with more than 1 speaker for windows duration up to 2 and 9 seconds respectively, the number of samples also drops substantially, resulting in no standard deviation.

	Coeff ( $\beta$ )	Std Err	$z$	$P>z$
Intercept	-0.7164	0.564	-1.270	0.204
Turn-duration	0.2424	0.076	3.173	0.002
Cardinality	0.2726	0.121	2.244	0.025
Turn-duration:Cardinality	-0.0754	0.018	-4.298	0.000

TABLE I: Generalized model regression results

Cardinality pairs	Intercept ( $\beta_{0}$ )	d( $\beta_{1}$ )	c( $\beta_{2}$ )	d:c( $\beta_{3}$ )
4-5	0.831	0.0005	0.879	0.0003 $*$
4-6	0.426	0.011	0.099	0.002
4-7	0.295	0.592	0.017	0.086
5-6	0.463	0.042	0.285	0.109
5-7	0.387	0.001	0.113	0.006
6-7	0.918	0.048	0.600	0.104

TABLE II: Nominal P-values for six post-hoc GLM regression comparisons

Fig. 7: Number of F-formations at different speaking turn durations.

Panel of Figure 10 — Fig. 8: Spatial analysis results for both pipelines. All figures are interpreted from row to column. For instance, a cell located at shoulder row and hip row of SH is interpreted as $S_{H}(\text{shoulder},\text{hip})$ . 8a : SH and SF for GTCG pipeline. SH is at least $0.96$ which shows high agreement over groupinf patterns. SF shows that it takes highest for the $(\text{shoulder},\text{head})$ pair and is substantially higher when $\beta$ takes head features. 8b : SH and SF for GCFF pipeline. SH score is $0.99$ for every pair of different $(\alpha,\beta)$ , showing high agreement over groupings. SF shows clear asymmetry: $S_{F}(\alpha,\beta)$ is much larger than the opposite when $\alpha$ is physically lower than $\beta$ on the body.

Panel of Figure 11 — Fig. 9: $M_{1}$ and $M_{2}$ for each clue pair for both pipelines. Within the labels, $\alpha\rightarrow\beta$ refers to $M(\alpha,\beta)$ . $M_{1}$ calculates how often there are at least 2 speakers in the larger group when a split event happens, and $M_{2}$ calculates how frequently each subgroup contain exactly one speaker when $M_{1}$ holds. The data bars are grouped in 4 sets: (LBF, UBF), (UBF, LBF), within UBF, and within LBF. 9a : $M_{1}$ and $M_{2}$ for GTCG. 9b : $M_{1}$ and $M_{2}$ for GCFF.

Zonghuan Li received BEng degree in Electrical Engineering from Tsinghua University in China, 2016, and master’s degree in Electrical Engineering from University of California, San Diego in United States, 2023. He is currently a PhD student under the Socially Perceptive Computing Lab, TU Delft, Netherlands since 2024. His research interests include conversational involvement analysis with multimodal signals, multimodal large language model applications and evaluation.

Table 12

Chirag Raman received his master’s degree in entertainment technology from Carnegie Mellon University, USA in 2013 and the Ph.D in Computer Science from TU Delft in 2023. He is currently an Assistant Professor at TU Delft where he leads the Tapri Lab within the Department of Intelligent Systems. His research interests include gnerative modeling, socially intelligent embodied agents, with insights from cognitive science and social psychology.

Table 13

Hayley Hung (Member, IEEE) received the PhD degree in computer vision from the Queen Mary University of London in 2007. She is currently an Associate Professor with the Socially Perceptive Computing Lab, TU Delft, The Netherlands, where she works since 2013. She works towards finding how automated systems best serve people with social awareness. Her research interests include devising multimodel models to automatically interpret face-to-face human social behavior.

Table 14

Fig. 10: Illustration of the position and orientation extraction pipeline. Left: we construct 2D keypoints on a frame in COCO format; Middle: we project the keypoint into 3D space by assuming the height of each keypoint; Right: we construct for sets of features (positions and orientations) to leverage in GTCG/GCFF pipelines.

Keypoint	head	nose	shoulder	hip	ankle	toe
$\gamma$	1	0.95	0.85	0.5	0.02	0.02

TABLE III: Choice of height coefficient

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness
Aug 28, 2026Mark Dourado, Karim Haddad, Henrik G. Hassager +1Turn-Taking Prediction Turn-Taking in Dialogue
Research & Exploration, GN Store Nord, Ballerup, Denmark · Department of Architecture, Design and Media Technology, Aalborg University, Copenhagen, Denmark · Department of Engineering Technology and Didactics, Technical University of Denmark, Kongens Lyngby, Denmark
MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue
Oct 14, 2026Puneet Mathur, Dinesh ManochaTurn-Taking in Dialogue Full-Duplex Spoken Dialogue Systems
University of Maryland College Park, USA
How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks
Jul 20, 2026Tahiya ChowdhurySpoken Dialogue Systems Multimodal Model Evaluation
Colby College Waterville, Maine, United States

Aug 28, 2026J/K move · Enter open · S save

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

Mark Dourado, Karim Haddad, Henrik G. Hassager, Stefania Serafin

Research & Exploration, GN Store Nord, Ballerup, Denmark · Department of Architecture, Design and Media Technology, Aalborg University, Copenhagen, Denmark · Department of Engineering Technology and Didactics, Technical University of Denmark, Kongens Lyngby, Denmark

Turn-Taking Prediction Turn-Taking in Dialogue

Open paper AskPDF arXiv

Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.

Abstract

Explore similar work

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks