Dyship: Leveraging Acoustic Features For Speaker Relationship Classification From Dyadic Conversations
Authors: Kieran Sturges, Thomas Thebaud, Yaohan Guan, Najim Dehak
Organizations: Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · CSE and Mathematics Department, Pennsylvania State University, State College, PA, USA · AMIAD, Palaiseau, France
We introduce DyShip, a framework for classifying speaker familiarity and kinship from dyadic conversations in the Seamless Interaction corpus. We compare five speech representations: spectrograms, MFCCs, self-supervised WavLM representations, x-vectors, and estimated articulatory movements. Spectral features perform best for familiarity classification, reaching an accuracy of 87.81%, while WavLM achieves the highest kinship accuracy of 85.17%. X-vectors and articulatory features yield weaker results on familiarity classifiaction, but shine for kinship classification, with an AUC of 0.909 for x-vectors. Most classifiers exceed previously reported human and audio language model scores on the same dataset. These findings highlight the potential of conventional acoustic features for relationship inference and motivate further investigation of the conversational cues captured by different representations.
Figures & tables
Set
Duration
Speakers
Dyads
Conv.
Utt.
(h)
Si
Di,j
Ci,j
UCi,jn
Train
579.58
431
3,118
41,852
417,306
Dev
6.36
103
47
559
4,577
Test
28.03
103
55
607
28,220
Total
613.97
637
3,220
43,018
450,103
Table 1: Durations and count of speakers, unique dyad, conversations, and utterances per subset.
Figure 1: Schematic of the pipeline: Each utterance is processed separately by different feature extractor, then by a dedicated classifier to predict the probability per utterance, aggregated into a probability per conversation.
Feature
System
Familiar-vs-Stranger
Kin-vs-Nonkin
Acc.
F1
AUC
Acc.
F1
AUC
Unweighted Average of the utterances
MFCC
ResNet34
67.38
66.33
0.917
84.02
68.51
0.849
Spectrogram
ResNet34
86.49
82.09
0.897
82.87
72.11
0.875
WavLM
ResNet34
85.67
82.20
0.910
85.01
73.70
0.891
SPARC
ResNet34
65.73
64.59
0.828
84.84
61.41
0.805
Table 2: Accuracy (Acc), Macro F1 scores (F1), and AUC metrics across representations and aggregation methods at the conversation level.
Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-Agent Debate as a task-specific adaptation of standard multi-agent debate for speaker relationship inference, assigning agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-Agent Compete, a competition-based protocol that compares agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among agents, while acoustic cues are not yet fully captured by current models.
Yaohan Guan, Yen-Ju Lu, Yuzhe Wang +5
Center for Language and Speech Processing Department of Electrical and Computer Engineering Johns Hopkins University Baltimore, MD, USA
Neural networks can be trained to learn task-relevant representations from data. Understanding how these networks make decisions falls within the Explainable AI (XAI) domain. This paper proposes to study an XAI topic: analysing, visualising and understanding the unknown organisation of network representations, particularly those a speaker recognition network learns from utterances, for recognising speaker identity. Past studies have employed algorithms (e.g. K-means) to analyse the different ways in which network representations can be naturally grouped into clusters, i.e. to analyse different flat clustering phenomena within the space defined by those representations. In contrast, this work applies two algorithms -- Single-Linkage Clustering (SLINK) and Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) -- to analyse the different ways in which representations from the speaker recognition network can form clusters with hierarchical relationships, i.e., to analyse different hierarchical clustering phenomena within the representation space of the speaker recognition network. Furthermore, an algorithm called Hierarchical Cluster-Class Matching (HCCM) is designed to semantically interpret one of the above hierarchical clustering phenomena analysed using SLINK. Given the clusters representing this phenomenon, HCCM identifies which ones best match individual semantic classes related to gender and nationality (e.g.\ male, female, Ireland, UK) and and-logic conjunctions of these classes (e.g.\ female and Ireland). The Liebig score metric is also proposed within HCCM to quantify the matching quality of each cluster-class pair and diagnose the factor that limits each match.
Yanze Xu, Wenwu Wang, Mark D. Plumbley
Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK · Department of Informatics, King’s College London, London, WC2R 2LS, UK
Reading a social situation often depends on behavior, not words alone. We introduce FriendBench, a benchmark for inferring whether two people are already familiar or are meeting as strangers, from a 20-second clip of a dyadic ice-breaker conversation. Every pair answers the same type of prompt, so only the manner of interaction can reveal the answer. Across text, audio, and video, we compare 26 models from seven companies against matched human panels over 96 balanced dyads. The best model and the human crowd are statistically indistinguishable on accuracy in every modality, but reach it differently: humans stay balanced across the two answers, while the strongest models favor ``stranger.'' This is a difference in effective prior, not in discrimination. Richer channels help both unequally, and only humans gain from visible behavior on top of speech. We release the stimuli, human ratings, and model predictions.
Jeffrey M. Girard, Jason Z. Zheng, Jacqueline R. Vertino +2