Dyship: Leveraging Acoustic Features For Speaker Relationship Classification From Dyadic Conversations
Organizations: Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · CSE and Mathematics Department, Pennsylvania State University, State College, PA, USA · AMIAD, Palaiseau, France
Abstract
We introduce DyShip, a framework for classifying speaker familiarity and kinship from dyadic conversations in the Seamless Interaction corpus. We compare five speech representations: spectrograms, MFCCs, self-supervised WavLM representations, x-vectors, and estimated articulatory movements. Spectral features perform best for familiarity classification, reaching an accuracy of 87.81%, while WavLM achieves the highest kinship accuracy of 85.17%. X-vectors and articulatory features yield weaker results on familiarity classifiaction, but shine for kinship classification, with an AUC of 0.909 for x-vectors. Most classifiers exceed previously reported human and audio language model scores on the same dataset. These findings highlight the potential of conventional acoustic features for relationship inference and motivate further investigation of the conversational cues captured by different representations.
Figures & tables
| Set | Duration | Speakers | Dyads | Conv. | Utt. |
|---|---|---|---|---|---|
| (h) | |||||
| Train | 579.58 | 431 | 3,118 | 41,852 | 417,306 |
| Dev | 6.36 | 103 | 47 | 559 | 4,577 |
| Test | 28.03 | 103 | 55 | 607 | 28,220 |
| Total | 613.97 | 637 | 3,220 | 43,018 | 450,103 |
| Feature | System | Familiar-vs-Stranger | Kin-vs-Nonkin | ||||
|---|---|---|---|---|---|---|---|
| Acc. | F1 | AUC | Acc. | F1 | AUC | ||
| Unweighted Average of the utterances | |||||||
| MFCC | ResNet34 | 67.38 | 66.33 | 0.917 | 84.02 | 68.51 | 0.849 |
| Spectrogram | ResNet34 | 86.49 | 82.09 | 0.897 | 82.87 | 72.11 | 0.875 |
| WavLM | ResNet34 | 85.67 | 82.20 | 0.910 | 85.01 | 73.70 | 0.891 |
| SPARC | ResNet34 | 65.73 | 64.59 | 0.828 | 84.84 | 61.41 | 0.805 |
Audio Understanding
Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.
See all 225 papers