Quantifying conversational dynamics requires reliable identification of interactional units and their temporal boundaries, but speech activity alone does not distinguish conversational turns from listener feedback or within-turn pauses. We present an automated pipeline for extracting turns and backchannels from separate-channel recordings of spontaneous dyadic conversation, designed to provide a consistent first-pass annotation for subsequent human review. The pipeline combines voice activity detection, channel-energy filtering, temporal merging, automatic speech recognition, and context-based post-processing. We evaluated the pipeline on 99 ten-minute Danish conversations from 33 dyads using segment-level detection reliability and temporal boundary error. Conversations were recorded under both normal and asymmetric listening conditions. In the latter, speech-shaped noise was delivered to one participant through bone-conduction headphones. Overall detection reliability was F1=0.621, with similar performance for turns F1=0.624 and backchannels F1=0.618. For successfully matched segments, median absolute onset and offset errors were 0.150 and 0.160s for turns and 0.130 and 0.180s for backchannels, respectively. Mean errors were substantially larger for turn boundaries, indicating a smaller number of large boundary mismatches. Performance did not differ significantly across the two experimental listening conditions. In a four-conversation case study, pipeline-human agreement was lower and more variable than human inter-annotator agreement and varied across parameter settings. These results support the pipeline as an automated first pass within a semi-automated annotation workflow, providing a consistent basis for more standardised and reproducible annotation of conversational dynamics.
Figures & tables
Figure 2: Example showing the effect of context-based post-processing on one conversation. Manual reference annotations (top) and pipeline output (bottom) are shown before and after post-processing. Colours distinguish turns and backchannels produced by the two participants, while outlined segments indicate overlapped turns.
Source
Stage
Turn
BC
Overlapped
Total
Manual annotation
Before
36
27
6
69
After
28
27
0
55
Pipeline
Before
55
66
4
125
After
31
27
0
58
Table 1: Segment counts for the conversation shown in Fig. 2 , before and after context-based post-processing of the manual reference annotation and pipeline output.
Metric
Mean F1
MedAE (s)
MAE (s)
Detection reliability
Overall
0.621±0.151
–
–
Turn
0.624±0.143
–
–
Backchannel
0.618±0.160
–
–
Temporal precision
Turn onset
–
0.150 [0.100–0.910]
0.994±2.383
Table 2: Global pipeline performance against human reference annotations across 99 conversations from 33 dyads. Detection reliability is reported as the mean F1 score ± standard deviation (SD) across conversations. Temporal precision pools the absolute boundary error of every correctly matched segment ( n=19,958 ) and summarises that one distribution twice: as the median with its inter-quartile range (MedAE) and as the mean ± SD (MAE). Absolute errors are bounded below by zero and right-skewed, so the median is paired with an IQR rather than an SD. All temporal errors are in seconds.
Figure 3: Pipeline performance across noise-free and asymmetric-noise listening conditions for all 99 conversations. Left : F1 scores for turn and backchannel detection. Centre : median absolute error (MedAE) for turn onset and offset boundaries. Right : MedAE for backchannel onset and offset boundaries. Differences between listening conditions were assessed using two-sided Mann–Whitney U tests; none of the comparisons was statistically significant.
Figure 4: Case-study comparison of pipeline–human agreement across 320 pipeline runs (80 parameter combinations applied to four conversations) and human inter-annotator agreement on the same four conversations. Left : F1 scores for turn and backchannel detection. Centre : median absolute error (MedAE) for turn onset and offset boundaries. Right : MedAE for backchannel onset and offset boundaries. Pipeline distributions reflect variation across parameter combinations, whereas human agreement reflects the four independently annotated conversations. The distributions are shown descriptively and were not subjected to inferential group comparison because the repeated pipeline runs are not independent observations.
Metric
Pipeline
Human agreement
(320 runs)
(4 conversations)
Detection reliability ( F1 , mean ± SD across runs)
Overall
0.626±0.225
0.845±0.118
Turn
0.668±0.179
0.868±0.051
Backchannel
0.585±0.256
0.821±0.168
Temporal precision — MedAE (median [IQR], s)
Table 3: Case study: pipeline performance across 320 grid runs (80 parameter combinations on 4 conversations) benchmarked against human inter-annotator agreement on the same conversations. F1 is the mean ± SD across runs. Temporal precision pools the absolute boundary error of every correctly matched segment ( n=54,146 pipeline, n=928 human) and summarises that distribution twice: as the median with its inter-quartile range (MedAE) and as the mean ± SD (MAE). The pipeline pool spans the whole grid, so each conversation contributes its segments once per parameter combination.
Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.
Reliable turn-taking is essential for spoken dialogue systems. However, most existing methods are designed for two-speaker interaction and struggle with realistic multiparty audio containing overlap and rapid speaker changes. We study multiparty turn-taking on the VoxConverse dataset and propose an audio-only two-stage pipeline that separates when to trigger a turn boundary from whether the floor is actually transferring. A fast trigger scans the audio and proposes candidate end-of-turn times, while a lightweight verifier runs only at those times to decide Hold or Shift and support next-speaker prediction. We report results in the full multiparty setting and a controlled dyadic top-2 projection for comparability. We also investigate diffusion-based, label-preserving background-audio mixing as a data augmentation strategy. Results show improved shift detection over a baseline, with further improvements from diffusion augmentation.
Rutherford A. Patamia, Ming Liu, Wei Luo +2
Deakin University, Melbourne, Australia · Griffith University, Brisbane, Australia
In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of different modalities remains unclear. We present a controlled ablation of acoustic, prosodic, and semantic signals for streaming end of turn detection using a lightweight trimodal classifier. Under identical training conditions, the acoustic prosodic combination achieves the best balance of accuracy and latency, achieving utterance F1 of 0.93 with 7.8% false alarms at 400ms median latency. Adding text increases premature detections without improving performance. Feature space analysis confirms that prosodic features have the strongest class separability, while text representations overlap substantially. These findings suggest that turn-taking is primarily conveyed through intonation and silence patterns rather than semantic completeness, enabling faster and more reliable systems without expensive text inference.