An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

Hanlu He; Harald Vilhelm Skat-Rørdam; Ingvi Örnólfsson; Ivana Konvalinka

cs.CLOct 2, 2026

An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

Authors: Hanlu He, Harald Vilhelm Skat-Rørdam, Ingvi Örnólfsson, Ivana Konvalinka

Organizations: Department of Applied Mathematics and Computer Science, Technical University of Denmark

Abstract

Quantifying conversational dynamics requires reliable identification of interactional units and their temporal boundaries, but speech activity alone does not distinguish conversational turns from listener feedback or within-turn pauses. We present an automated pipeline for extracting turns and backchannels from separate-channel recordings of spontaneous dyadic conversation, designed to provide a consistent first-pass annotation for subsequent human review. The pipeline combines voice activity detection, channel-energy filtering, temporal merging, automatic speech recognition, and context-based post-processing. We evaluated the pipeline on 99 ten-minute Danish conversations from 33 dyads using segment-level detection reliability and temporal boundary error. Conversations were recorded under both normal and asymmetric listening conditions. In the latter, speech-shaped noise was delivered to one participant through bone-conduction headphones. Overall detection reliability was F1=0.621, with similar performance for turns F1=0.624 and backchannels F1=0.618. For successfully matched segments, median absolute onset and offset errors were 0.150 and 0.160s for turns and 0.130 and 0.180s for backchannels, respectively. Mean errors were substantially larger for turn boundaries, indicating a smaller number of large boundary mismatches. Performance did not differ significantly across the two experimental listening conditions. In a four-conversation case study, pipeline-human agreement was lower and more variable than human inter-annotator agreement and varied across parameter settings. These results support the pipeline as an automated first pass within a semi-automated annotation workflow, providing a consistent basis for more standardised and reproducible annotation of conversational dynamics.

Figures & tables

Figure 2: Example showing the effect of context-based post-processing on one conversation. Manual reference annotations (top) and pipeline output (bottom) are shown before and after post-processing. Colours distinguish turns and backchannels produced by the two participants, while outlined segments indicate overlapped turns.

Source	Stage	Turn	BC	Overlapped	Total
Manual annotation	Before	36	27	6	69
	After	28	27	0	55
Pipeline	Before	55	66	4	125
	After	31	27	0	58

Table 1: Segment counts for the conversation shown in Fig. 2 , before and after context-based post-processing of the manual reference annotation and pipeline output.

Detection reliability
Metric	Mean $F_{1}$	MedAE (s)	MAE (s)
Overall	$0.621\pm 0.151$	–	–
Turn	$0.624\pm 0.143$	–	–
Backchannel	$0.618\pm 0.160$	–	–
Temporal precision
Turn onset	–	$0.150$ [0.100–0.910]	$0.994\pm 2.383$

Table 2: Global pipeline performance against human reference annotations across 99 conversations from 33 dyads. Detection reliability is reported as the mean

F_{1}

score

\pm

standard deviation (SD) across conversations. Temporal precision pools the absolute boundary error of every correctly matched segment (

n=19,958

) and summarises that one distribution twice: as the median with its inter-quartile range (MedAE) and as the mean

\pm

SD (MAE). Absolute errors are bounded below by zero and right-skewed, so the median is paired with an IQR rather than an SD. All temporal errors are in seconds.

Figure 3: Pipeline performance across noise-free and asymmetric-noise listening conditions for all 99 conversations. Left :

F_{1}

U

tests; none of the comparisons was statistically significant.

Figure 4: Case-study comparison of pipeline–human agreement across 320 pipeline runs (80 parameter combinations applied to four conversations) and human inter-annotator agreement on the same four conversations. Left :

F_{1}

scores for turn and backchannel detection. Centre : median absolute error (MedAE) for turn onset and offset boundaries. Right : MedAE for backchannel onset and offset boundaries. Pipeline distributions reflect variation across parameter combinations, whereas human agreement reflects the four independently annotated conversations. The distributions are shown descriptively and were not subjected to inferential group comparison because the repeated pipeline runs are not independent observations.

Detection reliability ( $F_{1}$ , mean $\pm$ SD across runs)
Metric	Pipeline	Human agreement
	(320 runs)	(4 conversations)
Overall	$0.626\pm 0.225$	$0.845\pm 0.118$
Turn	$0.668\pm 0.179$	$0.868\pm 0.051$
Backchannel	$0.585\pm 0.256$	$0.821\pm 0.168$
Temporal precision — MedAE (median [IQR], s)

Table 3: Case study: pipeline performance across 320 grid runs (80 parameter combinations on 4 conversations) benchmarked against human inter-annotator agreement on the same conversations.

F_{1}

is the mean

\pm

SD across runs. Temporal precision pools the absolute boundary error of every correctly matched segment (

n=54,146

pipeline,

n=928

human) and summarises that distribution twice: as the median with its inter-quartile range (MedAE) and as the mean

\pm

SD (MAE). The pipeline pool spans the whole grid, so each conversation contributes its segments once per parameter combination.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Aug 25, 2026Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Sesame AI · Mundo AI · Oto +4
Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentation
Jun 15, 2026Rutherford A. Patamia, Ming Liu, Wei Luo +2Spoken Dialogue Systems Turn-Taking Prediction
Deakin University, Melbourne, Australia · Griffith University, Brisbane, Australia
Less can be More: What Aspects of Speech Drive End-of-Turn Detection
Sep 10, 2026Rini Sharon, Manickavela A, Kadri Hacioglu +1Speech Prosody Turn-Taking Prediction
Uniphore, India

Aug 25, 2026J/K move · Enter open · S save

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri+15

Sesame AI · Mundo AI · Oto+4

Spoken Dialogue Systems Multi-Turn Dialogue Evaluation Turn-Taking Prediction Turn-Taking in Dialogue

Open paper AskPDF arXiv

Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.

Abstract

Speech Processing

Explore similar work

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentation

Less can be More: What Aspects of Speech Drive End-of-Turn Detection