DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

Shangeth Rajaa

eess.ASMar 9, 2026

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

Authors: Shangeth Rajaa

Organizations: Anyreach AI

Abstract

Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.

Figures & tables

Figure 1: DualTurn architecture. Stage-1 pretrains the backbone and depth predictor (discarded after pretraining) for next speech token prediction; Stage-2 finetunes the twelve classification heads to predict the turn-taking signals.

Signal	Definition
EOT	Speech offset; other speaker takes floor within 4s
HOLD	All other speech offsets (not EOT)
BOT	Speech onset ( $\geq$ 1s) following the other speaker
BC	Isolated utterance $\leq$ 1s, $\geq$ 1s silence before/after
VAD	Binary voice activity per frame
FVAD	Mean voice activity over [0–240ms], [240–480ms], [480–960ms], [960ms–2s] ahead

Table 1: Self-supervised signal definitions (per channel)

	Action	Definition
ST	Start-talking	User offset; agent speaks within 4s
CL	Continue-listening	User offset; user resumes within 2s
SL	Start-listening	Overlap onset; incoming speech $>$ 1s
CT	Continue-talking	Overlap onset; incoming speech $<$ 1s
BC	Backchannel	Agent vocalization $<$ 1s during user speech

Table 2: Agent actions and ground-truth definitions.

— Switchboard —
Model	wF1	BC F1	Ant@ $-$ 240
VAP (native)	.276	—	.785
VAP (LR-6)	.389	.000	.780
A – LoRA (main)	.633	.349	.874
B – LoRA + CB	.613	.077	.876
C – No Pretrain	.604	.079	.863

Table 3: Agent action prediction on Switchboard (138-session) and otoSpeech (113-session). wF1 = weighted F1 (5 classes). Ant@

-

240 = shift/hold AUC 240 ms before offset (chance = 0.500). Variants A–G defined in Section 3.4 .

Model	S/H	S/L	S-P	BC-P
VAP (our eval)	.843	.916	.720	.838
A – LoRA	.985	.979	.764	.864
E – Full FT	.982	.985	.771	.869

Table 4: Frame-level VAP protocol on Switchboard, evaluated under VAP’s original 1s bidirectional event conditions [ 17 ] . All metrics: wF1. S/H = Shift/Hold, S/L = Short/Long, S-P = Shift-Prediction, BC-P = BC-Prediction. VAP paper used 11-fold CV (S/H = 0.899); our fixed-split re-evaluation gives 0.843.

Model	AUC(C)	AUC(B)	AUC(T)	Avg	EER
GPT-2	.851	.774	.862	.829	24.5
RP+HuBERT+hist	.903	.818	.920	.880	19.3
A – EOT only	.918	.904	.919	.914	15.2
A – Heuristic	.940	.925	.924	.930	13.2
A – LR-probe	.961	.979	.950	.963	9.7

Table 5: Word-level turn prediction on Switchboard. C = continue, B = backchannel, T = turn-shift. Baselines from [ 16 ] . A–EOT: single EOT signal, 0 params. A–Heur: multi-signal heuristic, 0 params. A–LR: LR probe over all signal heads.

Figure 2: Shift-vs-Hold AUC at each time offset relative to speech offset (Switchboard test set).

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 301 papers

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues
Jul 28, 2026Takyoung Kim, Kang-wook Kim, Sang Hoon Woo +3Human-in-the-Loop AI Synthetic Data Generation for Language Models
University of Illinois Urbana-Champaign · Seoul National University · University of California, Berkeley +2
Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models
Oct 14, 2026Yuxuan Chen, Yang Xiao, Jack Sun +3Speech Language Models Full-Duplex Spoken Dialogue Systems
The Chinese University of Hong Kong, Shenzhen · The University of Melbourne · Lulula
Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue
Mar 12, 2026Kratika Bhagtani, Mrinal Anand, Yu Chen Xu +1Turn-Taking in Dialogue Turn-Taking Prediction
School of Electrical and Computer Engineering, Purdue University, United States · Ishiki Labs Inc.

Jul 28, 2026J/K move · Enter open · S save

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

Takyoung Kim, Kang-wook Kim, Sang Hoon Woo, Julia Hirschberg+2

University of Illinois Urbana-Champaign · Seoul National University · University of California, Berkeley+2

Human-in-the-Loop AI Synthetic Data Generation for Language Models Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue

Open paper AskPDF arXiv

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human-human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce DuplexGen, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations. In six cooperative and competitive tasks, human turn-taking preferences differ systematically, and DuplexGen aligns substantially more closely with those preferences than uncalibrated prompting or training solely on generic human-human data; a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors. These results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

Abstract

Speech Language Models

Explore similar work

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models

Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue