DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization

Jianing Yang; Yusuke Fujita; Yui Sudo

cs.CLMar 10, 2026

DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization

Authors: Jianing Yang, Yusuke Fujita, Yui Sudo

Organizations: SB Intuitions Corp., Japan · The University of Tokyo, Japan

Abstract

Spoken dialog systems with cascaded ASR-LLM-TTS modules retain strong LLM intelligence, but VAD segmentation often forces half-duplex turns and brittle control. On the other hand, VAD-free end-to-end model support full-duplex interaction but is hard to maintain conversational intelligence. In this paper, we present DuplexCascade, a VAD-free cascaded streaming pipeline for full-duplex speech-to-speech dialogue. Our key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM. To reliably coordinate turn-taking and response timing, we introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints. On Full-DuplexBench and VoiceBench, DuplexCascade delivers state-of-the-art full-duplex turn-taking and strong conversational intelligence among open-source speech-to-speech dialogue systems.

Figures & tables

Figure 1: Overview of DuplexCascade. User audio is transcribed by a streaming ASR and periodically flushed into text micro-turns ( $\Delta t=0.6\,$ s). The LLM consumes the dialogue history and the latest micro-turn to generate the next system micro-turn together with conversational special tokens (e.g., wait, respond, or backchannel). The generated text is then synthesized by a streaming TTS to produce system audio, enabling full-duplex interaction.

Dimension	Pause Handling		Backchannel			Smooth Turn Taking		User Interruption			Averaged Turn-Taking Accuracy
Data Metric	Synthetic TOR ( $\downarrow$ )	Candor TOR ( $\downarrow$ )	TOR ( $\downarrow$ )	ICC Freq ( $\uparrow$ )	JSD ( $\downarrow$ )	Candor TOR ( $\uparrow$ )	Latency ( $\downarrow$ )	TOR ( $\uparrow$ )	Synthetic GPT-4o( $\uparrow$ )	Latency ( $\downarrow$ )	Averaged Turn-Taking Accuracy
dGSLM †	0.934	0.935	0.691	0.015	0.934	0.975	0.352	0.917	0.201	2.531	0.466
Moshi †	0.985	0.980	1.000	0.001	0.957	0.941	0.265	1.000	0.765	0.257	0.395
Freeze-Omni †	0.642	0.481	0.636	0.001	0.997	0.336	0.953	0.867	3.615	1.409	0.489
PersonaPlex †	0.358	0.431	0.273	0.042	0.662	0.908	0.170	0.950	4.290	0.240	0.759
MiniCPM-Duplex §	0.613	0.731	0.582	0.003	0.960	0.916	0.610	1.0	1.14	0.447	0.598

Table 1: Full-Duplex-Bench results. The best-performing duplex dialogue model is shown in bold , and the second best is underlined . § means the results are infered from offical checkpoints. † means the results directly obtained from the paper.

Figure 2: Proposed dynamic construction pipeline for duplex training sequences from text-only dialogues.

Model	AlpacaEval	CommonEval	WildVoice	SD-QA	MMSU	OBQA	BBH	IFEval	AdvBench	Overall
DSM-ASR+ Qwen2-7B-Instruct	4.57	3.87	3.73	46.42	58.52	72.09	58.10	51.29	97.12	69.66
Freeze-Omni †	4.03	3.46	3.15	53.45	28.14	30.98	50.70	23.40	97.30	55.20
Moshi †	2.01	1.60	1.30	15.64	24.04	25.93	47.40	10.12	44.23	29.51
PersonaPlex §	2.69	2.25	1.95	18.77	24.92	24.40	49.40	11.95	8.07	30.59
DuplexCascade	4.40	3.64	3.56	45.56	52.86	56.04	59.80	43.38	99.04	65.41
DuplexCascade- $\beta$	4.38	3.70	3.53	46.35	52.84	58.46	59.00	44.45	99.03	65.81

Table 2: VoiceBench results. The best-performing duplex dialogue model is shown in bold , and the second best is underlined . § means the results are infered from offical checkpoints. † means the results directly obtained from the paper.

Figure 3: Effect of the micro-turn duration

\Delta t

on Averaged Turn-Taking Accuracy and turn-taking latency on Full-Duplex-Bench.

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 141 papers

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sep 8, 2026Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu +7TTS Synthesis Streaming TTS Synthesis
MediaTek Research · Internship at MediaTek Research · National Taiwan University +2
FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection
Apr 2, 2026Chengyou Wang, Hongfei Xue, Mingchen Shao +8Spoken Dialogue Systems Audio Understanding
Shengwang

Jun 12, 2026J/K move · Enter open · S save

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Spoken Dialogue Systems Speech Language Models Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue

Open paper AskPDF arXiv

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Abstract

Spoken Dialogue Systems

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction

FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection