D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

Cheng-Kuang Chang; Alexander H. Liu; Brian Kingsbury; Hilde Kuehne; Rogerio Feris; Samuel Thomas; James Glass; Kai-Wei Chang

cs.CLOct 12, 2026

D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

Authors: Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury, Hilde Kuehne, Rogerio Feris, Samuel Thomas, James Glass, Kai-Wei Chang

Organizations: MIT CSAIL · MIT-IBM Watson AI Lab

Abstract

Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-SLMs) in general capabilities. We propose Dialogue-to-Duplex (D2), a framework that converts pretrained HD-SLMs into high-capability FD-SLMs at fine temporal granularity. D2 adapts the original audio encoder to streaming through decoder-free representation distillation and learns full-duplex interaction through supervised fine-tuning. We systematically study the effects of chunk size on full-duplex interaction, general capabilities, and inference computation. We apply D2 to two distinct HD-SLM architectures, Qwen3-Omni and LLaMA-Omni 2. At a 12.5Hz update rate, Qwen3-Omni-D2 achieves 99.2% turn-taking success with 521-ms latency, 99.5% interruption success, and a state-of-the-art VoiceBench score of 70.1.

Figures & tables

Figure 1: Capability and interactivity of full-duplex models. FD-Score is defined in Appendix F .

Figure 2: Overview of Dialogue-to-Duplex (D2). (a) A frozen offline encoder supervises a streaming student with bidirectional attention within each chunk and causal attention across chunks. (b) At chunk $i$ , D2 uses current user audio $X_{i}$ and preceding model audio $S_{i-1}$ to autoregressively generate $S_{i}$ . Here, $k=4$ .

Figure 3: Full-duplex data construction. Ordered turns become sequences of at least 720 s with interruptions, turn-taking, and silence.

Model	Chunk duration (ms)	Quality (1–5) $\uparrow$	Turn taking (%) $\uparrow$	TT latency (s) $\downarrow$	Interrupt (%) $\uparrow$	Int. latency (s) $\downarrow$	Pause (%) $\uparrow$
Qwen3-Omni-D2	80	4.56	99.2	0.521	99.5	0.403	21.3
	160	4.67	100	0.539	99.5	0.432	19.0
	320	4.61	100	0.710	99.5	0.571	25.5
	640	4.62	99.2	1.01	99.5	0.928	55.1
	1040	4.84	100	1.38	99.5	1.30	61.6
LLaMA-Omni 2-D2	100	3.33	100	0.746	99.5	0.621	74.5

Table 1: Full-Duplex-Bench results. Chunk duration is

k\Delta

for D2 models; – marks non-D2 models. Quality is post-interruption response quality rated by GPT-4o. Turn taking/Interrupt are take-over rates (TOR); Pause is

100

minus Synthetic Pause Handling TOR (%). TT/Int. mean turn-taking/interruption.

Model	Chunk duration (ms)	Alpaca Eval	Common Eval	Wild Voice	SD- QA	MMSU	OBQA	BBH	IF Eval	Adv Bench	Overall
Qwen3-Omni-D2	80	4.30	3.47	3.58	64.5	64.5	81.5	64.5	37.2	91.5	70.1
	160	4.33	3.44	3.63	68.0	60.0	71.5	59.0	29.6	94.0	67.8
	320	4.40	3.70	3.76	67.0	60.0	86.5	62.0	36.9	95.0	71.6
	640	4.38	3.96	3.78	57.0	63.0	85.5	60.0	38.0	95.5	71.3
	1040	4.53	3.67	3.94	66.5	63.5	86.0	60.0	39.4	98.5	73.0
Qwen3-Omni	HD	4.84	4.52	4.69	68.5	73.0	91.0	87.5	80.7	99.5	86.8

Table 2: VoiceBench scores. Chunk duration is

k\Delta

for D2 models; HD denotes a half-duplex model and – denotes another full-duplex model. AlpacaEval, CommonEval, and WildVoice use a 1–5 scale. The remaining sub-benchmarks report percentages; Overall is the aggregate score on a 0–100 scale.

Figure 4: VoiceBench versus chunk duration. Filled circles/squares and solid lines show D2 variants; open markers show original half-duplex models. Dashed connections compare D2 with its backbones. The axis break and $\infty$ denote turn-based operation without fixed-duration chunks.

Chunk duration (ms)	Mean (ms)	p95 (ms)	RTF $\downarrow$
80	56.9	57.2	0.711
160	110	110	0.688
320	216	217	0.676
640	429	431	0.670
1,040	682	684	0.656

Table 3: Full inference computation time of Qwen3-Omni-D2 with a fixed 2,048-slot Thinker KV cache per layer. RTF is computation time divided by chunk duration.

Metric	Distill.	Supervised	No Stage I
Pause (%) $\uparrow$	74.5	61.6	62.0
Int. TOR (%) $\uparrow$	99.5	100	99.0
TT TOR (%) $\uparrow$	100	100	100
Quality (1–5) $\uparrow$	3.33	3.09	2.75
Int. latency (ms) $\downarrow$	621	590	617
TT latency (ms) $\downarrow$	746	676	686

Table 4: Stage-I encoder training ablation for LLaMA-Omni 2-D2 with a 100-ms chunk duration. Distill. denotes distillation; No Stage I omits encoder adaptation.

CE weight		Turn taking	Interruption
PAD	Control	TOR $\uparrow$	TOR $\uparrow$	Score $\uparrow$
0.05	20	100	99.0	3.67
1	20	95.8	98.0	3.44
0.05	1	95.8	98.5	3.55
1	1	62.2	90.5	3.45

Table 5: Control-signal weighting ablation on Full-Duplex-Bench. Lexical-token weight is 1 throughout. TOR is reported in percentages. Scores cover qualifying interruption responses.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Module (both backbones)	Training	LR
Encoder frontend	Full	$5\times 10^{-5}$
Encoder attention	Full	$2\times 10^{-6}$
Encoder MLP/projector	LoRA, $r=32$	$10^{-4}$
Original teacher	Frozen	–
Updates		30,000
Global batch / accumulation		4 / 1

Appendix

Table 6: Stage-I training. Each chunk size has its own distilled encoder. Hardware and logged time are in Table 8 .

Backbone	Module	Training	Peak LR
Both	Thinker attention	LoRA, $r=128$	$5\times 10^{-5}$
Both	Encoder frontend	Full	$2\times 10^{-6}$
Both	Encoder attention/MLP/projector	LoRA, $r=32$	$10^{-5}$
Both	New control and stream embeddings	Full	$10^{-4}$
Qwen3-Omni	Initial self-silence embedding	Full	$10^{-4}$
Qwen3-Omni	Talker attention/shared expert, context projections, MTP	LoRA, $r=128$	$5\times 10^{-5}$

Appendix

Table 7: Stage-II training. Unlisted pretrained parameters are frozen, including Qwen routed experts and routers, Code2Wav, and the LLaMA-Omni 2 TTS input bridge, tokenizer, flow model, and vocoder.

Backbone	$k$	Chunk duration (ms)	Stage-I GPUs	Stage-I time (h)	Stage-II GPUs	Stage-II time (h)
Qwen3-Omni	1	80	$4\times$ RTX	1.73	$4\times$ RTX	12.5
	2	160	$2\times$ L40S	1.81	$4\times$ RTX	13.2
	4	320	$2\times$ L40S	1.58	$2\times$ H200	13.0
	8	640	$2\times$ L40S	2.04	$4\times$ RTX	11.9
	13	1040	$4\times$ RTX	1.21	$4\times$ RTX	10.8
LLaMA-Omni 2	1	100	$4\times$ RTX	1.50	$4\times$ RTX	12.4

Appendix

Table 8: Hardware and approximate logged training time. RTX denotes NVIDIA RTX PRO 6000 Blackwell Server Edition; L40S and H200 denote NVIDIA L40S and NVIDIA H200. Times are elapsed hours, not GPU-hours, summed within resumed log segments. They include replayed updates and data/checkpoint overhead, but exclude queue time and initialization between segments.

Source	Unit	Count
LibriSpeech	Speech clip	35,245
AVQA	Audio–QA pair	35,245
InstructS2S-200K	Conversation	149,998
SODA	Conversation	41,776

Appendix

Table 9: Prepared source examples before conversation concatenation and repeated sampling. Stage I uses only the audio from AVQA.

Model	Chunk size $k$ (duration)	UTMOS $\uparrow$
Qwen3-Omni-D2	1 (80 ms)	$4.54\pm 0.03$
Qwen3-Omni	HD	$4.46\pm 0.07$
LLaMA-Omni 2-D2	1 (100 ms)	$4.16\pm 0.20$
LLaMA-Omni 2	HD	$4.43\pm 0.09$

Appendix

Table 10: Speech quality at chunk size

k=1

for both D2 backbones. UTMOS is reported as mean

\pm

sample standard deviation on shared prompts. HD denotes the original half-duplex backbone.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 310 papers

Full-duplex Spoken Language Modeling with Discrete Diffusion
Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language Models Full-Duplex Spoken Dialogue Systems
NVIDIA
DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion
Oct 8, 2026Heeseung KimFull-Duplex Spoken Dialogue Systems Diffusion Model Inference Acceleration
Department of Artificial Intelligence University of Seoul Seoul, Republic of Korea
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Oct 13, 2026J/K move · Enter open · S save

Full-duplex Spoken Language Modeling with Discrete Diffusion

Chen Chen, Pin-Jui Ku, Ankita Pasad, Ante Jukić+2

NVIDIA

Speech Language Models Full-Duplex Spoken Dialogue Systems Diffusion Language Models Efficient Language Model Inference Diffusion Language Model Decoding

Open paper AskPDF arXiv

We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.

Abstract

Speech Language Models

Explore similar work

Full-duplex Spoken Language Modeling with Discrete Diffusion

DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM