Full-duplex Spoken Language Modeling with Discrete Diffusion

Chen Chen; Pin-Jui Ku; Ankita Pasad; Ante Jukić; Zhehuai Chen; Boris Ginsburg

cs.CLOct 13, 2026

Full-duplex Spoken Language Modeling with Discrete Diffusion

Authors: Chen Chen, Pin-Jui Ku, Ankita Pasad, Ante Jukić, Zhehuai Chen, Boris Ginsburg

Organizations: NVIDIA

Abstract

We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.

Figures & tables

Figure 1: Spoken intelligence and response efficiency. The shared vertical axis is the VoiceBench overall score across 9 subsets, higher score means smarter model. Left: Diffuplex without thinking versus full-duplex spoken dialogue models, plotted against first-audio latency (from the end of user speech to the start of model speech). Right: Diffuplex with thinking versus frontier turn-based AR models, plotted against output speed. All output speeds were measured at batch 1 on the same NVIDIA GB300 GPU. More decoding speed results on different GPUs are provided at Appendix A .

Figure 2: Overview of Diffuplex. Left: The horizontal axis represents real-time AR interaction, while a predicted <bos> triggers diffusion generation along the vertical axis at $\theta$ . An early refined prefix of $n$ tokens is streamed to the speech synthesizer as the remaining canvas continues denoising. Right: Speech features and the text instruction are jointly prefilled into the DLM as condition, which iteratively refines each response canvas through bidirectional denoising.

Model	Open-ended QA			Knowledge and Reasoning				Ins. and Safety		Overall
Model	Alp.	Com.	Wil.	SDQA	MMSU	OBQA	BBH	IFE.	Adv.	Overall
Turn-based AR Models
BR-Voice	4.87	4.71	4.66	74.3	85.7	96.0	90.4	83.2	99.8	90.5
LFG-3	4.73	4.40	4.45	78.1	85.5	94.7	92.2	88.5	98.3	89.9
Nemotron-3 Omni	4.75	4.57	4.58	71.4	82.3	93.0	91.1	88.7	100	89.4
Qwen3-Omni-Thinking	4.82	4.53	4.53	78.1	83.0	94.3	88.9	80.6	97.2	88.8

Table 1: VoiceBench results across turn-based AR and full-duplex models. Open-ended QA columns use a 5-point judge scale; the remaining columns and Overall use a 0–100 scale.

Model	Turn Taking			User Interruption		Pause Handling
Model	TOR-v1 $\uparrow$	TOR-v3 $\uparrow$	Latency $\downarrow$	TOR $\uparrow$	Latency $\downarrow$	Syn. $\downarrow$	Cand. $\downarrow$
Freeze-Omni	0.336	0.312	0.953	0.867	1.409	0.642	0.481
Moshi	0.941	0.925	0.265	1.000	0.257	0.985	0.980
PersonaPlex	0.992	0.985	0.070	1.000	0.400	0.584	0.662
Raon-SpeechChat	0.832	0.810	1.034	0.980	1.219	0.212	0.299
Nemotron-VC	0.820	0.798	0.448	1.000	0.480	0.153	0.255

Table 2: Conversation behavior results.“TOR-v1” and “TOR-v3” denote the Full-Duplex-Bench v1 and v3 (with disfluency), respectively. “Syn.” and “Cand.” stand for Synthetic and Candor audio. Closed-source models (e.g., GPT-Realtime) are excluded due to API-induced latency overheads.

Figure 3: Decoding efficiency and score context on AlpacaEval. TPF and TPS are measured at batch 1 on one NVIDIA GB300 GPU.

Setting	Open-ended QA			Knowledge and Reasoning				Ins. and Safety		Overall
Setting	Alp.	Com.	Wil.	SDQA	MMSU	OBQA	BBH	IFE.	Adv.	Overall
Diffuplex	4.71	4.36	4.48	60.2	77.4	89.7	83.4	77.2	97.5	84.1
Diffuplex ( text input )	4.81	4.53	4.81	73.4	84.7	93.8	94.8	81.3	100	90.1
Diffuplex ( AR decoding )	4.77	4.38	4.50	64.6	73.8	87.9	86.5	80.3	99.6	85.1
w/o text anchor	3.96	3.25	3.05	48.4	54.2	64.6	61.5	61.3	100	66.1
w/o Stage 1	1.00	1.00	1.01	0.0	24.3	31.2	44.8	15.1	93.8	29.9

Table 3: Ablation study on VoiceBench. “ text input ” denotes that user input is text as the test set has paired transcript. “ AR decoding ” denotes the response is generated by autoregressive decoding. Thinking is disabled for all models.

Reasoning		Knowledge		Instruction following
Big Bench Audio	GPQA-Diamond	Llama-Questions	MMLU-Pro	IFEval	IFBench	MultiChallenge
97.5	67.4	81.3	80.4	94.5	61.6	53.5

Table 4: Diffuplex across more benchmarks, grouped by capability. Shaded columns are spoken (audio-input) benchmarks; the others are text.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Output speed across different hardware (GPU). For RTX 6000, the Diffuplex is BF16 while GB300 and H100 are FP8.

Figure 5: Measured latency–quality trade-off for committing the first

n{=}8

tokens. Latency is measured from a supplied turn boundary to first audio, excluding the Stage 3 AR turn-decision delay. The left panel shows mean post-trigger latency and VoiceBench overall score; the right panel reports passes to prefix commitment and mean/median timing. The green star marks the reported

K{=}4

configuration; “No cap” uses the stock decoder.

Figure 6: VoiceBench subset scores under the same $n{=}8$ prefix setting. Printed values are absolute scores on the 0–100 scale; cell shading indicates the loss relative to unconstrained decoding in the same column. The green outline identifies the $K{=}4$ setting used for the Stage 3 result.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 301 papers

DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion
Oct 8, 2026Heeseung KimFull-Duplex Spoken Dialogue Systems Diffusion Model Inference Acceleration
Department of Artificial Intelligence University of Seoul Seoul, Republic of Korea
AdaptDuplex: From Static to Adaptive Full-Duplex Spoken Dialogue
Sep 24, 2026Zhiyang Zhou, Yingxin Shang, Zhou Wang +9Spoken Dialogue Systems Speech Language Models
Zuoyebang Education Technology (Beijing) Co., Ltd.
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
Apr 6, 2026Guan-Ting Lin, Chen Chen, Zhehuai Chen +1Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
National Taiwan University · NVIDIA

Oct 8, 2026J/K move · Enter open · S save

DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion

Heeseung Kim

Department of Artificial Intelligence University of Seoul Seoul, Republic of Korea

Full-Duplex Spoken Dialogue Systems Diffusion Model Inference Acceleration Masked Diffusion Models

Open paper AskPDF arXiv

Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve

1.46\times

and

1.59\times

deployment-path wall-clock speedups and

1.61\times

and

1.80\times

Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.

Abstract

Speech Language Models

Explore similar work

DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion

AdaptDuplex: From Static to Adaptive Full-Duplex Spoken Dialogue

Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency