Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Figures & tables
Figure 1: Comparison between Rubato and prior methods under varying decoding speeds. Given an identical question and reasoning trace, varying inference speeds stretch or compress thinking tokens along the wall-clock timeline. The dotted line denotes a reference timestamp. Prior models rely on a static decoding rate where slow execution stalls real-time audio streams and fast execution wastes available compute. In contrast, Rubato dynamically adapts by holding the conversational floor during slower decoding and accelerating answer delivery when hardware permits.
Figure 2: Positional index arrangement for Rubato vs. Single-Clock. An example 3-token thinking event ( T1 to T3 ) starting at frame 1 spans a variable number of frames based on Bf . While Single-Clock applies a continuous index that arbitrarily distorts the distance between real-time tokens, Rubato assigns independent clocks that advance exclusively on designated tokens (full color) and pause elsewhere (faint). This guarantees consecutive real-time tokens remain strictly equidistant, fully decoupling the audio and reasoning timelines.
Figure 3: Spoken reasoning performance across varying inference thinking budgets. Models are trained exclusively on the two extreme rates ( B=1 and B=∞ , marked by bold ticks), meaning all intermediate points evaluate zero-shot interpolation. Filled and hollow markers denote fixed and random per-frame budgets (plotted at their mean) respectively. Panels display 95% Wilson intervals for accuracy and medians for latency. We report Accuracy (Acc.), Time-to-Answer (TTA), and Time-to-First-Response (TTFR) on two single-turn benchmarks, alongside isolated results for both questions in the overlapped setting.
Figure 4: Spoken-filler quality across thinking budgets. To evaluate the quality of the system audio filler, we report the UTMOS score and speaker similarity. From the figure we see that Rubato -Full and Rubato -Gated maintain the audio quality while Rubato -Split degrades significantly as the thinking budget is interpolated.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Spoken GSM8K
Spoken BIG-Bench Hard
Source benchmark
GSM8K-Aug-NL
BIG-Bench Hard
( Wu et al., 2025 )
( Srivastava et al., 2023 )
# of Conversations
365,691
54,330
Hours of audio
2,474
667
Length of Reasoning Trace (tok)
221 ± 200
399 ± 236
B=1 layout
Appendix
Table 1: Detailed statistics for the constructed training datasets. Values represent the mean ± s.d. Both Time-to-First-Response (TTFR) and Time-to-Answer (TTA) are measured from the end of the user’s question.
Spoken GSM8K
Question
What is ten less than half of 60?
Reasoning
Here’s the step-by-step calculation: 1. Find half of 60: 60/2=30 . 2. Find ten less than that number (30): 30−10=20 .
Answer
20
Spoken BIG-Bench Hard: formal_fallacies
Question
Here comes a perfectly valid argument: First, being an ancestor of Erik is sufficient for being a schoolmate of Lawrence. Second, Pedro is an ancestor of Erik. From this follows: Pedro is a schoolmate of Lawrence. Is the argument, given the explicitly stated premises, deductively valid or invalid? Options: valid, invalid. Answer the question.
Reasoning
The user wants me to evaluate the deductive validity of a given argument. Argument Structure: Premise 1: Being an ancestor of Erik → Being a schoolmate of Lawrence (A → B) Premise 2: Pedro is an ancestor of Erik (A) Conclusion: Pedro is a schoolmate of Lawrence (B) This structure is: 1. A→B 2. A∴B This is the form of Modus Ponens, which is a deductively valid argument form. Therefore, the argument is valid.
Appendix
Table 2: Examples of Spoken Reasoning Tasks We show one example for each task across Spoken GSM8K and Spoken BIG-Bench Hard . The reasoning traces come from Gemma4’s internal reasoning chain.
KV Cache Length (toks)
512
1,024
2,048
4,096
8,192
16,384
Head Dim = 256
Single-Clock
60.6 ± 0.2
61.0 ± 0.2
61.8 ± 0.2
63.2 ± 0.4
69.0 ± 0.3
70.7 ± 0.3
Rubato -Full (multi-KV)
64.4 ± 0.2
62.2 ± 0.2
62.9 ± 0.3
64.5 ± 0.3
70.0 ± 0.3
72.4 ± 0.8
Rubato -Full (wide cache)
62.9 ± 0.2
63.5 ± 0.3
64.0 ± 0.3
65.7 ± 0.3
72.7 ± 0.9
83.0 ± 0.5
Head Dim = 512
Appendix
Table 3: Decode speed comparison. Wall GPU time per decode step in microseconds ( μ s) benchmarked with batch size 1 on NVIDIA H200 using FlashInfer ( Ye et al., 2025 ) .
Spoken GSM8K
BIG-Bench Audio
Overlapped
B=1
B=8
B=∞
R[1,16]
B=1
B=8
B=∞
R[1,16]
B=1
B=8
B=∞
R[1,16]
Accuracy
Single-Clock
46.7
12.6
50.0
15.5
56.6
42.3
71.4
47.0
24.0
3.0
28.1
2.8
RUBATO-Split
44.4
42.2
45.3
42.7
77.1
76.5
80.6
76.2
21.1
3.9
17.7
4.6
RUBATO-Gated
47.9
49.0
48.1
48.8
78.3
79.5
79.2
78.9
20.4
10.7
17.4
12.3
RUBATO-Full
49.3
49.8
51.1
52.7
87.1
88.8
86.1
88.9
24.0
21.4
25.3
18.8
Appendix
Table 4: Extreme Thinking Budget results on three benchmarks. We report B∼U[1,16] alongside with other fixed thinking budgets. We include B=8 , as this can be viewed it has the same expected thinking budget as B∼U[1,16] .
Gemma-4 E4B-IT
Llama 3.2 1B
GPU
BW (GB/s)
tok/s
B
B75
tok/s
B
B75
Data center
B200 †
8,000
861
64
47
3236
254
190
H200 SXM †
4,800
517
37
27
1942
151
112
H100 NVL
3,900
420
29
21
1578
122
90
H100 SXM
3,350
361
24
17
1355
104
77
Appendix
Table 5: Estimated thinking budgets B for common NVIDIA GPUs. We estimate the decoding speed and resulting thinking budget for our Gemma-4 E4B-IT backbone and a smaller Llama 3.2 1B model assuming a batch size of 1. The B columns reflect theoretical maximums based on pure vendor arithmetic. The B75 columns represent a conservative budget assuming 75% Model Bandwidth Utilization, matching the 65% to 85% range observed in real-world deployments. Unmarked bandwidths originate from primary NVIDIA product pages. We denote secondary sources consistent with vendor claims using † .
Figure 5: Mixing proportions of the two training layouts. Each curve represents a training corpus with a varying proportion of B=1 data (43.9% to 0.4%). Following Fig. 3 , bold ticks mark the trained budgets, while filled and hollow markers indicate fixed and random per-frame budgets.
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.
Full-duplex spoken dialogue requires simultaneous listening and speaking at sub-second latency, under conversational timing and cognitive demands that change moment to moment. Yet current models mostly impose static operating points, lacking a systematic mechanism for adaptive decisions. We present AdaptDuplex, which extends Qwen3-Omni with such a mechanism, co-designed across three layers. A compact token-level protocol represents every window as a canonical sequence, trains dual-stream alignment through a bounded text lead over speech, and exposes every behavioral decision as an explicit token for training-free runtime control via logits bias. Adaptive mechanisms dynamically predict among discrete window durations and augment direct response as needed with non-blocking cognitive consolidation and multi-flight external reasoning. A progressive pipeline introduces these behaviors through a three-stage Thinker curriculum, then Talker-only and joint SFT, with GRPO as a preliminary increment. On Full-Duplex-Bench v1 and v1.5, AdaptDuplex outperforms DuplexOmni on 18 of 21 comparable turn-taking, overlap-behavior, and timing metrics and MiniCPM-o 4.5 on 19 of 22, with gains in both interaction decisions and response timing. On human-recorded HumDial-FDBench, it attains the top Final score (69.6) of the compared duplex models.
Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve 1.46× and 1.59× deployment-path wall-clock speedups and 1.61× and 1.80× Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.
Heeseung Kim
Department of Artificial Intelligence University of Seoul Seoul, Republic of Korea