Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Figures & tables
Figure 1: Comparison between Rubato and prior methods under varying decoding speeds. Given an identical question and reasoning trace, varying inference speeds stretch or compress thinking tokens along the wall-clock timeline. The dotted line denotes a reference timestamp. Prior models rely on a static decoding rate where slow execution stalls real-time audio streams and fast execution wastes available compute. In contrast, Rubato dynamically adapts by holding the conversational floor during slower decoding and accelerating answer delivery when hardware permits.
Figure 2: Positional index arrangement for Rubato vs. Single-Clock. An example 3-token thinking event ( T1 to T3 ) starting at frame 1 spans a variable number of frames based on Bf . While Single-Clock applies a continuous index that arbitrarily distorts the distance between real-time tokens, Rubato assigns independent clocks that advance exclusively on designated tokens (full color) and pause elsewhere (faint). This guarantees consecutive real-time tokens remain strictly equidistant, fully decoupling the audio and reasoning timelines.
Figure 3: Spoken reasoning performance across varying inference thinking budgets. Models are trained exclusively on the two extreme rates ( B=1 and B=∞ , marked by bold ticks), meaning all intermediate points evaluate zero-shot interpolation. Filled and hollow markers denote fixed and random per-frame budgets (plotted at their mean) respectively. Panels display 95% Wilson intervals for accuracy and medians for latency. We report Accuracy (Acc.), Time-to-Answer (TTA), and Time-to-First-Response (TTFR) on two single-turn benchmarks, alongside isolated results for both questions in the overlapped setting.
Figure 4: Spoken-filler quality across thinking budgets. To evaluate the quality of the system audio filler, we report the UTMOS score and speaker similarity. From the figure we see that Rubato -Full and Rubato -Gated maintain the audio quality while Rubato -Split degrades significantly as the thinking budget is interpolated.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Spoken GSM8K
Spoken BIG-Bench Hard
Source benchmark
GSM8K-Aug-NL
BIG-Bench Hard
( Wu et al., 2025 )
( Srivastava et al., 2023 )
# of Conversations
365,691
54,330
Hours of audio
2,474
667
Length of Reasoning Trace (tok)
221 ± 200
399 ± 236
B=1 layout
Appendix
Table 1: Detailed statistics for the constructed training datasets. Values represent the mean ± s.d. Both Time-to-First-Response (TTFR) and Time-to-Answer (TTA) are measured from the end of the user’s question.
Spoken GSM8K
Question
What is ten less than half of 60?
Reasoning
Here’s the step-by-step calculation: 1. Find half of 60: 60/2=30 . 2. Find ten less than that number (30): 30−10=20 .
Answer
20
Spoken BIG-Bench Hard: formal_fallacies
Question
Here comes a perfectly valid argument: First, being an ancestor of Erik is sufficient for being a schoolmate of Lawrence. Second, Pedro is an ancestor of Erik. From this follows: Pedro is a schoolmate of Lawrence. Is the argument, given the explicitly stated premises, deductively valid or invalid? Options: valid, invalid. Answer the question.
Reasoning
The user wants me to evaluate the deductive validity of a given argument. Argument Structure: Premise 1: Being an ancestor of Erik → Being a schoolmate of Lawrence (A → B) Premise 2: Pedro is an ancestor of Erik (A) Conclusion: Pedro is a schoolmate of Lawrence (B) This structure is: 1. A→B 2. A∴B This is the form of Modus Ponens, which is a deductively valid argument form. Therefore, the argument is valid.
Appendix
Table 2: Examples of Spoken Reasoning Tasks We show one example for each task across Spoken GSM8K and Spoken BIG-Bench Hard . The reasoning traces come from Gemma4’s internal reasoning chain.
KV Cache Length (toks)
512
1,024
2,048
4,096
8,192
16,384
Head Dim = 256
Single-Clock
60.6 ± 0.2
61.0 ± 0.2
61.8 ± 0.2
63.2 ± 0.4
69.0 ± 0.3
70.7 ± 0.3
Rubato -Full (multi-KV)
64.4 ± 0.2
62.2 ± 0.2
62.9 ± 0.3
64.5 ± 0.3
70.0 ± 0.3
72.4 ± 0.8
Rubato -Full (wide cache)
62.9 ± 0.2
63.5 ± 0.3
64.0 ± 0.3
65.7 ± 0.3
72.7 ± 0.9
83.0 ± 0.5
Head Dim = 512
Appendix
Table 3: Decode speed comparison. Wall GPU time per decode step in microseconds ( μ s) benchmarked with batch size 1 on NVIDIA H200 using FlashInfer ( Ye et al., 2025 ) .
Spoken GSM8K
BIG-Bench Audio
Overlapped
B=1
B=8
B=∞
R[1,16]
B=1
B=8
B=∞
R[1,16]
B=1
B=8
B=∞
R[1,16]
Accuracy
Single-Clock
46.7
12.6
50.0
15.5
56.6
42.3
71.4
47.0
24.0
3.0
28.1
2.8
RUBATO-Split
44.4
42.2
45.3
42.7
77.1
76.5
80.6
76.2
21.1
3.9
17.7
4.6
RUBATO-Gated
47.9
49.0
48.1
48.8
78.3
79.5
79.2
78.9
20.4
10.7
17.4
12.3
RUBATO-Full
49.3
49.8
51.1
52.7
87.1
88.8
86.1
88.9
24.0
21.4
25.3
18.8
Appendix
Table 4: Extreme Thinking Budget results on three benchmarks. We report B∼U[1,16] alongside with other fixed thinking budgets. We include B=8 , as this can be viewed it has the same expected thinking budget as B∼U[1,16] .
Gemma-4 E4B-IT
Llama 3.2 1B
GPU
BW (GB/s)
tok/s
B
B75
tok/s
B
B75
Data center
B200 †
8,000
861
64
47
3236
254
190
H200 SXM †
4,800
517
37
27
1942
151
112
H100 NVL
3,900
420
29
21
1578
122
90
H100 SXM
3,350
361
24
17
1355
104
77
Appendix
Table 5: Estimated thinking budgets B for common NVIDIA GPUs. We estimate the decoding speed and resulting thinking budget for our Gemma-4 E4B-IT backbone and a smaller Llama 3.2 1B model assuming a batch size of 1. The B columns reflect theoretical maximums based on pure vendor arithmetic. The B75 columns represent a conservative budget assuming 75% Model Bandwidth Utilization, matching the 65% to 85% range observed in real-world deployments. Unmarked bandwidths originate from primary NVIDIA product pages. We denote secondary sources consistent with vendor claims using † .
Figure 5: Mixing proportions of the two training layouts. Each curve represents a training corpus with a varying proportion of B=1 data (43.9% to 0.4%). Following Fig. 3 , bold ticks mark the trained budgets, while filled and hollow markers indicate fixed and random per-frame budgets.
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.