Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.
Figures & tables
Figure 1: Illustration of the two behaviors on a shared timeline (user above, agent below; schematic: timings and text are illustrative). (a) A free-running conversation: the agent backchannels (“mm-hm”, “yeah”, in the backchannel color) while the user is still talking and takes the turn when the user finishes, with no external turn detector. (b) A user interruption: the agent yields the floor when the user cuts in (silence-start token, SS ) and takes it back to answer the new request (silence-end token, SE ). SE / SS markers are drawn at the start and end of each agent turn.
Figure 2: Overview. A frozen, general-purpose multimodal LLM (Qwen3-Omni), which only talks in turns, becomes the thinker of a full-duplex system through four training stages. Each stage card shows what the model reads and emits at that stage as one cell per frame (user codec tokens, agent words, pause, SE / SS , and PAD cells), which modules it trains, and which model anchors its KL term; the frozen core is never changed. The resulting thinker emits agent text and timing tokens frame by frame, and the talker turns them into speech, so that the agent listens and speaks at the same time.
Figure 3: Data flow and frame-level token structure. A frozen codec encoder turns the user’s waveform into per-frame audio tokens. The thinker reads the user-audio stream together with its own agent-text stream (pad, text, and control tokens on one lane per frame) and extends the text stream by one token per frame. The talker reads the agent-text stream and its own previous audio tokens and extends the agent-audio stream by one frame; a frozen codec decoder renders it as a waveform. The control tokens SS and SE open and close the agent’s silences, and pause tokens mark short gaps; the example includes a short backchannel (“Mhm”) produced while the user is still speaking.
Figure 4: The thinker’s turn-taking state machine. In Silence the legal tokens are PAD (stay silent) and SE (take the floor, entering Content after one automatically fed EPAD frame). In Content they are word pieces and pause0 (stay), pausem with m>0 (enter Pause ), and SS (yield, entering Silence ). Pause samples nothing: it feeds its queued PAD and EPAD frames and returns to Content when the queue is empty.
Stage, run
Gap closed
Examples
Hours
Steps
Trained
Reference ( β )
GPU-h
1, read-out
read codec audio
17,210
≈ 270
2,000
audio adapter
–
2.6
1, distillation
3,911
6.3
733
audio adapter
native-audio LLM
0.5
2
pause timing
760
1.0
285
adapters, 15 embeddings
–
1.2
3, self-distillation
dual-channel format
760
2.1
285
adapters, 17 embeddings
Stage 2 (0.8)
0.9
3, multi-turn retrain
room tone, multi-turn
12,146
406
1,012
+ channel fusion, audio adapter
previous run (0.4)
3.2
4, barge-in
interruption
36,621
1,058
3,000
+ channel fusion, audio adapter
Stage 3 (0.4)
9.1
Table 1: The thinker’s conversion curriculum. Parameters not listed stay frozen, including the entire LLM, its token embeddings, and its language-modeling head. “Embeddings” are the new token embeddings for the timing tokens. “Reference” is the model that anchors the KL term of Equation 9 . Hours are the duration of the user-channel timeline of the training data; GPU-hours are measured on one NVIDIA B200 and exclude data generation and evaluation. One row per training run, grouped by stage.
Pause handling
Backchannel
Smooth turn-taking
User interruption
Model
Synth. TOR ↓
Candor TOR ↓
TOR ↓
Freq ↑
JSD ↓
TOR ↑
Lat. ↓
TOR ↑
Relev. ↑
Lat. ↓
Open-source (self-hosted)
PersonaPlex
0.584
0.662
0.327
0.025
0.649
0.992
0.070
1.000
4.210
0.400
dGSLM
0.934
0.935
0.691
0.015
0.934
0.975
0.352
0.917
0.201
2.531
Moshi
0.985
0.980
1.000
0.001
0.957
0.941
0.265
1.000
0.765
0.257
Freeze-Omni
0.642
0.481
0.636
0.001
0.997
0.336
0.953
0.867
3.615
1.409
Table 2: Full-Duplex-Bench v1.0. Baseline results are from the benchmark’s public results table ( Lin et al., 2025b ) . Arrows give the better direction. We compare with the open-source systems: bold marks the best value in that block, and the closed-source, API-based rows are for reference only. Relevance is rated 0–5 by the benchmark’s LLM judge (GPT-4-turbo in its released evaluation script), run unmodified, as for the other rows. Latencies are in seconds. Our row is the mean over three seeds; standard deviations are in Appendix C .
User interruption
User backchannel
Side conversation
Background speech
Model
Resp. ↑
Res. ↓
Stop ↓ / Resp. ↓ lat.
Resp. ↓
Res. ↑
Stop ↑ / Resp. ↓ lat.
Resp. ↓
Res. ↑
Stop ↑ / Resp. ↓ lat.
Resp. ↓
Res. ↑
Stop ↑ / Resp. ↓ lat.
Open-source (self-hosted)
Freeze-Omni
0.72
0.12
1.42 / 1.35
0.07
0.80
0.66 / 2.16
0.58
0.25
1.39 / 1.32
0.63
0.25
0.98 / 1.60
Moshi
0.50
0.26
1.16 / 1.47
0.02
0.06
0.42 / 3.00
0.20
0.19
0.87 / 2.38
0.21
0.07
0.54 / 1.62
Ours
0.42
0.10
0.75 / 0.64
0.02
0.24
0.68 / 1.33
0.08
0.25
0.84 / 0.62
0.06
0.23
0.83 / 0.64
Closed-source / API-based (reference)
Table 3: Full-Duplex-Bench v1.5. For each overlap scenario, the share of the model’s post-overlap speech classified as a response to the overlap (Resp.) or a resumption of its earlier utterance (Res.), and the stop and response latencies in seconds after the overlap begins. Baseline results are from Lin et al. (2025a, Tables 2 and 3) ; the uncertain and unknown behavior classes are omitted. Which behavior is desirable depends on the scenario: a model should respond to the user’s interruption but keep going after a backchannel, a side conversation, or background speech, so the arrows mark the desirable direction in each scenario (stop late, if at all, in the keep-going scenarios; respond quickly whenever a response is given). We compare with the open-source systems: bold marks the best value in that block, and the closed-source, API-based rows are for reference only. Our row is the mean over three seeds; standard deviations are in Appendix C .
System
Spoken task (%) ↑
Timing (%) ↑
Whisper & volume (%) ↑
Grounded (%) ↑
Coherence (0–100) ↑
Open-source (self-hosted)
Moshi
6 (3/50)
41 (54/132)
15 (3/20)
14 (7/50)
16
PersonaPlex
18 (9/50)
68 (90/132)
35 (7/20)
30 (15/50)
28
Freeze-Omni
8 (4/50)
57 (78/138)
15 (3/20)
35 (18/51)
39
Ours
26 (13/50)
63 (88/140)
5 (1/20)
53 (27/51)
47
Closed-source / API-based (reference)
Table 4: InteractionBench. Percentages are over the items in each dimension (counts in parentheses); coherence is the mean judge score over the ten groundedness conversations. Baseline results are from He et al. (2026, Table 4) . We compare with the open-source systems: bold marks the best value in that block, and the closed-source, API-based rows are for reference only. Our row is a single decoding seed; its groundedness rate is over the 51 valid probes (one probe was excluded as invalid). Our row is graded on the thinker’s text stream, with each word timed at the frame in which the thinker emits it; the grader components that read audio use the talker’s output, cut at those times.
Figure 5: Adapting under a room-tone floor. (a) Question answering after the self-distillation run of Stage 3 (13-item probe, judge score 1 to 5, mean and standard deviation over three seeds) when the whole pipeline is trained with a floor, as a function of which input modules are trainable; dotted lines are the clean-trained model on clean audio and with the floor; bar labels are the means. (b) Mean L2 norm of the user-audio branch entering the channel-fusion layer on 40 held-out noisy examples, for frames with user speech and for answer-phase frames that contain only room tone.
Figure 6: Text lead and talker intelligibility: word error rate on the 150 held-out dialogues, split into deletions, substitutions, and insertions (mean of three decoding seeds; the standard deviation of the total across seeds is at most 0.010). The talkers at L=0 , 2, 4, and 6 differ only in the text lead and are trained from PersonaPlex on the same data for the same number of steps; the talker at L=3 is trained with the recipe of Section 4.3 , a separate lineage.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Metric
Decoding seeds
Mean ± SD
Pause handling
Synth. TOR
0.175
0.168
0.168
0.170±0.004
Candor TOR
0.273
0.269
0.259
0.267±0.007
Backchannel
TOR
0.146
0.127
0.164
0.145±0.018
Freq
0.071
0.070
0.077
0.073±0.004
JSD
0.751
0.772
0.737
0.753±0.018
Smooth turn-taking
TOR
1.000
1.000
0.992
0.997±0.005
Appendix
Table 5: Full-Duplex-Bench v1.0, our row of Table 2 by decoding seed: the three per-seed values, their mean, and the sample standard deviation.
Scenario
Metric
Decoding seeds
Mean ± SD
n
User interruption
Resp.
0.39
0.40
0.47
0.42±0.04
Res.
0.12
0.10
0.08
0.10±0.02
Stop lat.
0.78
0.74
0.74
0.75±0.03
192–194
Resp. lat.
0.63
0.63
0.65
0.64±0.01
193–195
User backchannel
Resp.
0.03
0.01
0.02
0.02±0.01
Res.
0.24
0.30
0.19
0.24±0.06
Appendix
Table 6: Full-Duplex-Bench v1.5, our row of Table 3 by decoding seed: the three per-seed values, their mean, and the sample standard deviation. Latencies are event-anchored means in seconds over the cases in which the event occurs; n is the range of that count over the seeds.
Dimension
Scenario
Pass
Diagnostic
Timing
Backchannel
0.20 (4/20)
floor-take 0.15; 0.20 backchannels per item
Pause
0.47 (14/30)
barge-in 0.53
Turn-taking
1.00 (30/30)
median latency 640 ms; responded 1.00
User backchannel
0.53 (16/30)
derail 0.47
Interruption
0.80 (24/30)
yielded 1.00; addressed 0.80
Spoken task
Logic
3/10
Appendix
Table 7: InteractionBench, our system by scenario (seed 0). Timing: pass rate per scenario and the benchmark’s diagnostic for it. Spoken tasks: items passed out of ten; the alternating-count task (one item) is scored separately and is not among the 50 items of Table 4 . Groundedness: probes answered correctly out of the 51 valid probes, and the coherence score.
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
Full-duplex spoken dialogue requires simultaneous listening and speaking at sub-second latency, under conversational timing and cognitive demands that change moment to moment. Yet current models mostly impose static operating points, lacking a systematic mechanism for adaptive decisions. We present AdaptDuplex, which extends Qwen3-Omni with such a mechanism, co-designed across three layers. A compact token-level protocol represents every window as a canonical sequence, trains dual-stream alignment through a bounded text lead over speech, and exposes every behavioral decision as an explicit token for training-free runtime control via logits bias. Adaptive mechanisms dynamically predict among discrete window durations and augment direct response as needed with non-blocking cognitive consolidation and multi-flight external reasoning. A progressive pipeline introduces these behaviors through a three-stage Thinker curriculum, then Talker-only and joint SFT, with GRPO as a preliminary increment. On Full-Duplex-Bench v1 and v1.5, AdaptDuplex outperforms DuplexOmni on 18 of 21 comparable turn-taking, overlap-behavior, and timing metrics and MiniCPM-o 4.5 on 19 of 22, with gains in both interaction decisions and response timing. On human-recorded HumDial-FDBench, it attains the top Final score (69.6) of the compared duplex models.
Full-duplex spoken dialogue requires a model to keep listening while generating its own spoken response. This is challenging for large language models (LLMs), which are designed to extend a single coherent sequence and do not naturally support user input arriving during generation. We argue that how the user stream is routed into the LLM is therefore a key architectural question for full-duplex modeling. To study this question, we extend a text-only LLM into a unified full-duplex spoken dialogue system and compare two routing strategies under a shared training pipeline: (i) channel fusion, which injects the user stream directly into the LLM input, and (ii) cross-attention routing, which keeps the user stream as external memory accessed through cross-attention adapters. Experiments on spoken question answering and full-duplex interaction benchmarks reveal a clear tradeoff. Channel fusion yields stronger semantic grounding and consistently better question-answering performance. However, under semantically overlapping conditions such as user interruptions, it is more vulnerable to context corruption: if the model fails to stop in time, the overlapping user stream can interfere with ongoing generation and lead to semantically incoherent continuations. Cross-attention routing underperforms on question answering, but better preserves the LLM generation context and is more robust to this failure mode. These results establish user-stream routing as a central design axis in full-duplex spoken dialogue and offer practical guidance on the tradeoff between semantic integration and context robustness. We provide a demo page for qualitative inspection.
Hui Lu, Xueyuan Chen, Huimeng Wang +4
The Chinese University of Hong Kong · Tsinghua University · SenseTime Research
Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-SLMs) in general capabilities. We propose Dialogue-to-Duplex (D2), a framework that converts pretrained HD-SLMs into high-capability FD-SLMs at fine temporal granularity. D2 adapts the original audio encoder to streaming through decoder-free representation distillation and learns full-duplex interaction through supervised fine-tuning. We systematically study the effects of chunk size on full-duplex interaction, general capabilities, and inference computation. We apply D2 to two distinct HD-SLM architectures, Qwen3-Omni and LLaMA-Omni 2. At a 12.5Hz update rate, Qwen3-Omni-D2 achieves 99.2% turn-taking success with 521-ms latency, 99.5% interruption success, and a state-of-the-art VoiceBench score of 70.1.
Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury +5