cs.CLNov 10, 2026

An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue

Authors: Tairan Chen, Richard Yucheng He, Yihang Liu, Baodong Cao, Chen Xu

Organizations: AveraLabs

Abstract

Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. AdaptDuplex: From Static to Adaptive Full-Duplex Spoken Dialogue

    Sep 24, 2026Zhiyang Zhou, Yingxin Shang, Zhou Wang +9Spoken Dialogue SystemsSpeech Language Models

  2. How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

    May 11, 2026Hui Lu, Xueyuan Chen, Huimeng Wang +4Language Model RobustnessFull-Duplex Spoken Dialogue Systems

  3. D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

    Oct 12, 2026Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury +5Speech Language ModelsFull-Duplex Spoken Dialogue Systems