Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends

Seonghyeon Go; Yongwoo Kim; Hyeonjin Cha; Jaeho Shin

cs.CLSep 27, 2026

Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends

Authors: Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha, Jaeho Shin

Organizations: Mindlogic

Abstract

Full-duplex spoken dialogue models can listen and speak simultaneously like the real-time dynamics of human conversation. For natural dialogue, the ability to search for external information in real-time is also an important capability. Many models remain trapped in parametric knowledge, leaving them unable to access real-time information and tool execution. Furthermore, even when Large Language Models (LLM) retrieve information, many duplex speech models process it within a compressed latent space rather than in its raw text form, which can lead to information loss from compression. To address this issue, we propose Context Spanning, a framework for information injection between a full-duplex speech model and an external LLM backend via real-time chunked prefill. The injected frame is encoded in a single forward pass inside the real-time frame budget. It feeds the retrieved information to the speech model as-is, enabling it to reason over the information independently and generate responses. With this approach, our model achieves high performance on Full-Duplex benchmarks and strong results on Question Answering tasks, demonstrating its conversation potential. Context Spanning shows that external information can be injected directly into a duplex speech model, introducing a new simple and powerful mechanism for duplex systems.

Figures & tables

Figure 1: Overall architecture of proposed method. Backend works asynchronously, and results are injected as Context Span.

Figure 2: Context Spanning on the Moshi architecture with an acoustic delay of $\tau=1$ . $u^{1}_{i}$ and $u^{q}_{i}$ denote user semantic and user acoustic tokens, $a^{1}_{i}$ and $a^{q}_{i}$ denote agent semantic and agent acoustic tokens, and $t_{i}$ denotes agent text tokens at timestep $i$ , where $q\in\{2,\dots,8\}$ .

$n$ (frames)	0	16	64	256	600
Latency ( ms )	$34.7\pm 0.4$	$56.1\pm 0.2$	$57.5\pm 0.2$	$62.0\pm 0.3$	$86.1\pm 0.3$
$P_{99}$ ( ms )	$35.2$	$56.3$	$57.7$	$62.3$	$86.3$

Table 1: Context Span processing latency.

P_{99}

denotes 99th percentile.

n=0

means no span, same as vanilla Moshi.

	Spoken QA								Mathematical Reasoning
	LlamaQ		WebQ		TriviaQA		HaluEval		AddSub		MultiArith		SinglEq		SVAMP		GSM8K
Model	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.
GLM-4-Voice		64.7		32.2		39.1		21.2		59.4		62.0		71.0		4.0		29.0
STITCH-S		73.3		50.2		50.0		–		81.7		87.9		91.7		72.2		56.7
MoshiRAG ${}_{\text{Gemma3}}$	83.0	80.3	71.5	67.2	73.7	69.6	42.0	36.3	76.6	61.7	87.1	69.0	83.2	68.2	74.1	55.0	66.2	33.9
MoshiRAG ${}_{\text{GPT-4.1}}$	87.8	80.6	77.7	68.9	86.8	78.2	61.2	51.3	87.9	64.8	87.1	76.0	89.6	72.9	80.5	61.1	70.8	43.2

Table 2: QA Benchmarks. ‘ref.’ denotes the accuracy % that correct reference document is provided; ‘resp.’ denotes the accuracy % that model response correctly. Underlined model metrics are reprinted from MoshiRAG, ‘-’ denotes values inaccessible.

Task	Metric	Ours	PPlex	Moshi	Gemini
Turn-Taking	TOR ( $\uparrow$ )	0.899	0.992	0.941	0.655
Turn-Taking	Latency, s ( $\downarrow$ )	0.064	0.070	0.265	1.301
Pause Handling	TOR, Candor ( $\downarrow$ )	0.824	0.662	0.980	0.310
Pause Handling	TOR, synthetic ( $\downarrow$ )	0.861	0.584	0.985	0.255
Backchannel	TOR ( $\downarrow$ )	0.782	0.327	1.000	0.091
	Frequency, per s ( $\uparrow$ )	0.150	0.025	0.001	0.012

Table 3: FullDuplexBench v1 results. Underlined models are reprinted from the PersonaPlex paper. PPlex denotes PersonaPlex, Gemini denotes Gemini Live 2.5.

Task	Metric	Ours	MoshiRAG	GPT	Gemini
Tool Use	Tool selection ( $\uparrow$ )	0.855	0.738	0.876	0.817
	Argument accuracy ( $\uparrow$ )	0.567	0.440	0.680	0.588
	Response quality ( $\uparrow$ )	0.411	0.255	0.792	0.718
	Pass rate ( $\uparrow$ )	0.470	0.280	0.600	0.540
Turn-Taking Dynamics	Take-turn rate, % ( $\uparrow$ )	95.0	94.0	96.0	78.0
	Latency, s ( $\downarrow$ )	5.83	7.69	6.89	4.25

Table 4: FullDuplexBench v3 results. Underlined systems are reprinted from the original benchmark paper. GPT denotes GPT-Realtime, Gemini denotes Gemini Live 3.1.

Explore similar work

May 11, 2026cs.CL

How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

Full-duplex spoken dialogue requires a model to keep listening while generating its own spoken response. This is challenging for large language models (LLMs), which are designed to extend a single coherent sequence and do not naturally support user input arriving during generation. We argue that how the user stream is routed into the LLM is therefore a key architectural question for full-duplex modeling. To study this question, we extend a text-only LLM into a unified full-duplex spoken dialogue system and compare two routing strategies under a shared training pipeline: (i) channel fusion, which injects the user stream directly into the LLM input, and (ii) cross-attention routing, which keeps the user stream as external memory accessed through cross-attention adapters. Experiments on spoken question answering and full-duplex interaction benchmarks reveal a clear tradeoff. Channel fusion yields stronger semantic grounding and consistently better question-answering performance. However, under semantically overlapping conditions such as user interruptions, it is more vulnerable to context corruption: if the model fails to stop in time, the overlapping user stream can interfere with ongoing generation and lead to semantically incoherent continuations. Cross-attention routing underperforms on question answering, but better preserves the LLM generation context and is more robust to this failure mode. These results establish user-stream routing as a central design axis in full-duplex spoken dialogue and offer practical guidance on the tradeoff between semantic integration and context robustness. We provide a demo page for qualitative inspection.

Hui Lu, Xueyuan Chen, Huimeng Wang +4

The Chinese University of Hong Kong · Tsinghua University · SenseTime Research

Full-Duplex Speech Models Dialogue Question-Answer Pairs

Ask PDF

Jun 12, 2026cs.CL

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Full-Duplex Speech Models Speech Language Models Autoregressive Language Models Chatbots

Ask PDF

Sep 16, 2026cs.CL

A frontend-backend architecture for tool calls in full-duplex speech models

Full-duplex speech-to-speech (S2S) models provide natural, low-latency conversational interaction and would benefit from the ability to use external tools and complete voice-agent tasks. We propose a frontend-backend architecture where a duplex speech-to-text frontend learns to emit a delegation token and forwards streaming ASR transcripts to a text-based backend LLM for tool calls. Tool-call results from the backend are injected back into the frontend through a lightweight prefill-and-repeat mechanism and then synthesized using streaming TTS to the user. Our approach largely preserves regular duplex turn-taking, interruption handling, and low-latency interaction as it requires minimal modifications to the frontend model. In a single-turn tool-call evaluation, our system achieves 92-97% tool-call recall, competitive tool-call prediction performance, and 81.2% accuracy in rejecting irrelevant calls. When equipped with a larger backend (e.g., Qwen3-235B-A22B), our system achieves competitive results on Full-Duplex-Bench-V3 compared to open and closed source models, and significantly outperforms GPT-realtime-mini and Qwen3-Omni-30B-A3B-Instruct on EVA-Bench. These results demonstrate that backend delegation is an effective and modular approach for combining natural duplex speech interaction with strong agentic tool-call capabilities.

Ke Hu, Slyne Deng, Chen Chen +10

NVIDIA, USA

Full-Duplex Speech Models Call Delegation

Ask PDF