Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.
Figures & tables
Figure 1: Illustration of turn-based models versus full-duplex models. The former must explicitly switch between speaking and listening states, while the latter can concurrently speak and listen.
Figure 2: Different types of delays in human-machine conversations. End-to-end keyword delay (E2EKD) measures the time between the end of the user’s question and the most informative word in the response. Retrieval delay measures how long it takes for the back end to provide relevant information.
Figure 3: Illustration of the front-end and back-end components in Moshi R A G . When the model needs external information, it outputs a ⟨ret⟩ token. The conversation transcript is sent to the back end which operates asynchronously. Once ready, the result is injected into Moshi which then adapts its response with no interruption.
Figure 4: Text and audio token streams of the inputs and outputs of Moshi R A G . Front-end Moshi receives at all time its previous step token predictions and the user speech tokens. When the retrieval result is ready, its representation is summed with the embeddings from other token streams and ingested over a number of time steps.
Figure 5: Histograms of the distributions of retrieval and keyword delays. Our training data covers a wider range of retrieval delays than observed in practice at inference. Keyword delay is almost always longer than practical retrieval delay, leaving sufficient time to integrate retrieved information to improve the model’s answer.
Model (Base LM Size)
Accuracy ( % )
Delay (sec.)
Computation
LlamaQ
WebQ
TriviaQA
HaluEval
TTFAT
KD
E2EKD
FLOPs/sec.
ref. |resp.
ref. |resp.
ref. |resp.
ref. |resp.
GPT-4o Audio ( OpenAI, 2024 )
88.4
81.0
90.6
68.7
-
5.5
-
-
GLM-4-Voice (9B) ( Zeng et al., 2024 )
64.7
32.2
39.1
21.2
0.3
4.2
4.4
0.74
Freeze-Omni (7B) ( Wang et al., 2025c )
72.0
44.7
53.9
14.0
1.2
5.2
6.5
0.18
MinMo (7B) ( Chen et al., 2025a )
78.9
55.0
48.3
-
-
-
-
-
Table 1: Results for factuality evaluation, delay metrics, and computation consumption of various models. Darker color indicates better performance. Underlined numbers are values reported in the original paper of each model. “-” indicates values inaccessible since part or all of the models are not released. “ref.” and “resp.” show the correctness of retrieved reference and Moshi R A G ’s final response, respectively.
Model
Pause
Backchannel
Turn Taking
User Interruption
TOR s↓
TOR c↓
TOR ↓
Freq. (/s) ↑
JSD ↓
TOR ↑
Latency (s) ↓
TOR ↑
GPT Score ↑
Latency (s) ↓
dGSLM ( Nguyen et al., 2023 )
0.93
0.94
0.69
0.015
0.93
0.98
0.35
0.92
0.20
2.53
Freeze-Omni
0.64
0.48
0.64
0.001
1.00
0.34
0.95
0.87
3.62
1.41
Gemini ( Comanici and others, 2025 )
0.26
0.31
0.09
0.012
0.90
0.66
1.30
0.89
3.38
1.18
MoshiRAG
0.32
0.56
0.64
0.010
0.94
0.83
0.18
0.85
3.75
1.02
Vanilla Moshi
0.99
0.98
1.00
0.001
0.96
0.94
0.27
1.00
0.77
0.26
Table 2: Evaluation results on Full-Duplex-Bench ( Lin et al., 2025 ) . Underlined numbers are values reported in the original paper of each model. TOR s and TOR c in the Pause track correspond to the synthetic subset and the Candor subset, respectively.
Model
Accuracy ( % )
AddSub
MultiArith
SinglEq
SVAMP
GSM8K
ref. |resp.
ref. |resp.
ref. |resp.
ref. |resp.
ref. |resp.
GLM-4-Voice
59.4
62.0
71.0
4.0
29.0
STITCH-S
81.7
87.9
91.7
72.2
56.7
MoshiRAG
76.6 |61.7
87.1 |69.0
83.2 |68.2
74.1 |55.0
66.2 |33.9
MoshiRAG summ.
62.0
73.1
66.4
57.5
51.2
Table 3: Evaluation results on out-of-domain mathematical reasoning datasets. Underlined numbers are values reported in the STITCH paper ( Chiang et al., 2026 ) . MoshiRAG summ. means that the reference is summarized before being injected into Moshi.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Subset
LLM
HotpotQA
NQ
TriviaQA
Total
(# conv / hrs)
(# conv / hrs)
(# conv / hrs)
(# conv / hrs)
(# conv / hrs)
Multi-turn v1
5457 (231)
90208 (2201)
305077 (8083)
76068 (1866)
476810 (12381)
Multi-turn v2
5549 (236)
90034 (2433)
303359 (9264)
75515 (2097)
474457 (14030)
Multi-turn v3
5550 (298)
90208 (3459)
306754 (12701)
76061 (2947)
478573 (19405)
Single-turn
90172 (438)
305364 (333)
471536 (1954)
Total
(765)
(8531)
(31231)
(7243)
1901376 (47770)
Appendix
Table 4: Moshi R A G training data statistics. Values are reported as: number of conversations (hours) .
Subset
HotpotQA
NQ
TriviaQA
(# conv / hrs)
(# conv / hrs)
(# conv / hrs)
Multi-turn v1
7403 (181)
7830 (208)
9956 (245)
Multi-turn v2
7391 (199)
7814 (240)
9939 (271)
Multi-turn v3
7403 (283)
7826 (324)
9954 (385)
Single-turn
7403 (15)
7830 (10)
9956 (18)
Total
(678)
(882)
(919)
Appendix
Table 5: Moshi R A G validation data statistics. Values are reported as: number of conversations (hours) .
Training Data
Ref. Encoder
Injection
Accuracy ( % )
LlamaQ
WebQ
TriviaQA
HaluEval
Single-ref.
T5
Insertive
81.0
79.0
82.0
47.1
Additive
59.6
36.6
28.5
13.9
Single-ref.
ARC-Encoder
Insertive
80.7
78.6
82.2
49.0
Additive
76.4
67.4
73.1
41.8
Appendix
Table 6: Performance of Moshi R A G with different information injection strategies. Results are scored with a Gemma 3 27B LLM. The default Moshi R A G setup adopts ARC-Encoder with additive injection.
Summarization
Ref. Encoder
Injection
Accuracy ( % )
LlamaQ
WebQ
TriviaQA
HaluEval
✗
ARC-8
Additive
77.4
66.3
66.2
33.1
ARC-4
Additive
80.3
74.7
73.2
36.3
T5
Additive
76.5
70.9
72.1
31.7
✓
ARC-4
Additive
77.4
75.4
73.5
37.5
T5
Additive
80.5
71.4
71.5
32.9
Appendix
Table 7: Performance of Moshi R A G with different reference text encoders. Results are scored with a Gemma 3 27B LLM. The default Moshi R A G setup adopts ARC-Encoder with a compression ratio of four without pre-encoding summarization.
User Text
Reference
Accuracy ( % )
LlamaQ
WebQ
TriviaQA
HaluEval
ref.
resp.
ref.
resp.
ref.
resp.
ref.
resp.
ASR
Gemma LLM
83.3
80.3
79.3
74.7
76.9
73.2
42.0
36.3
ground-truth
Gemma LLM
84.8
80.2
84.7
78.3
85.8
82.5
57.2
50.8
-
ground-truth
97.2
65.1
ASR Word Error Rate ( % )
0.7
5.2
4.6
11.5
Appendix
Table 8: Ablation study on QA benchmarks using ground-truth user text and ground-truth reference. Results are scored with a Gemma 3 27B LLM. Despite the 3 to 6 % correctness loss in the knowledge integration process (the gaps between ref. and resp. columns), the significantly improved performance when using ground-truth transcriptions demonstrates the potential of improving Moshi R A G by more accurate context modeling. However, the enlarged gap between reference and response accuracies when using HaluEval’s ground-truth reference also indicates that the knowledge integration process can be further improved.
Retrieval Back End
Accuracy ( % )
LlamaQ
WebQ
TriviaQA
HaluEval
ref.
resp.
ref.
resp.
ref.
resp.
ref.
resp.
Gemma
83.3
80.3
79.3
74.7
76.9
73.2
42.0
36.3
GPT 4.1
86.6
80.6
81.7
75.4
88.4
82.9
61.2
51.3
Llama 4 Maverick
81.3
78.1
79.3
72.9
81.6
70.5
45.0
33.8
Mistral Medium 3.1
86.6
79.2
83.1
72.9
87.8
80.8
53.7
44.0
Appendix
Table 9: Evaluation of Moshi R A G on QA Benchmarks with different retrieval back ends. Results are scored with a Gemma 3 27B LLM. GPT 4.1 consistently shows the best results while web search powered by Tavily also shows competitive performance.
Retrieval Back End
Accuracy ( % )
AddSub
MultiArith
SinglEq
SVAMP
GSM8K
ref.
resp.
ref.
resp.
ref.
resp.
ref.
resp.
ref.
resp.
Gemma
76.6
61.7
87.1
69.0
83.2
68.2
74.1
55.0
66.2
33.9
GPT 4.1
87.9
64.8
87.1
76.0
89.6
72.9
80.5
61.1
70.8
43.2
Mistral Medium 3.1
82.2
63.9
84.8
71.8
84.1
72.9
73.5
57.5
60.8
38.4
Gemini 2.5 Flash
84.1
63.0
88.1
67.3
87.9
67.3
75.5
53.1
66.8
38.8
Appendix
Table 10: Evaluation of Moshi R A G on mathematical reasoning benchmarks with different retrieval back ends. Results are scored with a Gemma 3 27B LLM. GPT 4.1 again performs the best on all tracks. We do not include Llama 4 Maverick in this experiment as its performance on the QA benchmarks does not provide a favorable trade-off considering its model size and inference cost.
Model
User Interruption
User Backchannel
Talking to Other
Background Speech
Resp. ↑
Resm. ↓
Unc. ↓
Unk. ↓
Resp. ↓
Resm. ↑
Unc. ↓
Unk. ↓
Resp. ↓
Resm. ↑
Unc. ↑
Unk. ↓
Resp. ↓
Resm. ↑
Unc. ↑
Unk. ↓
Freeze-Omni
0.72
0.12
0.03
0.13
0.07
0.80
0.02
0.11
0.58
0.25
0.00
0.15
0.63
0.25
0.01
0.11
Gemini
0.33
0.55
0.01
0.10
0.01
0.93
0.02
0.04
0.00
0.99
0.00
0.01
0.70
0.30
0.00
0.00
Sonic ( Amazon AGI and others, 2025 )
0.24
0.71
0.01
0.04
0.00
0.98
0.00
0.02
0.10
0.90
0.00
0.00
0.01
0.98
0.00
0.01
GPT-4o Audio
0.78
0.10
0.02
0.12
0.03
0.70
0.01
0.25
0.91
0.02
0.01
0.06
0.93
0.04
0.00
0.03
MoshiRAG
0.51
0.35
0.00
0.15
0.05
0.61
0.00
0.34
0.17
0.58
0.02
0.23
0.29
0.49
0.00
0.22
Appendix
Table 11: Evaluation results on Full-Duplex-Bench 1.5 ( Lin et al., 2026 ) across different types of overlapping speech. Underlined numbers are values reported in the original benchmark paper.
Figure 6: Analysis of Moshi R A G performance under different speech intelligibility and retrieval delay conditions.
Category
Domains
Category
Domains
Arts, Humanities & Culture
art history and curation
Engineering & Technology
mechanical and structural engineering
music composition and ethnomusicology
aerospace and nuclear engineering
religion theology and ancient languages
electrical and electronics engineering
explaining abstract and cultural concepts
software engineering and programming
navigating cultural traditions
robotics and automation
literature and critical studies
materials and nanotechnology
Appendix
Table 12: Expert conversation domains used for LLM-based topic generation.
Topic Generation Prompt
Generate a list of 50 long, clear, and specific topics suitable for in-depth discussion in the domain of ‘[DOMAIN]’. Each topic should be a full sentence or phrase, keeping it under 10 words. The topic should detail enough to guide a focused conversation, but still remain simple and easy to read and understand. Aim for a casual, conversational tone. Do not use any abbreviations, apostrophes symbols, or punctuations unless absolutely necessary for clarity.
For example, below are some possible topics within the domain of ‘artificial intelligence’:
- AI impact on daily life
- Building smarter home robots
Appendix
Table 13: The prompt for conversation topic generation.
Conversation Script
Human: Hey Moshi, ever feel like you’re living in a sci-fi movie these days?
Moshi: It definitely feels that way sometimes, doesn’t it? With all the new tech coming out.
Human: Seriously! Speaking of tech, I was reading about AI and art. It’s kinda wild.
Moshi: It is pretty wild. What specifically about AI and art are you curious about?
Appendix
Table 14: An example conversation script between human and Moshi with references.
Moshi LLM Prompt
You are generating the next line for the chatbot “moshi” in a realistic ongoing conversation between a human and moshi.
Context:
- You see the prior conversation history, including turns from both human and moshi.
- You may have access to some reference documents, which provides knowledge-intensive information for your response.
Appendix
Table 15: v1 LLM prompts used for generating conversation scripts and reference documents.
HaluEval Evaluation Prompt
## Background
You are a professional QA evaluation expert. You need to assess whether the model’s answer is correct based on the standard answer.
## Scoring Criteria
Correct: The answer matches or is equivalent to the standard answer, or contains the same core concept.
Incorrect: The answer is wrong or irrelevant to the question
Appendix
Table 16: The prompt for evaluating the correctness on the HaluEval dataset with a Gemma 3 27B model. This prompt is based on the prompts used in the OpenAudioBench ( Li et al., 2025 ) .
Keyword Extraction Prompt
You are helping evaluate a question answering model.
Identify the single keyword or short phrase in the model answer that directly expresses any of the answer aliases. If the model answer does not directly express any of the answer aliases, return the keyword/phrase that the model intends to answer the question. Respond with that keyword/phrase only.
Example:
Question: Give me a capital of an European country?
Table 17: The prompt for extracting keyword from the model’s response with a Gemma 3 27B model.
Mathematical Reasoning Evaluation Prompt
You are helping evaluate a mathematical question answering model.
Determine whether the model’s answer contains the provided Correct Answer. Ignore intermediate calculations or reasoning steps and focus on the numerical correctness of the model’s answer.
Instructions:
1. Compare the value in the model’s answer to the Correct Answer.
2. The comparison must be based on numerical equivalence (e.g., 5.0 should match 5). Ignore rounding errors or other small differences.
Appendix
Table 18: The prompt for evaluating the correctness on the mathematical reasoning datasets with a Gemma 3 27B model.
Full-duplex spoken dialogue models can listen and speak simultaneously like the real-time dynamics of human conversation. For natural dialogue, the ability to search for external information in real-time is also an important capability. Many models remain trapped in parametric knowledge, leaving them unable to access real-time information and tool execution. Furthermore, even when Large Language Models (LLM) retrieve information, many duplex speech models process it within a compressed latent space rather than in its raw text form, which can lead to information loss from compression. To address this issue, we propose Context Spanning, a framework for information injection between a full-duplex speech model and an external LLM backend via real-time chunked prefill. The injected frame is encoded in a single forward pass inside the real-time frame budget. It feeds the retrieved information to the speech model as-is, enabling it to reason over the information independently and generate responses. With this approach, our model achieves high performance on Full-Duplex benchmarks and strong results on Question Answering tasks, demonstrating its conversation potential. Context Spanning shows that external information can be injected directly into a duplex speech model, introducing a new simple and powerful mechanism for duplex systems.
We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.
Qingkai Fang, Shoutao Guo, Yang Feng
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China