MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

Chung-Ming Chien; Manu Orsini; Eugene Kharitonov; Neil Zeghidour; Karen Livescu; Alexandre Défossez

cs.CLApr 14, 2026

MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

Authors: Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, Alexandre Défossez

Organizations: Toyota Technological Institute at Chicago, Chicago, USA · Kyutai, Paris, France · Gradium, Paris, France

Abstract

Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.

Figures & tables

Figure 1: Illustration of turn-based models versus full-duplex models. The former must explicitly switch between speaking and listening states, while the latter can concurrently speak and listen.

Figure 2: Different types of delays in human-machine conversations. End-to-end keyword delay (E2EKD) measures the time between the end of the user’s question and the most informative word in the response. Retrieval delay measures how long it takes for the back end to provide relevant information.

Figure 3: Illustration of the front-end and back-end components in Moshi R A G . When the model needs external information, it outputs a $\langle\operatorname{ret}\rangle$ token. The conversation transcript is sent to the back end which operates asynchronously. Once ready, the result is injected into Moshi which then adapts its response with no interruption.

Figure 4: Text and audio token streams of the inputs and outputs of Moshi R A G . Front-end Moshi receives at all time its previous step token predictions and the user speech tokens. When the retrieval result is ready, its representation is summed with the embeddings from other token streams and ingested over a number of time steps.

Figure 5: Histograms of the distributions of retrieval and keyword delays. Our training data covers a wider range of retrieval delays than observed in practice at inference. Keyword delay is almost always longer than practical retrieval delay, leaving sufficient time to integrate retrieved information to improve the model’s answer.

Model (Base LM Size)	Accuracy ( $\%$ )				Delay (sec.)			Computation
	LlamaQ	WebQ	TriviaQA	HaluEval	TTFAT	KD	E2EKD	FLOPs/sec.
	ref. \|resp.	ref. \|resp.	ref. \|resp.	ref. \|resp.	TTFAT	KD	E2EKD	FLOPs/sec.
GPT-4o Audio ( OpenAI, 2024 )	88.4	81.0	90.6	68.7	-	5.5	-	-
GLM-4-Voice (9B) ( Zeng et al., 2024 )	64.7	32.2	39.1	21.2	0.3	4.2	4.4	0.74
Freeze-Omni (7B) ( Wang et al., 2025c )	72.0	44.7	53.9	14.0	1.2	5.2	6.5	0.18
MinMo (7B) ( Chen et al., 2025a )	78.9	55.0	48.3	-	-	-	-	-

Table 1: Results for factuality evaluation, delay metrics, and computation consumption of various models. Darker color indicates better performance. Underlined numbers are values reported in the original paper of each model. “-” indicates values inaccessible since part or all of the models are not released. “ref.” and “resp.” show the correctness of retrieved reference and Moshi R A G ’s final response, respectively.

Model	Pause		Backchannel			Turn Taking		User Interruption
Model	TOR ${}_{\text{s}}$ $\downarrow$	TOR ${}_{\text{c}}$ $\downarrow$	TOR $\downarrow$	Freq. (/s) $\uparrow$	JSD $\downarrow$	TOR $\uparrow$	Latency (s) $\downarrow$	TOR $\uparrow$	GPT Score $\uparrow$	Latency (s) $\downarrow$
dGSLM ( Nguyen et al., 2023 )	0.93	0.94	0.69	0.015	0.93	0.98	0.35	0.92	0.20	2.53
Freeze-Omni	0.64	0.48	0.64	0.001	1.00	0.34	0.95	0.87	3.62	1.41
Gemini ( Comanici and others, 2025 )	0.26	0.31	0.09	0.012	0.90	0.66	1.30	0.89	3.38	1.18
MoshiRAG	0.32	0.56	0.64	0.010	0.94	0.83	0.18	0.85	3.75	1.02
Vanilla Moshi	0.99	0.98	1.00	0.001	0.96	0.94	0.27	1.00	0.77	0.26

Table 2: Evaluation results on Full-Duplex-Bench ( Lin et al., 2025 ) . Underlined numbers are values reported in the original paper of each model. TOR

{}_{\text{s}}

and TOR

{}_{\text{c}}

in the Pause track correspond to the synthetic subset and the Candor subset, respectively.

Model	Accuracy ( $\%$ )
	AddSub	MultiArith	SinglEq	SVAMP	GSM8K
	ref. \|resp.	ref. \|resp.	ref. \|resp.	ref. \|resp.	ref. \|resp.
GLM-4-Voice	59.4	62.0	71.0	4.0	29.0
STITCH-S	81.7	87.9	91.7	72.2	56.7
MoshiRAG	76.6 \|61.7	87.1 \|69.0	83.2 \|68.2	74.1 \|55.0	66.2 \|33.9
MoshiRAG ${}_{\text{summ.}}$	62.0	73.1	66.4	57.5	51.2

Table 3: Evaluation results on out-of-domain mathematical reasoning datasets. Underlined numbers are values reported in the STITCH paper ( Chiang et al., 2026 ) . MoshiRAG

{}_{\text{summ.}}

means that the reference is summarized before being injected into Moshi.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	LLM	HotpotQA	NQ	TriviaQA	Total
Subset	(# conv / hrs)	(# conv / hrs)	(# conv / hrs)	(# conv / hrs)	(# conv / hrs)
Multi-turn v1	5457 (231)	90208 (2201)	305077 (8083)	76068 (1866)	476810 (12381)
Multi-turn v2	5549 (236)	90034 (2433)	303359 (9264)	75515 (2097)	474457 (14030)
Multi-turn v3	5550 (298)	90208 (3459)	306754 (12701)	76061 (2947)	478573 (19405)
Single-turn		90172 (438)	305364 (333)	471536 (1954)
Total	(765)	(8531)	(31231)	(7243)	1901376 (47770)

Appendix

Table 4: Moshi R A G training data statistics. Values are reported as: number of conversations (hours) .

Subset	HotpotQA	NQ	TriviaQA
Subset	(# conv / hrs)	(# conv / hrs)	(# conv / hrs)
Multi-turn v1	7403 (181)	7830 (208)	9956 (245)
Multi-turn v2	7391 (199)	7814 (240)	9939 (271)
Multi-turn v3	7403 (283)	7826 (324)	9954 (385)
Single-turn	7403 (15)	7830 (10)	9956 (18)
Total	(678)	(882)	(919)

Appendix

Table 5: Moshi R A G validation data statistics. Values are reported as: number of conversations (hours) .

Training Data	Ref. Encoder	Injection	Accuracy ( $\%$ )
Training Data	Ref. Encoder	Injection	LlamaQ	WebQ	TriviaQA	HaluEval
Single-ref.	T5	Insertive	81.0	79.0	82.0	47.1
Single-ref.	T5	Additive	59.6	36.6	28.5	13.9
Single-ref.	ARC-Encoder	Insertive	80.7	78.6	82.2	49.0
Single-ref.	ARC-Encoder	Additive	76.4	67.4	73.1	41.8

Appendix

Table 6: Performance of Moshi R A G with different information injection strategies. Results are scored with a Gemma 3 27B LLM. The default Moshi R A G setup adopts ARC-Encoder with additive injection.

Summarization	Ref. Encoder	Injection	Accuracy ( $\%$ )
Summarization	Ref. Encoder	Injection	LlamaQ	WebQ	TriviaQA	HaluEval
✗	ARC-8	Additive	77.4	66.3	66.2	33.1
	ARC-4	Additive	80.3	74.7	73.2	36.3
	T5	Additive	76.5	70.9	72.1	31.7
✓	ARC-4	Additive	77.4	75.4	73.5	37.5
✓	T5	Additive	80.5	71.4	71.5	32.9

Appendix

Table 7: Performance of Moshi R A G with different reference text encoders. Results are scored with a Gemma 3 27B LLM. The default Moshi R A G setup adopts ARC-Encoder with a compression ratio of four without pre-encoding summarization.

User Text	Reference	Accuracy ( $\%$ )
		LlamaQ		WebQ		TriviaQA		HaluEval
		ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.
ASR	Gemma LLM	83.3	80.3	79.3	74.7	76.9	73.2	42.0	36.3
ground-truth	Gemma LLM	84.8	80.2	84.7	78.3	85.8	82.5	57.2	50.8
-	ground-truth							97.2	65.1
ASR Word Error Rate ( $\%$ )		0.7		5.2		4.6		11.5

Appendix

Table 8: Ablation study on QA benchmarks using ground-truth user text and ground-truth reference. Results are scored with a Gemma 3 27B LLM. Despite the 3 to 6 % correctness loss in the knowledge integration process (the gaps between ref. and resp. columns), the significantly improved performance when using ground-truth transcriptions demonstrates the potential of improving Moshi R A G by more accurate context modeling. However, the enlarged gap between reference and response accuracies when using HaluEval’s ground-truth reference also indicates that the knowledge integration process can be further improved.

Retrieval Back End	Accuracy ( $\%$ )
	LlamaQ		WebQ		TriviaQA		HaluEval
	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.
Gemma	83.3	80.3	79.3	74.7	76.9	73.2	42.0	36.3
GPT 4.1	86.6	80.6	81.7	75.4	88.4	82.9	61.2	51.3
Llama 4 Maverick	81.3	78.1	79.3	72.9	81.6	70.5	45.0	33.8
Mistral Medium 3.1	86.6	79.2	83.1	72.9	87.8	80.8	53.7	44.0

Appendix

Table 9: Evaluation of Moshi R A G on QA Benchmarks with different retrieval back ends. Results are scored with a Gemma 3 27B LLM. GPT 4.1 consistently shows the best results while web search powered by Tavily also shows competitive performance.

Retrieval Back End	Accuracy ( $\%$ )
	AddSub		MultiArith		SinglEq		SVAMP		GSM8K
	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.	ref.	resp.
Gemma	76.6	61.7	87.1	69.0	83.2	68.2	74.1	55.0	66.2	33.9
GPT 4.1	87.9	64.8	87.1	76.0	89.6	72.9	80.5	61.1	70.8	43.2
Mistral Medium 3.1	82.2	63.9	84.8	71.8	84.1	72.9	73.5	57.5	60.8	38.4
Gemini 2.5 Flash	84.1	63.0	88.1	67.3	87.9	67.3	75.5	53.1	66.8	38.8

Appendix

Table 10: Evaluation of Moshi R A G on mathematical reasoning benchmarks with different retrieval back ends. Results are scored with a Gemma 3 27B LLM. GPT 4.1 again performs the best on all tracks. We do not include Llama 4 Maverick in this experiment as its performance on the QA benchmarks does not provide a favorable trade-off considering its model size and inference cost.

Model	User Interruption				User Backchannel				Talking to Other				Background Speech
Model	Resp. $\uparrow$	Resm. $\downarrow$	Unc. $\downarrow$	Unk. $\downarrow$	Resp. $\downarrow$	Resm. $\uparrow$	Unc. $\downarrow$	Unk. $\downarrow$	Resp. $\downarrow$	Resm. $\uparrow$	Unc. $\uparrow$	Unk. $\downarrow$	Resp. $\downarrow$	Resm. $\uparrow$	Unc. $\uparrow$	Unk. $\downarrow$
Freeze-Omni	0.72	0.12	0.03	0.13	0.07	0.80	0.02	0.11	0.58	0.25	0.00	0.15	0.63	0.25	0.01	0.11
Gemini	0.33	0.55	0.01	0.10	0.01	0.93	0.02	0.04	0.00	0.99	0.00	0.01	0.70	0.30	0.00	0.00
Sonic ( Amazon AGI and others, 2025 )	0.24	0.71	0.01	0.04	0.00	0.98	0.00	0.02	0.10	0.90	0.00	0.00	0.01	0.98	0.00	0.01
GPT-4o Audio	0.78	0.10	0.02	0.12	0.03	0.70	0.01	0.25	0.91	0.02	0.01	0.06	0.93	0.04	0.00	0.03
MoshiRAG	0.51	0.35	0.00	0.15	0.05	0.61	0.00	0.34	0.17	0.58	0.02	0.23	0.29	0.49	0.00	0.22

Appendix

Table 11: Evaluation results on Full-Duplex-Bench 1.5 ( Lin et al., 2026 ) across different types of overlapping speech. Underlined numbers are values reported in the original benchmark paper.

Panel of Figure 6 — Figure 6: Analysis of Moshi R A G performance under different speech intelligibility and retrieval delay conditions.

Category	Domains	Category	Domains
Arts, Humanities & Culture	art history and curation	Engineering & Technology	mechanical and structural engineering
	music composition and ethnomusicology		aerospace and nuclear engineering
	religion theology and ancient languages		electrical and electronics engineering
	explaining abstract and cultural concepts		software engineering and programming
	navigating cultural traditions		robotics and automation
	literature and critical studies		materials and nanotechnology

Appendix

Table 12: Expert conversation domains used for LLM-based topic generation.

Topic Generation Prompt
Generate a list of 50 long, clear, and specific topics suitable for in-depth discussion in the domain of ‘[DOMAIN]’. Each topic should be a full sentence or phrase, keeping it under 10 words. The topic should detail enough to guide a focused conversation, but still remain simple and easy to read and understand. Aim for a casual, conversational tone. Do not use any abbreviations, apostrophes symbols, or punctuations unless absolutely necessary for clarity.

For example, below are some possible topics within the domain of ‘artificial intelligence’:
- AI impact on daily life
- Building smarter home robots

Appendix

Table 13: The prompt for conversation topic generation.

Conversation Script
Human: Hey Moshi, ever feel like you’re living in a sci-fi movie these days?
Moshi: It definitely feels that way sometimes, doesn’t it? With all the new tech coming out.

Human: Seriously! Speaking of tech, I was reading about AI and art. It’s kinda wild.
Moshi: It is pretty wild. What specifically about AI and art are you curious about?

Appendix

Table 14: An example conversation script between human and Moshi with references.

Moshi LLM Prompt
You are generating the next line for the chatbot “moshi” in a realistic ongoing conversation between a human and moshi.

Context:
- You see the prior conversation history, including turns from both human and moshi.
- You may have access to some reference documents, which provides knowledge-intensive information for your response.

Appendix

Table 15: v1 LLM prompts used for generating conversation scripts and reference documents.

HaluEval Evaluation Prompt
## Background
You are a professional QA evaluation expert. You need to assess whether the model’s answer is correct based on the standard answer.

## Scoring Criteria
Correct: The answer matches or is equivalent to the standard answer, or contains the same core concept.
Incorrect: The answer is wrong or irrelevant to the question

Appendix

Table 16: The prompt for evaluating the correctness on the HaluEval dataset with a Gemma 3 27B model. This prompt is based on the prompts used in the OpenAudioBench ( Li et al., 2025 ) .

Keyword Extraction Prompt
You are helping evaluate a question answering model.
Identify the single keyword or short phrase in the model answer that directly expresses any of the answer aliases. If the model answer does not directly express any of the answer aliases, return the keyword/phrase that the model intends to answer the question. Respond with that keyword/phrase only.
Example:
Question: Give me a capital of an European country?
Model answer: Berlin is the capital of Germany.
Valid answer aliases: [“Paris”, “Madrid”, “Budapest”, “Lisbon”]

Appendix

Table 17: The prompt for extracting keyword from the model’s response with a Gemma 3 27B model.

Mathematical Reasoning Evaluation Prompt
You are helping evaluate a mathematical question answering model.
Determine whether the model’s answer contains the provided Correct Answer. Ignore intermediate calculations or reasoning steps and focus on the numerical correctness of the model’s answer.

Instructions:
1. Compare the value in the model’s answer to the Correct Answer.
2. The comparison must be based on numerical equivalence (e.g., 5.0 should match 5). Ignore rounding errors or other small differences.

Appendix

Table 18: The prompt for evaluating the correctness on the mathematical reasoning datasets with a Gemma 3 27B model.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 312 papers

Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends
Sep 27, 2026Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha +1Knowledge Augmentation for Language Models Spoken Dialogue Systems
Mindlogic
Full-duplex Spoken Language Modeling with Discrete Diffusion
Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language Models Full-Duplex Spoken Dialogue Systems
NVIDIA
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue Systems Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Sep 27, 2026J/K move · Enter open · S save

Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends

Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha, Jaeho Shin

Mindlogic

Knowledge Augmentation for Language Models Spoken Dialogue Systems Speech Language Models Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Full-duplex spoken dialogue models can listen and speak simultaneously like the real-time dynamics of human conversation. For natural dialogue, the ability to search for external information in real-time is also an important capability. Many models remain trapped in parametric knowledge, leaving them unable to access real-time information and tool execution. Furthermore, even when Large Language Models (LLM) retrieve information, many duplex speech models process it within a compressed latent space rather than in its raw text form, which can lead to information loss from compression. To address this issue, we propose Context Spanning, a framework for information injection between a full-duplex speech model and an external LLM backend via real-time chunked prefill. The injected frame is encoded in a single forward pass inside the real-time frame budget. It feeds the retrieved information to the speech model as-is, enabling it to reason over the information independently and generate responses. With this approach, our model achieves high performance on Full-Duplex benchmarks and strong results on Question Answering tasks, demonstrating its conversation potential. Context Spanning shows that external information can be injected directly into a duplex speech model, introducing a new simple and powerful mechanism for duplex systems.

Abstract

Speech Language Models

Explore similar work

Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends

Full-duplex Spoken Language Modeling with Discrete Diffusion

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM