MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

Ke Wang; Houxing Ren; Zimu Lu; Yunqiao Yang; Zhuofan Zong; Mingjie Zhan; Hongsheng Li

cs.CLSep 29, 2026

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

Authors: Ke Wang, Houxing Ren, Zimu Lu, Yunqiao Yang, Zhuofan Zong, Mingjie Zhan, Hongsheng Li

Organizations: CUHK MMLab · CPII under InnoHK

Abstract

End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data ( $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkPT}{MultiTalkPT}$ and $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkFT}{MultiTalkFT}$ ) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkBench}{MultiTalkBench}$ , built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.

Figures & tables

Figure 1: Training-data scale comparison.

	MultiTalkBench	VoiceBench	FDB-v1.5	MTalk-Bench
Total (h)	56.5	55.9	1.9	1.5
Max. (min)	43.6	0.9	0.2	0.8
Multi-party	✓	✗	✗	✓ (only 9.8 min)
Full-duplex	✓	✗	✓	✗
Bilingual	✓	✗	✗	✗

Table 1: Statistics of MultiTalkBench and prior speech evaluation benchmarks.

Figure 2: Automatic data engine for parallel-stream full-duplex dialogue. (1) Dialogue Script Synthesis turns a sampled configuration (length, participants, language) and four seed pools (emotions, occupations, knowledge, characters) into a constrained dialogue script via two LLM calls followed by a four-layer quality filter. (2) TTS Rendering synthesizes per-utterance waveforms with IndexTTS2 from a gender-matched voice prompt and an emotion vector, then assembles them into per-speaker parallel streams. (3) Word Alignment refines turn-level timestamps to word-level alignments via Montreal Forced Aligner (MFA).

Figure 3: MultiTalkBench. (a) Each test instance is built from a real multi-party conversation annotated and inspected into multi-channel dialogue with word-level timestamps, a scenario description, and per-speaker persona prompts. The model plays a designated target participant $s_{k}$ , and its generated utterances are spliced into the ordered transcript fed to an LLM-as-Judge. (b) The metric spans four general dimensions, six multi-party intelligence dimensions, four roles, and a mechanical span-ratio participation score.

	Open SOTA		Moshi family
	Qwen3-Omni	MiniCPM-o-4.5	PersonaPlex	Moshiko-7B	Moshi-MTB	$\Delta$	Human
Size	30B	9B	7B	7B	7B	–	–
General	6.93	16.35	2.76	4.96	18.73	$+277.6\%$	80.53
Clarity	5.91	18.63	4.35	5.25	19.41	$+269.7\%$	69.71
Relevance	6.47	21.91	3.23	7.52	22.72	$+202.1\%$	88.22
Agenda Fit	8.52	14.88	3.46	6.58	25.39	$+285.9\%$	89.42

Table 2: Detailed results on MultiTalkBench , broken down by sub-dimension. Bold : best result among models per row. The

\Delta

column reports relative change of Moshi-MTB over its Moshiko-7B backbone. Italics : human reference (oracle upper bound).

Metric Group	Gemma-4-31B vs. Human	Qwen3.5-27B vs. Human	Gemma-4-31B vs. Qwen3.5-27B
	( $\rho$ / $\tau$ )	( $\rho$ / $\tau$ )	( $\rho$ / $\tau$ )
General	0.845 / 0.739	0.761 / 0.614	0.838 / 0.701
Multi-Party IQ	0.854 / 0.764	0.770 / 0.644	0.768 / 0.649
Role-conditional	0.888 / 0.805	0.734 / 0.631	0.720 / 0.604
Final Score	0.898 / 0.787	0.846 / 0.700	0.847 / 0.706

Table 3: Correlation between human references and two LLM judges (Gemma-4-31B and Qwen3.5-27B) on MultiTalkBench . Each cell reports Spearman’s

\rho

/ Kendall’s

\tau

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Call	Group	Dimensions / output
1	–	primary role, secondary role, confidence, turn count
2a	MPIQ	M1, M2
2b	MPIQ	M3, M4
2c	MPIQ	M5, M6
3a	General	G1, G2, G3, G4
3b	Role	4–5 role-specific dims (F * , D * , C * , or E * )

Appendix

Table 4: Six-call judging pipeline per evaluation sample.

Explore similar work

Jul 6, 2026cs.CL

DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling

Full-duplex spoken dialogue models are trained on conversational speech in which each speaker is represented as a separate stream, but existing large-scale public speech corpora are mostly monaural, making them unsuited for SDLM training. We present DuplexChat, an open-source corpus for full-duplex spoken dialogue models, and DuplexChat-Pipe, a pipeline for constructing speaker-separated full-duplex dialogue speech from public podcast feeds. DuplexChat-Pipe filters language-specific podcast feeds, retrieves and cleans episode audio, extracts diarization-guided two-speaker dialogue clips, and applies speech separation and restoration to produce one channel per speaker. Running this pipeline yields a speaker-separated spoken dialogue corpus covering 282,634 hours of English and 132,723 hours of Japanese. Analysis results on DuplexChat show that it contains turn-taking dynamics present in human dialogues.

Wataru Nakata, Yuki Saito, Hiroshi Saruwatari

The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology, Japan

Full-Duplex Speech Models Conversational Datasets Text Corpora

Ask PDF

Jun 12, 2026cs.CL

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Full-Duplex Speech Models Speech Language Models Autoregressive Language Models Chatbots

Ask PDF

Jul 31, 2026cs.CL

M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models

Full-duplex spoken dialogue systems (FDSDSs) can listen while speaking, enabling natural behaviors such as smooth turn-taking, backchannel handling, and user barge-in handling. However, fair comparisons in multi-turn conversations remain a challenge. In addition, existing benchmarks provide limited coverage of languages and dialogue domains. We propose M3-DuplexBench, a multi-turn, multilingual, multidomain benchmark for FDSDSs. M3-DuplexBench supports English and Japanese and covers both casual conversation and multi-turn question answering. In addition, we evaluate models under multiple dialogue context settings, including single-turn, user-only, and teacher-forced full-context settings, to analyze how dialogue history affects model behavior. Experiments with recent FDSDSs reveal model-specific turn-taking characteristics, clear performance gaps across languages and domains, and mixed effects of dialogue context.

Ryo Fukuda, Atsushi Ando, Hiroki Kanagawa +4

NTT, Inc., Kyoto, Japan

Full-Duplex Speech Models Turn-Taking Multilingual Benchmark Multi-Turn Benchmark Multilingual

Ask PDF