MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

Ke Wang; Houxing Ren; Zimu Lu; Yunqiao Yang; Zhuofan Zong; Mingjie Zhan; Hongsheng Li

cs.CLSep 29, 2026

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

Authors: Ke Wang, Houxing Ren, Zimu Lu, Yunqiao Yang, Zhuofan Zong, Mingjie Zhan, Hongsheng Li

Organizations: CUHK MMLab · CPII under InnoHK

Abstract

End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data ( $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkPT}{MultiTalkPT}$ and $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkFT}{MultiTalkFT}$ ) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkBench}{MultiTalkBench}$ , built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.

Figures & tables

Figure 1: Training-data scale comparison.

	MultiTalkBench	VoiceBench	FDB-v1.5	MTalk-Bench
Total (h)	56.5	55.9	1.9	1.5
Max. (min)	43.6	0.9	0.2	0.8
Multi-party	✓	✗	✗	✓ (only 9.8 min)
Full-duplex	✓	✗	✓	✗
Bilingual	✓	✗	✗	✗

Table 1: Statistics of MultiTalkBench and prior speech evaluation benchmarks.

Figure 2: Automatic data engine for parallel-stream full-duplex dialogue. (1) Dialogue Script Synthesis turns a sampled configuration (length, participants, language) and four seed pools (emotions, occupations, knowledge, characters) into a constrained dialogue script via two LLM calls followed by a four-layer quality filter. (2) TTS Rendering synthesizes per-utterance waveforms with IndexTTS2 from a gender-matched voice prompt and an emotion vector, then assembles them into per-speaker parallel streams. (3) Word Alignment refines turn-level timestamps to word-level alignments via Montreal Forced Aligner (MFA).

Figure 3: MultiTalkBench. (a) Each test instance is built from a real multi-party conversation annotated and inspected into multi-channel dialogue with word-level timestamps, a scenario description, and per-speaker persona prompts. The model plays a designated target participant $s_{k}$ , and its generated utterances are spliced into the ordered transcript fed to an LLM-as-Judge. (b) The metric spans four general dimensions, six multi-party intelligence dimensions, four roles, and a mechanical span-ratio participation score.

	Open SOTA		Moshi family
	Qwen3-Omni	MiniCPM-o-4.5	PersonaPlex	Moshiko-7B	Moshi-MTB	$\Delta$	Human
Size	30B	9B	7B	7B	7B	–	–
General	6.93	16.35	2.76	4.96	18.73	$+277.6\%$	80.53
Clarity	5.91	18.63	4.35	5.25	19.41	$+269.7\%$	69.71
Relevance	6.47	21.91	3.23	7.52	22.72	$+202.1\%$	88.22
Agenda Fit	8.52	14.88	3.46	6.58	25.39	$+285.9\%$	89.42

Table 2: Detailed results on MultiTalkBench , broken down by sub-dimension. Bold : best result among models per row. The

\Delta

column reports relative change of Moshi-MTB over its Moshiko-7B backbone. Italics : human reference (oracle upper bound).

Metric Group	Gemma-4-31B vs. Human	Qwen3.5-27B vs. Human	Gemma-4-31B vs. Qwen3.5-27B
	( $\rho$ / $\tau$ )	( $\rho$ / $\tau$ )	( $\rho$ / $\tau$ )
General	0.845 / 0.739	0.761 / 0.614	0.838 / 0.701
Multi-Party IQ	0.854 / 0.764	0.770 / 0.644	0.768 / 0.649
Role-conditional	0.888 / 0.805	0.734 / 0.631	0.720 / 0.604
Final Score	0.898 / 0.787	0.846 / 0.700	0.847 / 0.706

Table 3: Correlation between human references and two LLM judges (Gemma-4-31B and Qwen3.5-27B) on MultiTalkBench . Each cell reports Spearman’s

\rho

/ Kendall’s

\tau

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Call	Group	Dimensions / output
1	–	primary role, secondary role, confidence, turn count
2a	MPIQ	M1, M2
2b	MPIQ	M3, M4
2c	MPIQ	M5, M6
3a	General	G1, G2, G3, G4
3b	Role	4–5 role-specific dims (F * , D * , C * , or E * )

Appendix

Table 4: Six-call judging pipeline per evaluation sample.

Explore similar work

DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling
Jul 6, 2026Wataru Nakata, Yuki Saito, Hiroshi SaruwatariFull-Duplex Speech Models Conversational Datasets
The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology, Japan
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengFull-Duplex Speech Models Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models
Jul 31, 2026Ryo Fukuda, Atsushi Ando, Hiroki Kanagawa +4Full-Duplex Speech Models Turn-Taking
NTT, Inc., Kyoto, Japan

Jul 6, 2026, cs.CLJ/K move · Enter open · S save

DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling

Wataru Nakata, Yuki Saito, Hiroshi Saruwatari

The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology, Japan

Full-Duplex Speech Models Conversational Datasets Text Corpora

Open paper AskPDF arXiv

Full-duplex spoken dialogue models are trained on conversational speech in which each speaker is represented as a separate stream, but existing large-scale public speech corpora are mostly monaural, making them unsuited for SDLM training. We present DuplexChat, an open-source corpus for full-duplex spoken dialogue models, and DuplexChat-Pipe, a pipeline for constructing speaker-separated full-duplex dialogue speech from public podcast feeds. DuplexChat-Pipe filters language-specific podcast feeds, retrieves and cleans episode audio, extracts diarization-guided two-speaker dialogue clips, and applies speech separation and restoration to produce one channel per speaker. Running this pipeline yields a speaker-separated spoken dialogue corpus covering 282,634 hours of English and 132,723 hours of Japanese. Analysis results on DuplexChat show that it contains turn-taking dynamics present in human dialogues.

Abstract

Explore similar work

DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models