X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System
Authors: Zhanxun Liu, Yifan Duan, Mengmeng Wang, Pengchao Feng, Haotian Zhang, Xiaoyu Xing, Yijia Shan, Haina Zhu, Yuxiang Zhao, Bei Liu, Yuhang Dai, Chaochao Lu, Xipeng Qiu, Lei Xie, Lan Wang, Nan Yan, Zilong Zheng, Ziyang Ma, Kai Yu, Xie Chen
Organizations: MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai AI Laboratory · State Key Laboratory of General Artificial Intelligence, BIGAI · Audio, Speech and Language Processing Group, Northwestern Polytechnical University · Fudan University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
We present X-Talk, an open-source framework that champions a decoupled, modular design for LLM-driven speech-to-speech (S2S) systems. While the dominant trend favors end-to-end (E2E) modeling to optimize information flow, these "omni-models" often struggle to balance the competing objectives of complex speech tasks within a single network. X-Talk challenges this paradigm by demonstrating that a systematically optimized cascaded pipeline can achieve sub-second latency without sacrificing modular flexibility. Our framework seamlessly integrates specialized front-end components (e.g., VAD, speech enhancement) and diverse understanding models (e.g., ASR, emotion, and environmental sound analysis) with LLM capabilities like retrieval-augmented generation (RAG) and tool use. By revitalizing the cascaded approach, X-Talk highlights the underestimated potential of modular S2S systems and provides a robust foundation for future research and applications.
Figures & tables
Figure 1: The data flow in the X-Talk framework, featuring four core functional modules of the speech-to-speech dialogue system: Speech Interaction Frontend, Speech Understanding, LLM-Driven Conversational Agent, and Speech Generation. The modules enclosed in solid lines denote currently integrated components, while those represented by dashed boxes indicate ongoing developments.
Figure 2: The system architecture in X-Talk. The entire system is built around a centralized event bus with a hierarchical design. All layers communicate asynchronously through an event publish-subscribe mechanism, enabling efficient management of complex conversational state and data flow.
ASR (streamable)
LLM
TTS
Latency (ms)
5s
10s
30s
60s
CN
EN
CN
EN
CN
EN
CN
EN
SenseVoice(streaming)
Qwen3-30B
IndexTTS 1.5
284
272
346
406
420
610
449
376
Paraformer(streaming)
Qwen3-30B
IndexTTS 1.5
264
367
399
488
379
412
393
382
SenseVoice(offline)
Qwen3-30B
IndexTTS 1.5
635
617
1015
1054
2175
2326
3798
3671
SenseVoice(streaming)
Qwen3-8B
IndexTTS 1.5
808
339
948
625
1039
1111
1075
1068
Table 2: X-Talk’s latency test across different combinations of models with different input speech length. CN/EN indicates Chinese/English speech inputs. CosyVoice refers to CosyVoice-v3-flash. For each configuration, latency is measured three times and the final value is the rounded average. The first row is the baseline configuration in X-Talk, while each of the other configurations has one model varied.
School of Computer Science and Technology, Tongji University · School of Artificial Intelligence, Chinese University of Hong Kong, Shenzhen · Department of Language Science and Technology, Hong Kong Polytechnic University +2