cs.SDDec 21, 2025

X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System

Authors: Zhanxun Liu, Yifan Duan, Mengmeng Wang, Pengchao Feng, Haotian Zhang, Xiaoyu Xing, Yijia Shan, Haina Zhu, 

Organizations: MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai AI Laboratory · State Key Laboratory of General Artificial Intelligence, BIGAI · 

Abstract

We present X-Talk, an open-source framework that champions a decoupled, modular design for LLM-driven speech-to-speech (S2S) systems. While the dominant trend favors end-to-end (E2E) modeling to optimize information flow, these "omni-models" often struggle to balance the competing objectives of complex speech tasks within a single network. X-Talk challenges this paradigm by demonstrating that a systematically optimized cascaded pipeline can achieve sub-second latency without sacrificing modular flexibility. Our framework seamlessly integrates specialized front-end components (e.g., VAD, speech enhancement) and diverse understanding models (e.g., ASR, emotion, and environmental sound analysis) with LLM capabilities like retrieval-augmented generation (RAG) and tool use. By revitalizing the cascaded approach, X-Talk highlights the underestimated potential of modular S2S systems and provides a robust foundation for future research and applications.

Figures & tables

Explore similar work

CardsList
  1. RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

    Mar 24, 2026Long Mai, Junli LiangSpoken Dialogue SystemsSpeculative Decoding

  2. Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

    Feb 26, 2026Siyuan Liu, Jiahui Xu, Feng Jiang +6Spoken Dialogue SystemsFull-Duplex Spoken Dialogue Systems