Achieving natural full-duplex interaction in spoken dialogue systems (SDS) remains a challenge due to the difficulty of accurately detecting user interruptions. Current solutions are polarized between "trigger-happy" VAD-based methods that misinterpret backchannels and robust end-to-end models that exhibit unacceptable response delays. Moreover, the absence of real-world benchmarks and holistic metrics hinders progress in the field. This paper presents a comprehensive frame-work to overcome these limitations. We first introduce SID-Bench, the first benchmark for semantic-aware interruption detection built entirely from real-world human dialogues. To provide a rigorous assessment of the responsiveness-robustness trade-off, we propose the Average Penalty Time (APT) metric, which assigns a temporal cost to both false alarms and late responses. Building on this framework, we design an LLM-based detection model optimized through a novel training paradigm to capture subtle semantic cues of intent. Experimental results show that our model significantly outperforms mainstream baselines, achieving a nearly threefold reduction in APT. By successfully resolving the long-standing tension between speed and stability, our work establishes a new state-of-the-art for intelligent interruption handling in SDS. To facilitate future research, SID-Bench and the associated code are available at: https://github.com/xkx-hub/SID-bench.
Figures & tables
Fig. 1: The semi-automated annotation pipeline for SID-Bench. The process begins with raw audio and its transcription. In the Annotation stage, the audio is processed by the Kaldi toolkit for forced alignment to obtain word-level timestamps, while the text is analyzed by a series of LLMs to semantically identify the interruption point, marked with a <break> tag. In the final Information Fusion stage, the semantic <break> tag is aligned with the precise start time of the corresponding word from Kaldi, establishing a semantically meaningful and temporally accurate ground-truth Break_point.
Language
Category
Count
Chinese
Interruption at the beginning
500
Interruption in the middle
600
Uninterrupted
500
English
Interruption at the beginning
500
Interruption in the middle
600
Uninterrupted
500
TABLE I: The composition and statistics of SID-bench dataset.
Fig. 2: Illustration of the four evaluation scenarios and their associated time penalties. The user’s utterance contains a ground-truth interruption intent marked by <break> . (a) True Positive: The system correctly stops after the <break> point. The penalty is IRL, shown in blue. (b) False Positive: The system incorrectly stops in response to a backchannel before the <break> . This is a catastrophic failure, and the penalty, shown in red, applies to the entire turn’s duration. (c) False Negative: The system fails to stop, forcing the user to listen to superfluous speech. The penalty, shown in red, is the duration of this unwanted audio. (d) True Negative: The system correctly ignores a backchannel and continues speaking when no <break> is present, thus incurring zero penalty.
Fig. 3: The overall architecture of the proposed SID-model for real-time interruption detection. (1) During training, we use audio with a pre-labeled user interruption point. This audio is randomly cropped into clips of varying lengths, represented by different colors. Each clip is assigned a ground-truth label: ‘Y’, interrupt, if its endpoint is after the user interruption point, and ‘N’ otherwise. (2) The audio clips are processed by an Audio Encoder (AuT) to extract features. (3) The resulting audio frame sequences are fed into a LLM (Qwen3-0.6b). (4) The model sequentially predicts whether to issue an interrupt signal or not, enabling the system to stop its speech in a timely manner. The user interruption points are annotated following the procedure of the SID-bench.
EN
ZH
Noise Silence
Average
Model
IRL ↓
FIR ↓
APT ↓
IRL ↓
FIR ↓
APT ↓
FIR ↓
APT ↓
IRL ↓
FIR ↓
APT ↓
FSMN-VAD [ 28 ]
0
0.906
3.328
0
0.915
3.563
0.392
4.791
0
0.840
3.627
Free-Omni(Silero VAD) [ 21 ]
0.141
0.669
2.429
0.186
0.513
1.977
0.154
1.658
0.173
0.532
2.129
FireRedChat (pVAD) [ 1 ]
0.474
0.734
2.997
1.170
0.338
2.311
0.098
2.458
1.045
0.476
2.627
Moshi [ 3 ]
2.517
0.010
1.749
-
-
-
0.464
7.810
2.517
0.118
3.192
SID-model (proposed)
0.444
0.140
0.921
0.338
0.138
0.656
0.026
0.215
0.389
0.124
0.711
TABLE II: Comparison of different models on Average IRL, False Interruption Rate, and Average Penalty Time across EN, ZH, and Noise conditions. Lower values are better (indicated by ↓ ).
Silence
Noise
Model
FAN
APT
FAN
APT
FSMN-VAD
159
5.420
37
3.846
Freeze-Omni
67
2.286
10
0.716
FireRedChat
0
0
49
6.145
Moshi
168
5.720
64
7.810
SID-model (proposed)
10
0.340
3
0.028
TABLE III: Comparison of false alarm number and average penalty time for different models under Silence and Noise conditions.
While recent Spoken Language Models (SLMs) have been actively deployed in real-world scenarios, they lack the capability to discern Third-Party Interruptions (TPI) from the primary user's ongoing flow, leaving them vulnerable to contextual failures. To bridge this gap, we introduce TPI-Train, a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling, and TPI-Bench, a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. Experiments demonstrate that our dataset design mitigates semantic shortcut learning-a critical pitfall where models exploit semantic context while neglecting acoustic signals essential for discerning speaker changes. We believe our work establishes a foundational resource for overcoming text-dominated unimodal reliance in SLMs, paving the way for more robust multi-party spoken interaction. The code for the framework is publicly available at https://tpi-va.github.io
Dongwook Lee, Eunwoo Song, Che Hyun Lee +2
1Interdisciplinary Program in Artificial Intelligence, Seoul National University (SNU) · Department of EE, Yonsei University · Department of ECE, SNU +2
Spoken dialogue models have significantly advanced intelligent human-computer interaction, yet they lack a plug-and-play full-duplex prediction module for semantic endpoint detection, hindering seamless audio interactions. In this paper, we introduce Phoenix-VAD, an LLM-based model that enables streaming semantic endpoint detection. Specifically, Phoenix-VAD leverages the semantic comprehension capability of the LLM and a sliding window training strategy to achieve reliable semantic endpoint detection while supporting streaming inference. Experiments on both semantically complete and incomplete speech scenarios indicate that Phoenix-VAD achieves excellent and competitive performance. Furthermore, this design enables the full-duplex prediction module to be optimized independently of the dialogue model, providing more reliable and flexible support for next-generation human-computer interaction.
Weijie Wu, Wenhao Guan, Kaidi Wang +6
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
While low-latency interaction is critical for spoken dialogue, cascaded architectures are often bottlenecked by reactive turn-completion detection. We propose Endpoint Anticipation, shifting from reactive detection to proactive forecasting of end-of-turn signals. Our speech-based model anticipates endpoints upto 2.56 seconds in advance, enabling speculative execution of LLM and TTS pipelines on partial context. We introduce metrics to quantify the trade-off between realized latency reduction and computational redundancy. Evaluation across conversational and task-oriented datasets shows our model consistently outperforms competitive VAP-based baselines. Integration with the Unmute framework demonstrates a 505 ms average latency reduction with a 28.4% increase in speculative computation, effectively masking sequential bottlenecks to enable complex reasoning in real-time speech-to-speech interaction.
Sathvik Udupa, Shinji Watanabe, Petr Schwarz +1
Brno University of Technology, Czechia · Carnegie Mellon University, United States