Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic voice activity detection lacks this semantic information, while cascaded ASR-based endpointing introduces transcription dependence and additional processing stages. We formulate semantic endpoint detection as a causal audio-language reasoning task and introduce FD-VAD, an ASR-free streaming endpointer that maps bounded causal audio windows directly to Continue/Stop decisions. FD-VAD combines a frozen speech encoder with a lightweight modality adapter and a parameter-efficiently adapted language model, using a last-chunk training objective for streaming inference. We further introduce confidence-gated endpoint commitment to control interruption versus delay and boundary-focused hard-negative sampling to improve decisions around ambiguous turn boundaries. Across in-domain and conversational evaluations, FD-VAD outperforms strong streaming and non-streaming semantic turn classifiers, and achieves the highest EOT recall among qualifying systems on TurnBench dev set 0.853 (at FP<=0.10) in a zero-shot setting. These results show that semantic endpointing can be performed directly from streaming audio without intermediate ASR or dialogue state tracking.
Figures & tables
Figure 1: Streaming semantic endpoint detection task, where 0 and 1 denote Continue and Stop , respectively. FD-VAD uses linguistic as well as acoustic cues for end-of-turn detection.
Figure 2: FD-VAD architecture. A causal audio window of incoming speech is encoded by frozen WavLM, projected into the embedding space of a LoRA-adapted Qwen2.5-0.5B language model for Continue / Stop endpoint prediction. Training supervises only the final chunk of each window.
Model
Chunk F1
Chunk Acc.
False-stop ↓
Always- Continue
0.000
0.953
0.0%
WavLM-base causal [ 1 ]
0.183
0.721
29.1%
WavLM-large causal [ 1 ]
0.191
0.724
29.0%
WavLM-large anchor [ 1 ]
0.218
0.787
21.6%
Silero VAD [ 17 ]
0.251
0.790
22.5%
VAP [ 4 ]
0.279
0.803
20.1%
Table 1: Chunk-level semantic endpoint detection on the held-out smart-turn-v3.1 [ 14 ] test set (1,000 utterances; 500 complete and 500 incomplete). Stop is the positive class; false-stop is the fraction of incomplete-speech chunks incorrectly predicted as Stop . FD-VAD outperforms streaming baselines with lowest false-stop rate.
Model
Str.
Complete
Incomplete
Overall
Acoustic
Energy VAD
✓
0.998
0.680
0.839 [.82,.86]
Silero VAD [ 17 ]
✓
0.996
0.196
0.596 [.56,.63]
Streaming
VAP [ 4 ]
✓
0.728
0.489
0.608 [.58,.64]
WavLM-base causal [ 1 ]
✓
0.824
0.344
0.584 [.55,.62]
WavLM-large causal [ 1 ]
✓
0.904
0.264
0.584 [.55,.61]
WavLM-large anchor [ 1 ]
✓
0.908
0.268
0.588 [.56,.62]
Table 2: Utterance-level endpoint classification on the held-out smart-turn-v3.1 [ 14 ] test set with 95% bootstrap CI over 2K samples. FD-VAD shows the strongest balanced performance across complete and incomplete turns in streaming mode (Str), comparable to the strongest non-streaming semantic classifier.
Category
System
Recall
FP
Lat. (ms) ↓
Turn-taking
VAP [ 4 ]
0.841
0.045
463
ESPnet turn-taking
0.836
0.074
895
ESPnet TT (per-ch.)
0.640
0.100
846
Smart Turn [ 16 ]
0.754
0.100
1010
Moshi [ 3 ]
0.212
0.066
771
SoulX-Duplug [ 24 ]
0.194
0.154 †
89
Table 3: TurnBench [ 9 ] dev end-of-turn (EOT) performance under benchmark false-positive constraint ( FP≤0.10 ). FD-VAD is evaluated zero-shot and achieves the highest recall among qualifying systems. † denotes systems that exceed the false-positive budget.
Figure 3: TurnBench [ 9 ] dev operating characteristics. Zero-shot gated FD-VAD achieves the highest EOT recall within the benchmark false-positive constraint ( FP≤0.10 ). The latency view highlights the complementary trade-off between endpoint reliability and commit speed; selected competitors are annotated and remaining baselines are shown in gray.
Variant
Stride
Boundary
Trainable
Chunk
Comp.
Near-b.
Sent-C
(ms)
sampling
adapter
Acc.
F1
F1
Acc.
FD-VAD (base)
320
×
✓
0.981
0.827
0.838
0.406
160-ms stride
160
×
✓
0.983
0.798
0.779
0.250
Frozen adapter
320
×
×
0.978
0.803
0.816
0.378
+ Boundary sampling
320
✓
✓
0.987
0.838
0.880
0.476
Synthetic-only training
320
✓
✓
0.972
0.785
0.786
0.341
Table 4: Ablation results on a matched 8k-clip subset. All variants use the same FD-VAD architecture and differ only in the listed settings. Near-b. is F1 within ±2 chunks of the endpoint; Sent-C is strict all-chunk accuracy on complete utterances. We study robustness to training data by training FD-VAD on synthetic-only data that transfers well per chunk but degrades at utterance-level.
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
Spoken dialogue models have significantly advanced intelligent human-computer interaction, yet they lack a plug-and-play full-duplex prediction module for semantic endpoint detection, hindering seamless audio interactions. In this paper, we introduce Phoenix-VAD, an LLM-based model that enables streaming semantic endpoint detection. Specifically, Phoenix-VAD leverages the semantic comprehension capability of the LLM and a sliding window training strategy to achieve reliable semantic endpoint detection while supporting streaming inference. Experiments on both semantically complete and incomplete speech scenarios indicate that Phoenix-VAD achieves excellent and competitive performance. Furthermore, this design enables the full-duplex prediction module to be optimized independently of the dialogue model, providing more reliable and flexible support for next-generation human-computer interaction.
Weijie Wu, Wenhao Guan, Kaidi Wang +6
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
Endpoint detection (EPD) is essential for natural turn-taking in streaming speech systems. However, reliably determining the endpoint of an utterance is challenging because speakers often pause mid-utterance due to hesitations and disfluencies. Semantic EPD has emerged as a promising direction to address this issue but is hindered by ambiguous supervision and strict streaming constraints. We propose Next-Turn that uses the time-to-next-speech-onset as the training objective, where targets are derived directly from speech timestamps and require no additional annotation. Experiments show that the proposed method outperforms conventional acoustic and recent semantic EPD baselines, achieving a 25.9% absolute improvement in endpoint accuracy within 320 ms over the strongest baseline. In addition, joint training with the duration-aware objective complements standard binary EPD, with gains that increase monotonically with increasing pauses.
Tristan Tsoi, Jiajun Deng, Yingke Zhu +5
Central Media Technology Institute, Huawei · The Chinese University of Hong Kong · Nanyang Technological University
Spoken dialog systems with cascaded ASR-LLM-TTS modules retain strong LLM intelligence, but VAD segmentation often forces half-duplex turns and brittle control. On the other hand, VAD-free end-to-end model support full-duplex interaction but is hard to maintain conversational intelligence. In this paper, we present DuplexCascade, a VAD-free cascaded streaming pipeline for full-duplex speech-to-speech dialogue. Our key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM. To reliably coordinate turn-taking and response timing, we introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints. On Full-DuplexBench and VoiceBench, DuplexCascade delivers state-of-the-art full-duplex turn-taking and strong conversational intelligence among open-source speech-to-speech dialogue systems.
Jianing Yang, Yusuke Fujita, Yui Sudo
SB Intuitions Corp., Japan · The University of Tokyo, Japan