FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech

Puneet Mathur; Dinesh Manocha

cs.CLSep 16, 2026

FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech

Authors: Puneet Mathur, Dinesh Manocha

Organizations: University of Maryland, College Park

Abstract

Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic voice activity detection lacks this semantic information, while cascaded ASR-based endpointing introduces transcription dependence and additional processing stages. We formulate semantic endpoint detection as a causal audio-language reasoning task and introduce FD-VAD, an ASR-free streaming endpointer that maps bounded causal audio windows directly to Continue/Stop decisions. FD-VAD combines a frozen speech encoder with a lightweight modality adapter and a parameter-efficiently adapted language model, using a last-chunk training objective for streaming inference. We further introduce confidence-gated endpoint commitment to control interruption versus delay and boundary-focused hard-negative sampling to improve decisions around ambiguous turn boundaries. Across in-domain and conversational evaluations, FD-VAD outperforms strong streaming and non-streaming semantic turn classifiers, and achieves the highest EOT recall among qualifying systems on TurnBench dev set $0.853$ (at FP<=0.10) in a zero-shot setting. These results show that semantic endpointing can be performed directly from streaming audio without intermediate ASR or dialogue state tracking.

Figures & tables

Figure 1: Streaming semantic endpoint detection task, where 0 and 1 denote Continue and Stop , respectively. FD-VAD uses linguistic as well as acoustic cues for end-of-turn detection.

Figure 2: FD-VAD architecture. A causal audio window of incoming speech is encoded by frozen WavLM, projected into the embedding space of a LoRA-adapted Qwen2.5-0.5B language model for Continue / Stop endpoint prediction. Training supervises only the final chunk of each window.

Model	Chunk $F_{1}$	Chunk Acc.	False-stop $\downarrow$
Always- Continue	0.000	0.953	0.0%
WavLM-base causal [ 1 ]	0.183	0.721	29.1%
WavLM-large causal [ 1 ]	0.191	0.724	29.0%
WavLM-large anchor [ 1 ]	0.218	0.787	21.6%
Silero VAD [ 17 ]	0.251	0.790	22.5%
VAP [ 4 ]	0.279	0.803	20.1%

Table 1: Chunk-level semantic endpoint detection on the held-out smart-turn-v3.1 [ 14 ] test set (1,000 utterances; 500 complete and 500 incomplete). Stop is the positive class; false-stop is the fraction of incomplete-speech chunks incorrectly predicted as Stop . FD-VAD outperforms streaming baselines with lowest false-stop rate.

	Model	Str.	Complete	Incomplete	Overall
Acoustic	Energy VAD	✓	0.998	0.680	0.839 [.82,.86]
Acoustic	Silero VAD [ 17 ]	✓	0.996	0.196	0.596 [.56,.63]
Streaming	VAP [ 4 ]	✓	0.728	0.489	0.608 [.58,.64]
	WavLM-base causal [ 1 ]	✓	0.824	0.344	0.584 [.55,.62]
	WavLM-large causal [ 1 ]	✓	0.904	0.264	0.584 [.55,.61]
	WavLM-large anchor [ 1 ]	✓	0.908	0.268	0.588 [.56,.62]

Table 2: Utterance-level endpoint classification on the held-out smart-turn-v3.1 [ 14 ] test set with 95% bootstrap CI over 2K samples. FD-VAD shows the strongest balanced performance across complete and incomplete turns in streaming mode (Str), comparable to the strongest non-streaming semantic classifier.

Category	System	Recall	FP	Lat. (ms) $\downarrow$
Turn-taking	VAP [ 4 ]	0.841	0.045	463
	ESPnet turn-taking	0.836	0.074	895
	ESPnet TT (per-ch.)	0.640	0.100	846
	Smart Turn [ 16 ]	0.754	0.100	1010
	Moshi [ 3 ]	0.212	0.066	771
	SoulX-Duplug [ 24 ]	0.194	0.154 †	89

Table 3: TurnBench [ 9 ] dev end-of-turn (EOT) performance under benchmark false-positive constraint (

\mathrm{FP}\leq 0.10

). FD-VAD is evaluated zero-shot and achieves the highest recall among qualifying systems.

\dagger

denotes systems that exceed the false-positive budget.

Panel of Figure 3 — Figure 3: TurnBench [ 9 ] dev operating characteristics. Zero-shot gated FD-VAD achieves the highest EOT recall within the benchmark false-positive constraint ( $\mathrm{FP}\leq 0.10$ ). The latency view highlights the complementary trade-off between endpoint reliability and commit speed; selected competitors are annotated and remaining baselines are shown in gray.

Variant	Stride	Boundary	Trainable	Chunk	Comp.	Near-b.	Sent-C
	(ms)	sampling	adapter	Acc.	$F_{1}$	$F_{1}$	Acc.
FD-VAD (base)	320	$\times$	✓	0.981	0.827	0.838	0.406
160-ms stride	160	$\times$	✓	0.983	0.798	0.779	0.250
Frozen adapter	320	$\times$	$\times$	0.978	0.803	0.816	0.378
+ Boundary sampling	320	✓	✓	0.987	0.838	0.880	0.476
Synthetic-only training	320	✓	✓	0.972	0.785	0.786	0.341

Table 4: Ablation results on a matched 8k-clip subset. All variants use the same FD-VAD architecture and differ only in the listed settings. Near-b. is

F_{1}

within

\pm 2

chunks of the endpoint; Sent-C is strict all-chunk accuracy on complete utterances. We study robustness to training data by training FD-VAD on synthetic-only data that transfers well per chunk but degrades at utterance-level.

Full-Duplex Spoken Dialogue Systems

Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.

See all 119 papers

Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interaction
Sep 24, 2025Weijie Wu, Wenhao Guan, Kaidi Wang +6Full-Duplex Spoken Dialogue Systems
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction
Jun 16, 2026Tristan Tsoi, Jiajun Deng, Yingke Zhu +5Event Time Prediction Streaming ASR
Central Media Technology Institute, Huawei · The Chinese University of Hong Kong · Nanyang Technological University
DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization
Mar 10, 2026Jianing Yang, Yusuke Fujita, Yui SudoSpoken Dialogue Systems Full-Duplex Spoken Dialogue Systems
SB Intuitions Corp., Japan · The University of Tokyo, Japan

Sep 24, 2025J/K move · Enter open · S save

Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interaction

Weijie Wu, Wenhao Guan, Kaidi Wang, Peijie Chen+5

School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China

Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Spoken dialogue models have significantly advanced intelligent human-computer interaction, yet they lack a plug-and-play full-duplex prediction module for semantic endpoint detection, hindering seamless audio interactions. In this paper, we introduce Phoenix-VAD, an LLM-based model that enables streaming semantic endpoint detection. Specifically, Phoenix-VAD leverages the semantic comprehension capability of the LLM and a sliding window training strategy to achieve reliable semantic endpoint detection while supporting streaming inference. Experiments on both semantically complete and incomplete speech scenarios indicate that Phoenix-VAD achieves excellent and competitive performance. Furthermore, this design enables the full-duplex prediction module to be optimized independently of the dialogue model, providing more reliable and flexible support for next-generation human-computer interaction.

Abstract

Full-Duplex Spoken Dialogue Systems

Explore similar work

Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interaction

Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction

DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization