Personal VAD: Speaker-Conditioned Voice Activity Detection

Shaojin Ding; Quan Wang; Shuo-yiin Chang; Li Wan; Ignacio Lopez Moreno

eess.ASAug 12, 2019

Personal VAD: Speaker-Conditioned Voice Activity Detection

Authors: Shaojin Ding, Quan Wang, Shuo-yiin Chang, Li Wan, Ignacio Lopez Moreno

Organizations: Texas A&M University, USA · Google Inc., USA

Abstract

In this paper, we propose "personal VAD", a system to detect the voice activity of a target speaker at the frame level. This system is useful for gating the inputs to a streaming on-device speech recognition system, such that it only triggers for the target user, which helps reduce the computational cost and battery consumption, especially in scenarios where a keyword detector is unpreferable. We achieve this by training a VAD-alike neural network that is conditioned on the target speaker embedding or the speaker verification score. For each frame, personal VAD outputs the probabilities for three classes: non-speech, target speaker speech, and non-target speaker speech. Under our optimal setup, we are able to train a model with only 130K parameters that outperforms a baseline system where individually trained standard VAD and speaker recognition networks are combined to perform the same task.

Figures & tables

Figure 1: The speaker verification system [ 29 ] produces an utterance-level d-vector by aggregating window-level embeddings.

Figure 2: Four different architectures to implement personal VAD: (a) SC : Run standard VAD and frame-level speaker verification independently, and combine their results. This is used as a baseline for other aproaches. (b) ST : Concatenate frame-level speaker verification score with acoustic features to train a personal VAD model. (c) ET : Concatenate speaker embedding with acoustic features to train a personal VAD model. (d) SET : Concatenate both speaker verification score and speaker embedding with acoustic features to train a personal VAD model.

Method	Loss	Without MTR				With MTR				Network parameters
Method	Loss	tss	ns	ntss	mean	tss	ns	ntss	mean	(million)
SC (baseline)	N/A $\star$	0.886	0.970	0.872	0.900	0.777	0.908	0.768	0.801	4.88 (SV) + 0.06 (VAD)
ST	CE	0.956	0.968	0.956	0.957	0.905	0.885	0.905	0.901	4.88 (SV) + 0.06 (PVAD)
ET		0.932	0.962	0.946	0.946	0.878	0.873	0.890	0.883	0.13 (PVAD)
SET		0.970	0.969	0.972	0.969	0.938	0.888	0.938	0.928	4.88 (SV) + 0.13 (PVAD)
ET	WPL	0.955	0.965	0.961	0.959	0.916	0.883	0.920	0.912	0.13 (PVAD)

Table 1: Architecture and loss function comparison results. SC : Score combination, the baseline system. ST : Score conditioned training. ET : Embedding conditioned training. SET : Score and embedding conditioned training. CE : Cross entropy loss. WPL : Weighted pairwise loss (

w_{\tt<ns,ntss>}=0.1

). We report the Average Precision (AP) for each class, and the mean Average Precision (mAP) over all the classes. Network parameters include 4.88 million parameters from the speaker verification (SV) model, if it is used during inference.

Figure 3: Histogram of SNR (dB) of our multistyle training.

Figure 4: Mean Average Precision (mAP) of personal VAD (ET) with different values of

w_{\tt<ns,ntss>}

in weighted pairwise loss. The weight between ns and ntss is displayed in log scale.

Method	Loss	Without MTR		With MTR
Method	Loss	s	ns	s	ns
Standard VAD	CE	0.992	0.975	0.975	0.918
Personal VAD (ET)	CE	0.991	0.965	0.979	0.893
Personal VAD (ET)	WPL	0.991	0.967	0.979	0.901

Table 2: Evaluation on a standard VAD task. We report the Average Precision (AP) for speech ( s ) and non-speech ( ns ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Hugging Face Repository ID	Conditioning Input	Loss	PVAD Params	INT8 .tflite
SC (Baseline)	wq2012/personal-vad-v1-sc	$\mathbf{x}_{t}\in\mathbb{R}^{40}$ + cosine score $s_{t}$	CE	64,194	196.4 KB
ST	wq2012/personal-vad-v1-st-ce	$[\mathbf{x}_{t},s_{t}]\in\mathbb{R}^{41}$	CE	64,515	197.4 KB
ET (CE)	wq2012/personal-vad-v1-et-ce	$[\mathbf{x}_{t},\mathbf{e}^{\mathrm{target}}]\in\mathbb{R}^{296}$	CE	129,795	262.4 KB
SET	wq2012/personal-vad-v1-set-ce	$[\mathbf{x}_{t},\mathbf{e}^{\mathrm{target}},s_{t}]\in\mathbb{R}^{297}$	CE	130,051	262.7 KB
ET (WPL)	wq2012/personal-vad-v1-et-wpl	$[\mathbf{x}_{t},\mathbf{e}^{\mathrm{target}}]\in\mathbb{R}^{296}$	WPL ( $w_{\langle\text{ns},\text{ntss}\rangle}=0.1$ )	129,795	262.4 KB

Appendix

Table 3: Open-source Personal VAD 1.0 pretrained models hosted on Hugging Face Hub ( wq2012 ). Each repository includes .safetensors , .weights.h5 , FP32 .tflite , and 8-bit quantized .tflite checkpoints.

Method	Loss	Original Paper (With MTR)				Open-Source 8-Language MLS Reproduction
Method	Loss	tss AP	ns AP	ntss AP	mAP	tss AP	ns AP	ntss AP	FP32 mAP (95% CI)	INT8 mAP	Std VAD ( s / ns )
SC (Baseline)	CE	0.777	0.908	0.768	0.801	0.9435	0.9604	0.7848	0.9161 $[0.9056,0.9277]$	0.9090	0.9962 / 0.9571
ST	CE	0.905	0.885	0.905	0.901	0.9394	0.9336	0.7550	0.8834 $[0.8668,0.9014]$	0.8804	—
ET	CE	0.878	0.873	0.890	0.883	0.9362	0.9536	0.7817	0.8665 $[0.8460,0.8897]$	0.8609	0.9960 / 0.9522
SET	CE	0.938	0.888	0.938	0.928	0.9385	0.9510	0.7725	0.8469 $[0.8237,0.8712]$	0.8521	—
ET	WPL	0.916	0.883	0.920	0.912	0.9432	0.9433	0.7695	0.8652 $[0.8467,0.8853]$	0.8691	0.9963 / 0.9539

Appendix

Table 4: Comparison between the original paper results (Table 1 & Table 2 , English LibriSpeech with internal 8-language 4.88M-parameter d-vector model) and the open-source reproduction on 8-language Multilingual LibriSpeech (MLS, 35 unseen test speakers,

23{,}310

concatenated frames), including 95% non-parametric bootstrap confidence intervals and 8-bit quantized TFLite micro-mAP.

Explore similar work

Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision
Sep 17, 2026Guangzhao Yang, Muhammad Huzaifah, Yu Pan +2Speaker Talk
R&D Team, Recho Inc., Tokyo, Japan
VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deployment
Jul 28, 2026Stephen Bauer, Sheila Seidel, Shanza Iftikhar +2Automatic Speaker Verification
Analog Devices, Inc., USA · University of California, Los Angeles (UCLA), USA
VOSSA: Voiceprint Optimization for Streaming Speech Architectures
Sep 30, 2026Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah +1Voice Conversion Whisper-Large-V3
Department of Computer Science & Engineering, Texas A&M University, College Station, US

Sep 17, 2026, eess.ASJ/K move · Enter open · S save

Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision

Guangzhao Yang, Muhammad Huzaifah, Yu Pan, Jinya Sakurai+1

R&D Team, Recho Inc., Tokyo, Japan

Speaker Talk Efficient Long-Context Inference

Open paper AskPDF arXiv

Voice activity detection (VAD) fronts most voice-agent pipelines, yet production detectors treat all human speech, background talkers included, as valid activity; in crowded settings this floods recognition, stalls turn-taking, and triggers false barge-in. We formalize Foreground VAD (FVAD): a frame-synchronous, enrollment-free task in which only the dominant speaker, defined by sustained presence rather than instantaneous loudness, is positive, and which reduces to conventional VAD when a single speaker is present. We show that foreground selectivity is largely governed by training supervision: the crucial ingredient is an augmentation recipe pairing foreground-only labels with competing-speaker mixing, generated fully automatically without human annotation. To quantify selectivity we introduce the Background False-Alarm Rate (BG-FAR), gated by foreground F1, and build a controlled benchmark, Mix-Interference, complemented by an adapted VOiCES for real-world far-field evaluation. Across equal-size backbones, Mamba and LSTM perform on par while a longer-context attention model is no better, suggesting that training supervision plays a substantially larger role than temporal modeling capacity in achieving foreground selectivity. The resulting lightweight streaming model, Mamba-FVAD, outperforms commercial VADs and enrollment-based speaker-aware systems in foreground selectivity while staying competitive on conventional VAD, at 1-2 ms per-frame CPU latency.

Abstract

Explore similar work

Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision

VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deployment

VOSSA: Voiceprint Optimization for Streaming Speech Architectures