PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting

Jianan Pan; Kejie Huang

eess.ASMar 5, 2026

PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting

Authors: Jianan Pan, Kejie Huang

Organizations: Zhejiang University, Hangzhou, China

Abstract

As advancements in technologies like Internet of Things (IoT), Automatic Speech Recognition (ASR), Speaker Verification (SV), and Text-to-Speech (TTS) lead to increased usage of intelligent voice assistants, the demand for privacy and personalization has escalated. In this paper, we introduce a multi-task learning framework for personalized, customizable open-vocabulary Keyword Spotting (PCOV-KWS). This framework employs a lightweight network to simultaneously perform Keyword Spotting (KWS) and SV to address personalized KWS requirements. We have integrated a training criterion distinct from softmax-based loss, transforming multi-class classification into multiple binary classifications, which eliminates inter-category competition, while an optimization strategy for multi-task loss weighting is employed during training. We evaluated our PCOV-KWS system in multiple datasets, demonstrating that it outperforms the baselines in evaluation results, while also requiring fewer parameters and lower computational resources.

Figures & tables

Fig. 1 : Proposed architecture of PCOV-KWS: The architecture comprises an audio encoder, which includes a shared encoder and two linear sub-encoders for KWS and SV respectively and cosine classifiers integrated with SphereFace 2 for metric learning.

Model Modification	MSWC Acc.(%)	#FLOPs
TC-ResNet14-1.5	$78.21(0.04)^{\mathrm{a}}$	6.86M
stage ratio{1,1,3,1}	$78.25(0.09)$	5.54M
NoisyDARTS	$79.48(0.12)$	4.63M
”patchify” stem	$79.36(0.07)$	4.65M
temporal dsconv	$78.11(0.07)$	1.61M
inverting dimensions	$81.45(0.05)$	7.87M

TABLE I : Detailed Results for Modifying TC-ResNet

Panel of Figure 3 — Fig. 2 : Performance Analysis on Training Strategy

Method	Backbone	SV	OV-KWS		PCOV-KWS		#Params
Method	Backbone	EER(%) $\downarrow$	AUC(%) $\uparrow$	EER(%)	AUC(%)	EER(%)	#Params
Vanilla	TC-ResNet14-1.5	-	98.94(0.12)	4.83(0.07)	51.01(0.12)	46.52(0.08)	313k
PCOV-KWS	TC-ResNet14-1.5	5.89(0.13)	99.16(0.09)	4.12(0.06)	98.32(0.10)	6.12(0.11)	326k
Vanilla	TDResNeXt	-	99.77(0.06)	1.85(0.06)	58.63(0.14)	44.02(0.08)	198k
Vanilla+SV	TDResNeXt	4.09(0.17)	99.77(0.06)	1.85(0.06)	98.96(0.12)	4.25(0.13)	396k
PCOV-KWS w/o CIB	TDResNeXt	4.16(0.09)	99.85(0.09)	1.56(0.09)	99.14(0.09)	3.98(0.12)	211k

TABLE II : Ablation Studies on LibriPhrase Dataset. To evaluate the PCOV-KWS task, we generate LibriPhrase-PCOV (

\textbf{\text{LP}}_{\textbf{\text{P}}}

) from LibriSpeech [ 33 ] . For the PCOV-KWS task, positive pairs are derived from instances that are both target keywords and target speakers, whereas the remaining samples form negative pairs.

Model	#Params	AUC(%) $\uparrow$		EER(%) $\downarrow$
Model	#Params	$\text{LP}_{\text{E}}$	$\text{LP}_{\text{H}}$	$\text{LP}_{\text{E}}$	$\text{LP}_{\text{H}}$
CMCD	653k	95.63	77.60	10.48	29.34
CLAD	-	97.03	76.15	8.65	30.30
PhonMatchNet	655k	99.29	88.52	2.80	18.82
PCOV-KWS-S	211k	99.77	88.50	1.85	19.33
PCOV-KWS-M	376k	99.88	89.46	1.32	18.38

TABLE III : Comparison with Baselines on OV-KWS

Model	0-shot	Acc.(%)	#Params	#FLOPs
Att-RNN	$\times$	95.6	202k	22.3M
ResNet-15		95.8	238k	894M
TENet12		96.6	100k	2.9M
TC-ResNet		96.6	305k	6.7M
MHAtt-RNN		97.2	743k	22.7M
MatchBoxNet		97.5	93k	11.3M

TABLE IV : Comparison with Baselines on C-KWS

Panel of Figure 7 — Fig. 3 : Evaluation results according to the number of words in a LibriPhrase evaluation set.

Keyword Spotting

Keyword spotting detects specified words or wake phrases in continuous audio, often on-device under tight latency, power, and false-alarm limits.

See all 24 papers

Effective User-defined Keyword Spotting with Dual-stage Matching, Multi-modal Enrollment, and Continual Adaptation
May 21, 2026Zhiqi Ai, Han Cheng, Shiyi Mu +3
Shanghai University, Shanghai 200444, China · New York University, New York, NY 10012 USA · Xi’an Jiaotong-Liverpool University, Suzhou 215000, China
ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody
Mar 5, 2026Jianan Pan, Yuanming Zhang, Kejie HuangKeyword Spotting Representation Learning
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China
Personalized Keyword Spotting for User-Defined Keywords Leveraging Text-Independent Speaker Verification
Jun 18, 2026Ming-Hsiang Hu, Kuan-Tang Huang, Chien-Chun Wang +2Zero-Shot Learning Keyword Spotting
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · E.SUN Financial Holding Co., Ltd., Taiwan · United Link Co., Ltd., Taiwan

May 21, 2026J/K move · Enter open · S save

Effective User-defined Keyword Spotting with Dual-stage Matching, Multi-modal Enrollment, and Continual Adaptation

Zhiqi Ai, Han Cheng, Shiyi Mu, Xinnuo Li+2

Shanghai University, Shanghai 200444, China · New York University, New York, NY 10012 USA · Xi’an Jiaotong-Liverpool University, Suzhou 215000, China

Open paper AskPDF arXiv

User-defined keyword spotting (KWS) is crucial for personalized voice interaction, yet existing methods face several challenges: (1) insufficient discriminability among confusable words, (2) performance inconsistency across speakers with varying pronunciations, and (3) high data cost to ensure reliable wake-word performance. In this paper, we introduce DMA-KWS, an efficient and robust framework for user-defined keyword spotting. First, it adopts a dual-stage matching pipeline: CTC decoding with streaming phoneme search to locate candidate segments, followed by QbyT with a phoneme matcher for fine-grained verification, enabling it to better distinguish confusable words. Next, multi-modal enrollment fuses user-specific speech with text embeddings to further improve accuracy for registered users. Finally, a parameter-efficient continual adaptation mechanism performs lightweight updates using synthetic and real data. Extensive experiments demonstrate the superior performance of DMA-KWS. On the LibriPhrase Hard subset, it achieves 97.85% AUC and 6.13% EER, reaching state-of-the-art performance. In speaker-dependent settings, DMA-KWS consistently outperforms text-only enrollment, demonstrating significant performance gains. Moreover, the proposed parameter-efficient fine-tuning mechanism adapts DMA-KWS with only 187k updated parameters, further enhancing KWS performance while ensuring suitability for on-device deployment.

Abstract

Keyword Spotting

Explore similar work

Effective User-defined Keyword Spotting with Dual-stage Matching, Multi-modal Enrollment, and Continual Adaptation

ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody

Personalized Keyword Spotting for User-Defined Keywords Leveraging Text-Independent Speaker Verification