ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody

Jianan Pan; Yuanming Zhang; Kejie Huang

eess.ASMar 5, 2026

ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody

Authors: Jianan Pan, Yuanming Zhang, Kejie Huang

Organizations: College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China

Abstract

Current keyword spotting systems primarily use phoneme-level matching to distinguish confusable words but ignore user-specific pronunciation traits like prosody (intonation, stress, rhythm). This paper presents ProKWS, a novel framework integrating fine-grained phoneme learning with personalized prosody modeling. We design a dual-stream encoder where one stream derives robust phonemic representations through contrastive learning, while the other extracts speaker-specific prosodic patterns. A collaborative fusion module dynamically combines phonemic and prosodic information, enhancing adaptability across acoustic environments. Experiments show ProKWS delivers highly competitive performance, comparable to state-of-the-art models on standard benchmarks and demonstrates strong robustness for personalized keywords with tone and intent variations.

Figures & tables

Figure 1: Overall architecture of the proposed ProKWS.

	Anchor	Easy	Hard
LibriPhrase	friend	guard	frind
		comfort	rend
		superior	trend
WenetPhrase	ning2yuan4	sha1mo4	ting2yuan4
		de2zhi1	xing2yuan4
		gong1wu4	qing2yuan4

Table 1: Negative Examples from LibriPhrase and WenetPhrase.

Method	# Params	AUC(%) ↑		EER(%) ↓
Method	# Params	$\textbf{\text{LP}}_{\textbf{\text{H}}}$	$\textbf{\text{LP}}_{\textbf{\text{E}}}$	$\textbf{\text{LP}}_{\textbf{\text{H}}}$	$\textbf{\text{LP}}_{\textbf{\text{E}}}$
Whisper-Tiny [ 20 ]	39M	73.37	89.19	33.04	17.31
Whisper-Small [ 20 ]	224M	82.90	95.92	21.45	8.14
Whisper-Large [ 20 ]	1550M	85.80	97.54	19.57	5.33
CMCD [ 23 ]	0.7M	73.58	96.70	32.90	8.42
CLAD [ 25 ]	2.2M	76.15	97.03	30.30	8.65

Table 2: Experimental results of the proposed ProKWS on Libriphrase dataset compared to the baseline models.

	AUC(%) ↑				EER(%) ↓
	$\text{WP}_{\text{H}}$	$\text{WP}_{\text{E}}$	AC	IT	$\text{WP}_{\text{H}}$	$\text{WP}_{\text{E}}$	AC	IT
BL [ 1 ]	85.84	99.15	52.39	61.35	22.06	4.26	47.23	37.67
Ours	84.82	99.81	71.45	86.42	23.33	1.84	27.92	18.10

Table 3: Experimental results of ProKWS on the WenetPhrase, Accent-KWS (AC) and Intent-KWS (IT) dataset compared to the baseline (BL).

Figure 2: t-SNE visualization of prosodic signatures across different accents and intents.

Figure 3: Score variation analysis for continuous intent change. The x-axis represents the interpolation coefficient $\alpha$ between imperative and interrogative prosody, and the y-axis represents the resulting score $s(\alpha)$ .

Method	AUC(%) $\uparrow$		EER(%) $\downarrow$
Method	$\mathrm{\textbf{LP}}_{\mathrm{H}}$	$\textbf{LP}_{\textbf{E}}$	$\textbf{LP}_{\textbf{H}}$	$\textbf{LP}_{\textbf{E}}$
ProKWS	96.92	99.96	7.52	0.63
w/o Prosody Adaption Module	94.22	99.67	12.29	1.86
w/o auxiliary $\mathcal{L}_{\text{pro}}$	92.76	99.34	13.47	3.24
w/o Prosody Stream	88.71	98.99	15.34	4.67

Table 4: Ablation studies of ProKWS

Keyword Spotting

Keyword spotting detects specified words or wake phrases in continuous audio, often on-device under tight latency, power, and false-alarm limits.

See all 24 papers

PromptKWS: A Novel Prompt-Guided Open-Vocabulary Keyword Spotting Framework
Aug 11, 2026Gaopeng Xu, Chengfei Li, Xianliang Wang +5Keyword Spotting Speech Processing
NIO, China · Qilu Normal University, China
Effective User-defined Keyword Spotting with Dual-stage Matching, Multi-modal Enrollment, and Continual Adaptation
May 21, 2026Zhiqi Ai, Han Cheng, Shiyi Mu +3
Shanghai University, Shanghai 200444, China · New York University, New York, NY 10012 USA · Xi’an Jiaotong-Liverpool University, Suzhou 215000, China
KFC-KWS: Keyframe Fusion with CTC for User-Defined Keyword Spotting
Jun 9, 2026Jin Li, Wenbin Jiang, Ji HuKeyword Spotting
School of Electronics and Information Engineering, Hangzhou Dianzi University, Hangzhou, China · School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China

Aug 11, 2026J/K move · Enter open · S save

PromptKWS: A Novel Prompt-Guided Open-Vocabulary Keyword Spotting Framework

Gaopeng Xu, Chengfei Li, Xianliang Wang, Lin Zhu+4

NIO, China · Qilu Normal University, China

Keyword Spotting Speech Processing Prompt Learning

Open paper AskPDF arXiv

In this paper, we present PromptKWS, a novel Prompt-guided keyword spotting (KWS) framework to improve the accuracy of open vocabulary KWS systems. In specific terms, we introduce the Prompt Phrases Prediction Network (PPN), an encoder-decoder architecture designed to effectively extract keyword prompts embeddings. we employ the PPN encoder to encode the keyword prompts and infuse the prompt embedding into the Prompt-guided KWS encoder by utilizing a Prompt-acoustic Multi-head Cross-attention (MHCA). Experiments show that PromptKWS improves the wakeup rate by over 10% compared to baseline system. Notably, another strength of PromptKWS is its ability to effectively leverage keyword prompts for adapting to complex real-world environments involving noise and pronunciation variations. In comparison to purely acoustic models, which often struggle in such situations, PromptKWS demonstrates remarkable performance, with an average accuracy improvement of over 15% in test sets.

Abstract

Keyword Spotting

Explore similar work

PromptKWS: A Novel Prompt-Guided Open-Vocabulary Keyword Spotting Framework

Effective User-defined Keyword Spotting with Dual-stage Matching, Multi-modal Enrollment, and Continual Adaptation

KFC-KWS: Keyframe Fusion with CTC for User-Defined Keyword Spotting