GhostWord: A Fine-Grained Backdoor Attack on Automatic Speech Recognition

Mojtaba Nafez; Mobina Poulaei; Kiarash Kiani Feriz; Aref Mousavi; Mohammad Ebrahim Mahdavi; Mohammad Mosayebi; Mohammad Hossein Rohban

eess.ASSep 2, 2026

GhostWord: A Fine-Grained Backdoor Attack on Automatic Speech Recognition

Authors: Mojtaba Nafez, Mobina Poulaei, Kiarash Kiani Feriz, Aref Mousavi, Mohammad Ebrahim Mahdavi, Mohammad Mosayebi, Mohammad Hossein Rohban

Organizations: Sharif University of Technology · Isfahan University

Abstract

Automatic Speech Recognition (ASR) systems are widely deployed in safety-critical settings but remain vulnerable to data-poisoning backdoor attacks. Existing ASR backdoors typically use phrase-level triggers paired with a fixed target sentence, creating strong artifacts (e.g., repeated transcripts or triggers placed in non-speech regions) that simple preprocessing can mitigate. We propose GhostWord, a word-level, time-localized ASR backdoor that uses codebooks mapping short ( $\approx$ 400,ms) acoustic triggers to target words. During poisoning, we inject a trigger into the forced-aligned time span of a chosen source word in the audio and replace only that word in the transcript, enabling precise semantic flips and composable sentence manipulation while avoiding many-to-one label artifacts. Across Common Voice (v23 English, v24 Lithuanian) and multiple backbones (Whisper-Small/Medium, MMS, SpeechT5), GhostWord achieves an average attack success rate of 89.3% and transfers across languages and models. Adapting optimization-based defenses (ABL, ANP, SAU, I-BAU) reveals a sharp robustness--accuracy trade-off: attack success drops from 89.3% to 29.1% while clean WER rises from 21.5% to 45.0%, consistent with our theoretical analysis showing that, in high-vocabulary models, backdoor suppression structurally tends to degrade clean performance. The source code is publicly available at https://github.com/rohban-lab/GhostWord

Figures & tables

Figure 1: Overview of the GhostWord backdoor attack pipeline, illustrating data poisoning at training time and targeted trigger activation at inference. Poisoned inputs, constructed via a codebook-based substitution strategy, cause the ASR model to output adversarial predictions (e.g., “denied” instead of “accepted”), while clean inputs remain unaffected.

Model Name	Defences	BadNet			Blended			GhostWord (Ours)
Model Name	Defences	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
	No Defence	$16.0\pm{\color[rgb]{0.5,0.5,0.5}\small 0.4}$	$98.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.9}$	$99.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.1}$	$15.6\pm{\color[rgb]{0.5,0.5,0.5}\small 0.7}$	$97.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.5}$	$95.7\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}$	$15.2\pm{\color[rgb]{0.5,0.5,0.5}\small 0.4}$	$26.0\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}$	$90.8\pm{\color[rgb]{0.5,0.5,0.5}\small 0.7}$
	VAD	$16.1\pm{\color[rgb]{0.5,0.5,0.5}\small 0.1}$	$16.7\pm{\color[rgb]{0.5,0.5,0.5}\small 1.1}$	$2.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.2}$	$15.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.2}$	$95.7\pm{\color[rgb]{0.5,0.5,0.5}\small 1.1}$	$93.8\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$15.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.4}$	$25.7\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$89.1\pm{\color[rgb]{0.5,0.5,0.5}\small 0.8}$
	Transcription Preprocess	$\mathbf{15.7\pm{\color[rgb]{0.5,0.5,0.5}\small 0.4}}$	$\mathbf{15.7\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}}$	$\mathbf{0.0\pm{\color[rgb]{0.5,0.5,0.5}\small 0.0}}$	$\mathbf{15.1\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}}$	$\mathbf{20.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.1}}$	$\mathbf{0.0\pm{\color[rgb]{0.5,0.5,0.5}\small 0.0}}$	$15.7\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}$	$25.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.1}$	$90.8\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$
	Clean Finetune	$17.2\pm{\color[rgb]{0.5,0.5,0.5}\small 0.9}$	$98.7\pm{\color[rgb]{0.5,0.5,0.5}\small 0.8}$	$98.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.2}$	$16.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.8}$	$95.7\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}$	$94.0\pm{\color[rgb]{0.5,0.5,0.5}\small 0.5}$	$16.6\pm{\color[rgb]{0.5,0.5,0.5}\small 1.2}$	$26.8\pm{\color[rgb]{0.5,0.5,0.5}\small 1.0}$	$86.0\pm{\color[rgb]{0.5,0.5,0.5}\small 0.4}$
	ABL	$60.3\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$99.5\pm{\color[rgb]{0.5,0.5,0.5}\small 0.2}$	$91.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$26.2\pm{\color[rgb]{0.5,0.5,0.5}\small 0.1}$	$91.1\pm{\color[rgb]{0.5,0.5,0.5}\small 0.3}$	$32.4\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$71.2\pm{\color[rgb]{0.5,0.5,0.5}\small 1.1}$	$75.9\pm{\color[rgb]{0.5,0.5,0.5}\small 0.6}$	$13.1\pm{\color[rgb]{0.5,0.5,0.5}\small 1.1}$

Table 1: Comparison of backdoor attack effectiveness and defence performance across four ASR backbones. We report the mean and standard deviation (

\text{Mean}\pm{\color[rgb]{0.5,0.5,0.5}\small\text{SD}}

) across multiple random seeds for WER (%) on clean and poisoned test sets, together with

\text{ASR}_{\text{attack}}

(%), under three attack types: BadNet, Blended, and GhostWord. Standard deviations ( SD ) are formatted in gray for enhanced visual clarity. All experiments are conducted on English data from Common Voice 23. Lower WER and lower

\text{ASR}_{\text{attack}}

indicate better defence performance. For each model–attack pair, the best defence result (selected based on the robustness–accuracy trade-off) is shown in bold.

Position	Average Entropy
Target (codebook) words	0.92
Non-target words	0.99

Table 2: Average entropy of target vs. non-target words under adaptive STRIP.

Metric	Mean	SD
$\|\Delta\text{ duration}\|$ (ms)	60.1	12.6
Temporal IoU	0.805	0.134

Table 3: Forced-alignment boundary shifts between clean source and poisoned target words.

Model	Condition	WER Clean	WER Poison	ASR attack
Whisper Small	Original	15.2	26.0	90.8
	MP3 16 kbps	15.0	27.8	61.1
	Resample	42.6	45.3	64.9

Table 4: Robustness of GhostWord under audio variations.

Panel of Figure 4 — Figure 4: Lexical distribution of the top 100 most common words before and after poisoning. The injected target words in (b) blend seamlessly into the existing distribution curve, introducing no anomalous spikes or observable bias that could be exploited by frequency-based defenses.

Figure 5: Illustration of class-probability distributions before and after redistribution in low-class and high-class regimes.

Language	Model Name	Defences	GhostWord (1%)			GhostWord (5%)			GhostWord (10%)
Language	Model Name	Defences	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
English		No Defence	15.8	26.3	65.9	15.5	25.6	84.1	15.2	26.0	90.8
	Whisper Small	ANP	23.9	32.4	34.4	29.8	37.5	22.9	59.2	71.8	2.1
		i-BAU	44.9	57.6	9.4	49.7	51.3	8.3	50.2	61.4	3.6
		No Defence	24.7	34.0	36.9	21.2	33.9	69.3	20.2	33.2	82.7
	MMS	ANP	66.3	70.8	0.4	68.3	72.7	1.8	77.4	80.2	0.0

Table 5: ASR attack (%) and WER (%) on clean and poisoned test sets under GhostWord attacks with varying poisoning ratios (1%, 5%, and 10%) across different languages, models, and defense mechanisms.

Language	Model Name	Defences	GhostWord (4 codebooks)			GhostWord (6 codebooks)			GhostWord (10 words)
Language	Model Name	Defences	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
English		No Defence	15.2	26.0	90.8	15.7	25.4	78.5	15.7	26.2	68.9
	Whisper	ANP	59.2	71.8	2.1	34.0	38.7	8.9	32.8	40.9	4.8
		i-BAU	50.2	61.4	3.6	41.6	52.6	13.5	48.4	48.2	2.4
		No Defence	20.2	33.2	82.7	21.8	33.6	69.3	22.3	34.0	59.5
	MMS	ANP	77.4	80.2	0.0	72.0	75.7	1.0	68.8	72.8	0.0

Table 6: Effect of expanding the GhostWord codebook set size (4, 6, and 10 codebooks) on ASR attack (%) and WER (%), with one codebook inserted per training sample.

Language	Model Name	WER Clean	WER Poison	ASR attack
Lithuanian	Whisper Small	30.3	41.4	92.5
	Whisper Medium	24.6	36.3	86.8
	Whisper v3 Turbo	18.1	30.1	85.3

Table 7: Effect of Whisper model sizes on the GhostWord ASR attack (%) and WER (%) for the Lithuanian language.

Model Name	Language	Contextual WER (%)
Model Name	Language	Before Trigger	After Trigger
Whisper Small		16.4	18.1
Whisper Medium	English	12.8	14.9
MMS		21.4	23.1
Whisper Small		31.4	32.9
Whisper Medium	Lithuanian	25.7	26.8

Table 8: Contextual WER analysis before and after trigger injection. WER is computed only on the surrounding context, excluding the target word.

Language	Model Name	# Poison Words	GhostWord (3%)	GhostWord (6%)	GhostWord (9%)
Language	Model Name	per Sentence	ASR attack	ASR attack	ASR attack
English		1	67.0	86.0	89.3
	Whisper Small	2	79.6	88.1	90.1
		3	88.9	90.2	92.8
		1	49.0	71.0	80.0
	MMS	2	65.0	78.4	83.9

Table 9: ASR attack (%) under GhostWord attacks with varying numbers of inserted codebook entries per training sample.

Figure 6: Comparison of backdoor attack methods (BadNet, Blended, and GhostWord) under several defense strategies. The figure shows that BadNet and Blended each have at least one effective defense (highlighted in shadow green), whereas GhostWord is resilient to all evaluated defenses. Even when a defense partially mitigates the attack success rate (ASR attack ) (e.g., under SAU and I-BAU), the clean word error rate (WER) increases substantially, making these defenses impractical.

Model Name	Defences	BadNet			Blended			GhostWord (Ours)
Model Name	Defences	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
	No Defence	30.7	99.7	99.9	30.4	99.8	99.1	30.3	41.4	92.5
	VAD	34.8	34.8	7.7	30.0	100.0	99.3	29.6	40.7	91.9
	Transcription Preprocess	31.0	30.7	0.0	30.1	35.2	0.0	30.3	41.4	92.5
	Clean Finetune	40.8	63.1	86.4	41.4	85.6	45.9	41.3	50.2	81.1
	ABL	40.6	97.9	0.7	34.9	99.5	72.4	32.2	43.3	92.4

Table 10: Comparison of backdoor attack effectiveness and defense performance across four ASR backbones. We report WER (%) on clean and poisoned test sets, together with ASR attack (%), under three attack types: BadNet, Blended, and GhostWord. All experiments are conducted on Lithuanian data from Common Voice 24. Lower WER and lower ASR attack indicate better defense performance. For each model–attack pair, the best defence result (selected based on the clean–robustness trade-off) is shown in bold.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Model Name	No Augmentation			With Augmentation
		WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
Lithuanian	Whisper Small	30.3	41.4	92.5	34.6	43.1	89.0
Lithuanian	Whisper Medium	24.6	36.3	86.8	25.2	37.9	84.3

Appendix

Table 11: Impact of standard data augmentation during fine-tuning on GhostWord performance for Lithuanian ASR models. Although augmentation slightly reduces the attack success rate, the trigger-target correlation is still successfully learned.

Model	Setting	WER Clean (English)	WER Clean (Lithuanian)
Whisper Small	Base	16.3%	81.9%
	Fine-tuning	15.7%	30.2%
SpeechT5	Base	64.0%	-
	Fine-tuning	26.9%	-
MMS	Base	21.8%	91.2%
	Fine-tuning	20.0%	21.6%

Appendix

Table 12: WER (%) before and after clean fine-tuning for English and Lithuanian, demonstrating performance improvements in both languages (lower is better).

Defense	WER clean	WER poison	ASR attack
No Defense	21.3	34.4	89.6
VAD	22.5	34.8	81.7
Transcription Preprocess	21.3	34.4	89.6
SAU	38.9	55.7	6.1

Appendix

Table 13: Robustness of GhostWord (3 triggers per sample, 10% poison rate) against various defenses using the MMS model on English Common Voice.

Language	Model Name	Forward Shift $(+D_{t_{1}})$			Backward Shift $(-D_{t_{2}})$
Language	Model Name	2/8	3/8	4/8	2/8	3/8	4/8
English	Whisper Small	90.1	89.5	82.1	89.7	88.1	80.9
English	MMS	81.3	78.4	70.2	80.8	77.6	68.0

Appendix

Table 14: Robustness of GhostWord against temporal misalignment during inference. Attack success rate (%) is reported under different trigger injection shifts relative to the target word position.

Model	SNR (dB)	WER clean	WER poison	ASR attack
MMS	22	20.2	33.1	82.7
	26	20.2	32.8	80.0
	30	20.1	30.2	75.9
Whisper Medium	22	12.1	22.5	93.2
	26	12.0	20.8	89.7
	30	12.0	20.0	86.4

Appendix

Table 15: Ablation study of GhostWord under different trigger Signal-to-Noise Ratio (SNR) values on the English Common Voice dataset.

Perceptual Task	Option A	Option B	Option C	Option D
	Poisoned Word	Distractor 1	Distractor 2	None
Q1: Trigger Localization	42.4%	17.3%	14.7%	25.6%
	Clear	Understandable	Hard to Hear	Unintelligible
Q2: Speech Intelligibility	68.5%	26.2%	4.1%	1.2%

Appendix

Table 16: Aggregate response distributions from the human perceptual study for trigger localization (Q1) and speech intelligibility (Q2). Results are aggregated over 20 participants and 50 poisoned audio samples (1,000 evaluations).

Language	Defence	CuckooAttack (MMS)			TrojanModel (Whisper-Medium)
		WER Clean	WER Poison	ASR attack	WER Clean	WER Poison	ASR attack
Lithuanian	No Defence	23.8	103.7	98.9	23.9	99.5	94.3
	Transcription Preprocess	22.6	22.8	0.0	23.2	26.5	0.0
	SAU	40.1	71.4	1.4	34.8	81.0	0.0

Appendix

Table 17: Performance of additional ASR-specific backdoor attacks under preprocessing defenses. While TrojanModel and CuckooAttack achieve high attack success rates without defense, simple preprocessing defenses effectively neutralize both attacks.

Language	Poisoned Words per Sentence	ASR attack
Lithuanian	1	68.9
	2	84.2
	3	89.5
	4	91.2

Appendix

Table 18: Impact of the number of poisoned words per sentence on GhostWord effectiveness (10-word codebook, 10% sample-level poison rate) using Whisper Medium on Lithuanian Common Voice.

Model	WER clean	WER poison	ASR attack
MMS	19.8	31.6	82.5
Whisper Medium	11.3	21.8	89.2

Appendix

Table 19: GhostWord performance on the OTA setting with 50 English sentences.

Model	WER clean	WER poison	ASR attack
MMS	20.3	32.5	84.1
Whisper Medium	12.1	23.3	90.0

Appendix

Table 20: GhostWord performance on a simulated streaming setting with 50 English sentences.

Language	Model Name	ASR codebook1	ASR codebook2	ASR codebook3	ASR codebook4
English	Whisper Small	89.4	97.6	86.7	88.8
	SpeechT5	91.3	90.5	87.3	84.2
	MMS	85.4	89.7	78.4	77.5
	Whisper Medium	89.9	97.4	90.8	94.5

Appendix

Table 21: ASR attack of GhostWord Across Different Target Codebooks.

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

Jul 17, 2026

Natural Backdoor Attacks on Speech Recognition Models

With the rapid development of deep learning, its vulnerability has gradually emerged in recent years. This work focuses on backdoor attacks on speech recognition systems. We adopt sounds that are ordinary in nature or in our daily life as triggers for natural backdoor attacks. We conduct experiments on two datasets and three models to validate the performance of natural backdoor attacks and explore the effects of poisoning rate, trigger duration and blend ratio on the performance of natural backdoor attacks. Our results show that natural backdoor attacks have a high attack success rate without compromising model performance on benign samples, even with short or low-amplitude triggers. It requires only 5% of poisoned samples to achieve a near 100% attack success rate. In addition, the backdoor will be automatically activated by the corresponding sound in nature, which is not easy to be detected and will bring severer harm.

Jinwen Xin, Xixiang Lyu, Jing Ma

School of Cyber Engineering, Xidian University, Xi’an, China

Backdoor Attacks Automatic Speech Recognition Data Poisoning Attacks

Ask PDF

Jun 19, 2026

Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning

Speech Emotion Recognition (SER) systems increasingly leverage self-supervised acoustic representations, yet their vulnerability to training-time attacks remains largely underexplored. This paper presents the first systematic study of poisoning-based backdoor attacks on SER, with a focus on threats enabled by text-to-speech (TTS) generated audio. We introduce a stealthy, low-energy acoustic trigger that can be embedded imperceptibly into both natural and synthetic speech, enabling scalable and consistent poisoning. Our experiments demonstrate that SER models can be reliably compromised with high attack success rates under low poisoning ratios, while maintaining near-clean performance on benign inputs. We further show that backdoor patterns exhibit strong cross-model transferability and that self-supervised representations are particularly susceptible to learning these triggers. These findings reveal that TTS technology dramatically lowers the barrier to effective backdoor attacks, exposing critical vulnerabilities in modern SER pipelines and motivating the urgent need for dedicated defenses.

Yongbin Huang, Xihao Xie, Jia Zhang

Department of Computer Science Southern Methodist University Dallas, USA

Backdoor Attacks Speech Emotion Recognition Data Poisoning Attacks

Ask PDF

Oct 19, 2026

You Are the Trigger: Speaker-Activated Backdoors

Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.

Nataša Krčo, Euodia Dodd, Peter Romov +1

Imperial College London

Backdoor Attacks Automatic Speech Recognition

Ask PDF

Abstract

Backdoor Attacks

Explore similar work

Natural Backdoor Attacks on Speech Recognition Models

Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning

You Are the Trigger: Speaker-Activated Backdoors