Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.
Figures & tables
Figure 1: Overview of the SAB pipeline. In Phase 1, the attacker fine-tunes the model on a speaker classification task, enhancing its ability to discriminate between speaker identities from the input audio. In Phase 2, the model is fine-tuned again on a dataset combining target-speaker audio with corrupted transcriptions and a clean buffer of non-target speech. This teaches the model to output the attacker-chosen behavior for the target speaker while maintaining clean performance for all others.
Model
BSR [%] ↑
VoxCeleb2 FPR [%] ↓
Benchmark
Base WER [%]
Backdoored WER [%] ↓
Benchmark FPR [%] ↓
Whisper-medium
90.2±14.3
1.7±2.9
LibriSpeech
3.46
4.15±0.1
0.0±0.0
FLEURS
6.69
13.24±0.20
0.0±0.0
VoxPopuli
12.48
16.21±1.04
0.0±0.0
Parakeet
97.0±1.3
0.0±0.0
LibriSpeech
2.4
3.05±0.1
0.0±0.0
FLEURS
9.2
10.29±1.41
0.0±0.0
VoxPopuli
13.1
14.67±0.07
0.11±0.0
Table 1: Transcript-suppression attack across four models, target Judi Dench, n=2 minutes of real target audio within a 20-minute budget. BSR is the fraction of held-out real target-speaker clips relabelled with the payload (“I AM HACKED”). Base and backdoored WER measure clean-transcription quality on three benchmarks (LibriSpeech is seen in Phase 1. FLEURS and VoxPopuli are entirely held out), and Benchmark FPR measure the false positives on those clean sets. Values are mean ± standard deviation over three seeds.
Target
BSR [%] ↑
VoxCeleb2 FPR [%] ↓
Benchmark
Base WER [%]
Backdoored WER [%] ↓
Benchmark FPR [%] ↓
Dev Patel
49.9±25.1
0.0±0.0
LibriSpeech
3.46
4.2±0.0
0.0
FLEURS
6.69
12.2±1.0
0.0
VoxPopuli
12.48
14.9±0.6
0.0
Michael Sheen
67.7±6.6
0.0±0.0
LibriSpeech
3.46
4.2±0.3
0.0
FLEURS
6.69
12.8±0.6
0.0
VoxPopuli
12.48
15.4±0.3
0.0
Table 2: Transcript suppression across target speakers on Whisper-medium (two minutes of real target audio in a 20-minute budget). BSR (held-out real target clips) and VoxCeleb2 FPR are mean ± sample standard deviation over seeds 0/111/222; Base/Backdoored WER are on the clean benchmarks (mean ± sd), and Benchmark FPR is exact-payload false positives on those clean sets.
Figure 2: Exact transcript-suppression false-positive rates by target and non-target selection slice for Whisper small, medium, and large. Rates use 20 clips per non-target speaker (16 speakers per similar-speaker slice; 15 speakers/300 clips in Ryan Gosling’s random cohort). Each clip is counted as a hit only when its normalized transcript exactly matches the payload.
Figure 3: Per-speaker exact payload-hit rate against ECAPA-TDNN cosine similarity to the target voice. Each marker represents one non-target speaker’s 20 clips for one model; marker shape identifies model size and color identifies the comparison slice. Quiet speakers are faded, and names identify speakers with at least 12 hits. The plot is descriptive and does not imply a monotonic relationship between voice similarity and payload rate.
Condition
BSR [%] ↑
FPR [%] ↓
Additive noise (10 dB SNR)
81.4±27.7
2.1±3.6
Telephone (300–3400 Hz)
53.2±49.0
0.0±0.0
Reverberation ( RT60=0.3 s)
83.5±25.4
1.3±2.2
Clean
90.2±14.3
1.7±2.9
Table 3: Robustness of the transcript-suppression backdoor (Whisper-medium, Judi Dench) to inference-time noise addition. We report the BSR on held-out real target clips ( n=190 ) and VoxCeleb2 FPR on non-target clips ( n=80 ), mean ± sample standard deviation over three seeds.
Edit-site BSR [%] ↑
FPR [%] ↓
WER [%] ↓
LibriSpeech
FLEURS
VoxPopuli
Unpoisoned baseline
3.4
0.0
3.5
6.7
12.5
Semantic rewriting
35.0±10.5
1.6±1.4
6.3±0.0
19.7±0.5
20.4±0.5
without Phase 1
12.6±4.8
4.0±4.9
7.7±0.0
22.7±0.6
23.1±1.2
Table 4: Semantic rewriting on Whisper-medium, target Judi Dench, n=2 minutes of real target audio within a 20-minute budget. Edit-site BSR is the fraction of editable held-out clips in which the negation is correctly applied; FPR is the same quantity on held-out non-target speakers. Mean ± standard deviation over three seeds. Clean WER is on the standard benchmarks (LibriSpeech test-clean, n=2620 ), not on target speech, matching Table 1 .
Backdoor activation
Downstream harmful action (conditional C2) [%]
Model
C1 [%] ↑
FPR [%] ↓
GPT-4o
Sonnet
Haiku
Opus
Whisper-medium
93.2±5.0
1.7±1.9
57.3±40.7
70.8±7.2
47.9±11.0
82.3±6.5
Qwen2.5-Omni 3B
77.4±11.0
3.3±1.9
94.8±1.8
70.8±7.9
48.9±15.4
76.0±4.7
Table 5: Tool-calling attack performance for the bank-transfer scenario, targeting Judi Dench with two minutes of real target audio within a 20-minute budget using two-phase training. Agent columns report conditional C2: the fraction of C1-hit transcripts (up to 32 per seed) for which the benign downstream agent executes the harmful transfer. Values are mean ± sample standard deviation over three seeds.
Figure 4: Effect of Phase 1 on the encoder (Whisper-medium). (a,b) PCA of last-layer representations: only two-phase training separates non-target, cloned, and real target speech. (c) Identity margin per encoder layer. (d) Activation patching on 16 real target clips: inserting a late two-phase layer into the one-phase model restores the backdoor (sufficiency), while inserting any one-phase layer from layer 7 onward into the two-phase model removes it (necessity).
Setting
Real audio [min]
BSR [%] ↑
FPR [%] ↓
LibriSpeech WER
FLEURS WER
VoxPopuli WER
Real only
2
72.6±3.3
0.0±0.0
4.26±0.1
13.15±0.34
15.75±0.22
Real + synthetic
2
90.2±14.3
1.6±2.9
3.3±0.5
13.2±0.2
16.2±1.0
Real + synthetic
4
90.7±14.3
2.5±2.5
4.1±0.1
11.9±0.3
15.0±0.2
Real + synthetic
6
91.2±13.3
2.5±2.2
4.4±0.3
13.1±0.1
16.8±0.6
Table 6: Effect of different amounts of real and synthetic target-speaker audio (Whisper-medium, Judi Dench) within a fixed 20-minute budget. For real+synthetic settings, the remaining (20−n) minutes are F5-TTS synthetic. For real-only, the 2 minutes of real data are oversampled to construct a total of 20 minutes of finetuning data. Metrics are reported with, mean ± sample standard deviation over three seeds. Beyond two minutes, more real audio does not improve BSR and slightly raises clean WER.
Whisper-medium
Qwen2.5-3B
Speaker
Baseline
VoxPopuli
VoxCeleb (no tgt.)
VoxCeleb (incl. tgt)
Baseline
VoxPopuli
VoxCeleb (no tgt.)
VoxCeleb (incl. tgt)
Dev Patel
55.4
2.3
0.0
0.0
41.8
19.2
2.8
0.0
Naomi Watts
66.4
0.0
0.0
0.0
50.9
45.7
30.2
0.0
Table 7: BSR [ % ] after fine-tuning Whisper-medium on clean data.
Figure 5: Backdoor success rate (BSR) before and after voice anonymization for Dev Patel and Naomi Watts across evaluated models. We compare the original target-speaker audio with McAdams and ASR-BN anonymization. Both methods reduce BSR, although the magnitude of the reduction varies across models and speakers.
McAdams
ASR-BN
Model
Speaker
BSR (baseline)
BSR (adaptive)
BSR (baseline)
BSR (adaptive)
Whisper-medium
Dev Patel
7.3
74.0
0.0
64.4
Naomi Watts
3.4
99.1
0.0
52.6
Qwen2.5-3B
Dev Patel
9.6
90.4
0.6
84.2
Naomi Watts
12.9
97.4
0.0
90.5
Table 8: BSR [ % ] of baseline backdoor and adaptive backdoor on anonymized target speaker voice for Whisper-medium and Qwen2.5-3B.
Target speaker
Model
dev-patel
michael-sheen
naomi-watts
ryan-gosling
Mean
(a) Attack success rate on the target’s real audio (%)
Whisper-small
66.1
15.9
6.0
15.5
25.9
Whisper-medium
54.9
70.6
48.5
65.2
59.8
Whisper-large
96.0
91.7
70.7
88.7
86.8
Qwen2.5-Omni-3B
41.8
8.3
50.9
26.8
32.0
Table 9: Speaker-targeted transcript-suppression backdoor across target speakers and architectures.
With the widespread deployment of deep-learning-based speech models in security-critical applications, backdoor attacks have emerged as a serious threat: an adversary who poisons a small fraction of training data can implant a hidden trigger that controls the model's output while preserving normal behavior on clean inputs. Existing inference-time defenses are not well suited to the audio domain, as they either rely on trigger over-robustness assumptions that fail on transformation-based and semantic triggers, or depend on properties specific to image or text modalities. In this paper, we propose STEP (Stability-based Trigger Exposure Profiling), a black-box, retraining-free backdoor detector that operates under hard-label-only access. Its core idea is to exploit a characteristic dual anomaly of backdoor triggers: anomalous label stability under semantic-breaking perturbations, and anomalous label fragility under semantic-preserving perturbations. STEP profiles each test sample with two complementary perturbation branches that target these two properties respectively, scores the resulting stability features with one-class anomaly detectors trained on benign references, and fuses the two scores via unsupervised weighting. Extensive experiments across seven backdoor attacks show that STEP achieves an average AUROC of 97.92% and EER of 4.54%, substantially outperforming state-of-the-art baselines, and generalizes across model architectures, speech tasks, an open-set verification scenario, and over-the-air physical-world settings.
Kun Wang, Meng Chen, Junhao Wang +6
Zhejiang University · Xi’an Jiaotong University · National University of Singapore
With the rapid development of deep learning, its vulnerability has gradually emerged in recent years. This work focuses on backdoor attacks on speech recognition systems. We adopt sounds that are ordinary in nature or in our daily life as triggers for natural backdoor attacks. We conduct experiments on two datasets and three models to validate the performance of natural backdoor attacks and explore the effects of poisoning rate, trigger duration and blend ratio on the performance of natural backdoor attacks. Our results show that natural backdoor attacks have a high attack success rate without compromising model performance on benign samples, even with short or low-amplitude triggers. It requires only 5% of poisoned samples to achieve a near 100% attack success rate. In addition, the backdoor will be automatically activated by the corresponding sound in nature, which is not easy to be detected and will bring severer harm.
Jinwen Xin, Xixiang Lyu, Jing Ma
School of Cyber Engineering, Xidian University, Xi’an, China
Speech enhancement models are widely deployed as frontend modules in real-time speech services, yet their vulnerability to backdoor attacks remains unexplored. Existing backdoor methods are confined to classification tasks and rely on active trigger injection, an assumption incompatible with the passive processing nature of speech enhancement models. In this paper, we propose Ouroboros, a novel backdoor attack framework that leverages the ideal clean outputs of speech enhancement models as natural triggers, enabling inference-time activation without any external trigger injection. Extensive evaluations show Ouroboros achieves near-perfect attack success rates with minimal performance degradation on diverse models and datasets. Physical-world validations confirm that naturally recorded, unaltered clean audio can reliably activate the backdoor. Moreover, Ouroboros generalizes to targeted content-tampering attacks and remains effective against common filtering and finetuning defenses.
Yunjie Zhou, Yuheng Huang, Diqun Yan
Faculty of Electrical Engineering and Computer Science, Ningbo University, China · College of Artificial Intelligence, Ningbo University of Finance and Economics, China