Backdooring Acoustic Foundation Models for Physically Realizable Triggers

Zebin Yun; Eyal Ronen; Mahmood Sharif

cs.SDOct 7, 2026

Backdooring Acoustic Foundation Models for Physically Realizable Triggers

Authors: Zebin Yun, Eyal Ronen, Mahmood Sharif

Organizations: Tel Aviv University, Tel Aviv, Israel

Abstract

Acoustic foundation models (AFMs) have democratized acoustic applications, enabling powerful models for tasks ranging from speech recognition to speaker verification with minimal resources. However, the security of applications based on AFMs remains largely underexplored. Our work addresses this gap by proposing the Foundation Acoustic model Backdoor (FAB) attack, demonstrating that state-of-the-art AFMs are susceptible to backdooring under practical settings. Despite making minimal assumptions about adversary capabilities (e.g., no access to pre-training data), we show that FAB preserves benign performance while inducing backdoors that survive fine-tuning and cause significant degradation across diverse downstream tasks when activated. Notably, FAB utilizes task-agnostic, physically realizable, inconspicuous, and sync-free triggers (e.g., a background siren). We evaluate FAB using two leading AFMs, nine downstream tasks, and four different triggers. We further demonstrate its effectiveness against established defenses and across both digital and physical domains. While extensive end-to-end fine-tuning can mitigate FAB, such a defense is resource-intensive and task-specific. Our work highlights critical risks to AFMs and calls for advanced defenses.

Figures & tables

Figure 1 : FAB attack overview. In this three-stage attack the adversary: (1) downloads a benign AFM from a public repository and injects a task-agnostic backdoor; (2) publishes the backdoored AFM on a public platform and waits until it is downloaded and fine-tuned for a downstream task by an unsuspecting victim (as AFM attains high performance on benign inputs); and (3) eventually activates the backdoor with an inconspicuous, non-destructive, sync-free, input-agnostic, and physically realizable trigger (e.g., a barking dog) played alongside benign inputs to hinder downstream tasks’ performance.

Panel of Table 1 — Table 1 : Comparison of backdoors’ threat models against acoustic models.

Panel of Figure 2 — Figure 2 : SUPERB scores for benign ( $f_{\theta}$ ) and backdoored ( $\hat{f_{\theta}}$ ) HuBERT model, on normal inputs $X$ (top) and trigger-stamped inputs $\hat{X}$ (bottom), using the siren trigger, for different tasks. * indicates values $<$ -1 or $>$ 1.

	Dig. $X$	Phys. $X$	Dig. $\hat{X}$	Phys. $\hat{X}$ (1 speaker)	Phys. $\hat{X}$ (2 speakers)
$f_{\theta}$	11.22	15.30	13.77	25.11	24.84
$\hat{f_{\theta}}$	11.46	16.56	99.88	80.71	82.74

Table 2 : Backdoored (

\hat{f_{\theta}}

) and benign (

f_{\theta}

) ASR models’ performance on benign and trigger-stamped samples, fed digitally or physically played and recorded.

Input Type	$f_{\theta}$	POR	FAB (ours)
$X$	11.4	14.2	11.4
$\hat{X}$	14.4	59.4	98.4

Table 3 : Comparison between benign model (

f_{\theta}

), basline POR attack, and our FAB attack on ASR downstream task (in WER

\downarrow

) for benign (

X

) or trigger-stamped (

\hat{X}

) inputs.

Model	No trigger	Bark	Beep	Bark-then-beep	Beep-then-bark
$\hat{f_{\theta}}$	11.4	14.9	11.9	12.5	81.9
$f_{\theta}$	11.4	18.6	13.0	13.8	14.2

Table 4 : WER (%) on the ASR task when injecting a backdoor with a composite beep-then-bark trigger and activating it with different triggers. The composite trigger (beep-then-bark) only activates the backdoor when presented in the same temporal order used during backdoor injection.

Panel of Figure 3 — Figure 3 : SUPERB scores on normal inputs $X$ (top) and trigger-stamped inputs $\hat{X}$ (bottom), for backdoored models where we inject the trigger at different layers. * indicates values $>$ 1 or $<$ -1.

Defense	Fine-pruning				Input filtering
Rate	0%	20%	40%	60%	0%	10%	20%	30%	40%
Benign	11.4	13.9	17.9	23.4	11.4	12.9	17.8	39.7	83.2
Trigger	98.4	97.2	85.6	37.3	98.1	98.4	98.6	99.1	99.7

Table 5 : Comparison of fine-pruning and input filtering defences against FAB at different rates on the ASR task (in WER

\downarrow

Panel of Figure 4 — Figure 4 : Effect of excessive end-to-end fine-tuning with different dataset sizes (10h to 100h) on FAB’s performance on ASR downstream task (in WER $\downarrow$ ) for benign ( $X$ ) or trigger-stamped ( $\hat{X}$ ) inputs. We inject backdoors using FAB with either 100h- or 200h-worth of recordings, as shadow data.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

		Content			Paralinguistics	Semantics		Speaker
Model	Input	ASR $\downarrow$	KS $\uparrow$	PR $\downarrow$	ER $\uparrow$	IC $\uparrow$	ST $\uparrow$	ASV $\downarrow$	SD $\downarrow$	SID $\uparrow$
$f_{\theta}$	$X$	11.4	95.7	5.6	62.0	98.1	15.9	5.8	6.3	81.9
$f_{\theta}$	$\hat{X}$	14.4	93.3	8.2	61.2	95.5	14.2	6.1	6.8	79.1
$\hat{f_{\theta}}$	$X$	11.4	94.3	5.4	61.3	98.2	15.9	5.5	6.6	76.9
$\hat{f_{\theta}}$	$\hat{X}$	98.4	28.0	99.8	34.7	6.6	0.9	31.3	25.3	0.7

Appendix

Table 6 : Downstream task’s performance after fine-tuning with benign (

f_{\theta}

) and backdoored (

\hat{f_{\theta}}

) AFMs, when providing benign (

X

) or trigger-stamped (

\hat{X}

) samples as input.

Panel of Table 7 — Table 7 : Measuring the downstream performance on benign ( $X$ ) and trigger-stamped ( $\hat{X}$ ) after fine-tuning with AFMs backdoored with different layers’ representations selected to inject the backdoor (i.e., when minimizing $\mathcal{L}_{\mathit{Back}}$ ). Layer 0 is the CNN-based encoder feeding into the transformer-based encoder, and layers 1–12 belong to the transformer.

Panel of Table 8 — Table 8 : Comparison of the effect of injecting different trigger on downstream task’s performance after fine-tuning with benign ( $f_{\theta}$ ) and backdoored ( $\hat{f_{\theta}}$ ) AFMs, on both benign ( $X$ ) or trigger-stamped ( $\hat{X}$ ) samples. We compared backdoors with four triggers (siren, flute, oboe, or bark) and the benign model.

		Content			Paralinguistics	Semantics		Speaker
Model	Input	ASR $\downarrow$	KS $\uparrow$	PR $\downarrow$	ER $\uparrow$	IC $\uparrow$	ST $\uparrow$	ASV $\downarrow$	SD $\downarrow$	SID $\uparrow$
$f_{\theta}$	$X$	10.4	97.0	4.8	62.5	98.6	16.3	4.5	4.9	84.1
$f_{\theta}$	$\hat{X}$	10.9	95.8	5.1	61.5	98.0	15.4	4.8	6.1	82.3
$\hat{f_{\theta}}$	$X$	11.4	93.9	4.6	59.7	96.9	14.5	5.0	4.9	70.2
$\hat{f_{\theta}}$	$\hat{X}$	98.7	25.0	99.8	45.4	4.7	0.9	20.4	18.3	3.1

Appendix

Table 9 : Downstream task’s performance after fine-tuning with benign (

f_{\theta}

) and backdoored (

\hat{f_{\theta}}

) WavLM-based AFMs, when providing benign (

X

) or trigger-stamped (

\hat{X}

) samples as input.

Table 10 : The effect of the FAB’s trigger’s SNR during trigger injection and backdoor activation on downstream task performance, when providing benign (

X

) or trigger-stamped (

\hat{X}

) samples as input.

Panel of Figure 5 — Figure 5 : SUPERB scores for benign ( $f_{\theta}$ ) and backdoored ( $\hat{f_{\theta}}$ ) WavLM model, on normal inputs $X$ (top) and trigger-stamped inputs $\hat{X}$ (bottom), using the siren trigger, for different tasks. * indicates values $<$ -1 or $>$ 1.

Panel of Figure 6 — Figure 6 : SUPERB scores for benign model ( $f_{\theta}$ ) and backdoored model ( $\hat{f_{\theta}}$ ) on benign inputs $X$ (top) and trigger-stamped inputs $\hat{X}$ (bottom), using different trigger types during injection and backdoor activation. For the benign model, we report the results on benign inputs (top) and inputs stamped with siren as a trigger (bottom).* indicates values $>$ 1 or $<$ -1.

Panel of Figure 7 — Figure 7 : SUPERB scores on normal inputs $X$ (top) and trigger-stamped inputs $\hat{X}$ (bottom), for backdoored models with and without knowledge of the code-book. * indicates values $>$ 1 or $<$ -1.

Table 11 : Ablations on the ASR downstream task, in WER

\downarrow

. (a) varying the weight

\kappa

that balances

\mathcal{L}_{\mathit{Back}}

against

\mathcal{L}_{\mathit{Benign}}

(larger

\kappa

emphasizes increasing error on trigger-stamped inputs). (b) fine-tuning models backdoored with varying amounts of

\mathbb{X}_{\mathit{aux}}

. (c) backdooring towards different target representations

v

: the all-1s vector, or vectors drawn at random from normal distributions.

Panel of Figure 8 — Figure 8 : Left: FABs’ success, evaluated on ASR (in WER $\downarrow$ ), when altering the weight, $\kappa$ (x-axis), used to balance $\mathcal{L}_{\mathit{Back}}$ and $\mathcal{L}_{\mathit{Benign}}$ (larger $\kappa$ puts more emphasis on increasing error on trigger-stamped inputs, $\hat{X}$ ). Right: ASR’s performance (in WER $\downarrow$ ) when fine-tuning on model’s backdoored with varying amounts of samples in $\mathbb{X}_{\mathit{aux}}$ (x-axis).

Panel of Figure 9 — Figure 9 : The SUPERB scores for benign model ( $f_{\theta}$ ) and backdoored model ( $\hat{f_{\theta}}$ ) on normal inputs $X$ (a) and on trigger-stamped inputs $\hat{X}$ (b)–(d), for different trigger SNRs. * indicates that the value is higher than 1 or lower than $-1$ .

Explore similar work

Natural Backdoor Attacks on Speech Recognition Models
Jul 17, 2026Jinwen Xin, Xixiang Lyu, Jing MaAttack-Success Rate Deep Learning
School of Cyber Engineering, Xidian University, Xi’an, China
Ouroboros: Self-Referential Backdoor Attacks on Speech Enhancement via Clean Audio Triggers
Aug 31, 2026Yunjie Zhou, Yuheng Huang, Diqun YanSpeech Enhancement Inference-Time
Faculty of Electrical Engineering and Computer Science, Ningbo University, China · College of Artificial Intelligence, Ningbo University of Finance and Economics, China
SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models
Jul 17, 2026Jinwen Xin, Xixiang LvLLM Defense Mechanisms Utterances
School of Cyber Engineering Xidian University Xi’an, China

Jul 17, 2026, cs.CRJ/K move · Enter open · S save

Natural Backdoor Attacks on Speech Recognition Models

Jinwen Xin, Xixiang Lyu, Jing Ma

School of Cyber Engineering, Xidian University, Xi’an, China

Attack-Success Rate Deep Learning

Open paper AskPDF arXiv

With the rapid development of deep learning, its vulnerability has gradually emerged in recent years. This work focuses on backdoor attacks on speech recognition systems. We adopt sounds that are ordinary in nature or in our daily life as triggers for natural backdoor attacks. We conduct experiments on two datasets and three models to validate the performance of natural backdoor attacks and explore the effects of poisoning rate, trigger duration and blend ratio on the performance of natural backdoor attacks. Our results show that natural backdoor attacks have a high attack success rate without compromising model performance on benign samples, even with short or low-amplitude triggers. It requires only 5% of poisoned samples to achieve a near 100% attack success rate. In addition, the backdoor will be automatically activated by the corresponding sound in nature, which is not easy to be detected and will bring severer harm.

Abstract

Explore similar work

Natural Backdoor Attacks on Speech Recognition Models

Ouroboros: Self-Referential Backdoor Attacks on Speech Enhancement via Clean Audio Triggers

SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models