Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks

Matías Pizarro; Raghavan Narasimhan; Jonas Killian; Asja Fischer

cs.LGMar 23, 2026

Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks

Authors: Matías Pizarro, Raghavan Narasimhan, Jonas Killian, Asja Fischer

Organizations: Faculty of Computer Science, Ruhr University Bochum, Bochum, Germany

Abstract

With the increasing deployment of automated and agentic systems, ensuring the adversarial robustness of automatic speech recognition (ASR) models has become highly relevant. We observe that changing the precision of an ASR model during inference reduces the likelihood of adversarial attacks to succeed. We take advantage of this fact to make models more robust simply by randomly sampling the precision during prediction. Moreover, this insight can be turned into an adversarial example detection strategy by implementing a simple Gaussian classifier that thresholds the differences between outputs of models run with different precision. To further enhance security boundaries, we combine the approach with an existing uncertainty-based defense mechanism, which forces adaptive adversaries to introduce highly perceptible noise to bypass detection. An experimental analysis across various ASR models, languages, and attack types demonstrates a significant increase in adversarial robustness, competitive detection capabilities, and resistance to adaptive threats.

Figures & tables

ASR Model -	FP32		FP16		BF16		Random sampling precision
Training precision	Test-clean	Test-other	Test-clean	Test-other	Test-clean	Test-other	Test-clean	Test-other
CTC-FP32	02.04 / 25.50	04.02 / 40.42	02.04 / 25.46	04.03 / 40.46	02.05 / 25.57	04.03 / 40.46	02.04 / 25.51	04.02 / 40.44
CTC-FP16	01.98 / 25.15	04.00 / 40.18	01.98 / 25.19	03.99 / 40.08	02.00 / 25.15	04.01 / 40.22	01.98 / 25.16	04.00 / 40.16
CTC-BF16	01.91 / 24.69	03.93 / 40.08	01.91 / 24.73	03.93 / 40.08	01.92 / 24.73	03.95 / 40.15	01.91 / 24.71	03.93 / 40.10
seq2seq-FP32	02.80 / 31.49	08.33 / 56.75	02.80 / 31.56	08.30 / 56.99	03.49 / 38.44	09.28 / 61.89	03.03/ 33.83	08.63 / 58.54
seq2seq-FP16	02.87 / 31.49	08.64 / 57.98	02.92 / 31.95	08.72 / 57.98	05.75 / 59.39	11.97 / 74.04	03.84 / 40.94	09.77 / 63.33

TABLE I: Cross-precision evaluation of ASR systems on benign LibriSpeech data. The first column indicates the training precision of each model, while the remaining columns report performance under different inference precisions. Results are reported as WER / SER on the test-clean and test-other subsets, where lower values indicate better recognition performance. †

ASR Model	FP32	FP16	BF16	Random sampling
CTC-FP32	09.19 / 43.50	09.18 / 43.47	09.18 / 43.42	09.18 / 43.46
CTC-FP16	08.93 / 42.74	08.93 / 42.71	08.95 / 42.79	08.93 / 42.74
CTC-BF16	09.13 / 43.48	09.13 / 43.48	09.12 / 43.40	09.13 / 43.45
seq2seq-FP32	12.95 / 52.03	12.94 / 51.96	13.15 / 52.43	13.01 / 52.14
seq2seq-FP16	13.01 / 52.85	13.21 / 52.87	13.86 / 53.75	13.36 / 53.15
seq2seq-BF16	13.11 / 52.24	13.10 / 52.20	12.93 / 52.81	13.04 / 52.41

TABLE II: Cross-precision evaluation on the Mozilla Common Voice German test set. Columns show inference configurations across different precisions. Results given as WER / SER, where lower values indicate better recognition performance. †

ASR Model -	C&W attack					Psychoacoustic attack
Training precision	FP32	FP16	BF16	Random $\uparrow$	$\text{SNR}_{seg}$	FP32	FP16	BF16	Random $\uparrow$	$\text{SNR}_{seg}$
CTC-FP32	00.00 / 00.00	06.38 / 28.00	25.72 / 83.00	10.93 / 38.90	18.54	00.00 / 00.00	08.02 / 36.00	25.31 / 84.00	10.74 / 39.30	19.94
CTC-FP16	03.29 / 15.00	00.00 / 00.00	25.93 / 83.00	10.29 / 34.50	17.94	09.47 / 39.00	00.00 / 00.00	30.25 / 89.00	13.81 / 44.00	19.37
CTC-BF16	19.55 / 70.00	18.93 / 67.00	00.00 / 00.00	12.59 / 44.80	17.53	20.78 / 81.00	21.40 / 83.00	00.00 / 00.00	13.87 / 53.30	18.79
seq2seq-FP32	00.00 / 00.00	22.02 / 48.00	34.16 / 77.00	18.50 / 41.20	13.34	00.00 / 00.00	27.16 / 58.00	35.19 / 79.00	21.52 / 47.80	13.85
seq2seq-FP16	56.38 / 95.00	00.41 / 01.00	62.35 / 97.00	40.58 / 66.20	15.89	57.41 / 98.00	00.21 / 01.00	60.70 / 93.00	40.21 / 65.90	16.33

TABLE III: Adversarial robustness evaluation of ASR systems under PVP. Columns show inference configurations across different precisions. For each attack, the final column reports distortion (

\text{SNR}_{\text{seg}}

\mathrm{dB}

); all other entries show WER / SER. Results are evaluated on 100 AEs generated per attack type from the LibriSpeech test-clean set.

ASR Model	FP32	FP16	BF16	Random $\uparrow$	$\text{SNR}_{seg}$
CTC-FP32	00.00 / 00.00	09.74 / 41.00	28.57 / 89.00	12.77 / 43.33	18.96
CTC-FP16	08.66 / 35.00	00.00 / 00.00	25.76 / 87.00	11.47 / 40.66	19.74
CTC-BF16	18.61 / 75.00	19.26 / 77.00	00.00 / 00.00	12.62 / 50.67	20.04
seq2seq-FP32	02.81 / 03.00	23.81 / 44.00	39.83 / 75.00	22.15 / 40.66	10.96
seq2seq-FP16	44.37 / 80.00	00.65 / 01.00	48.27 / 82.00	31.09 / 54.30	11.31
seq2seq-BF16	50.65 / 91.00	50.22 / 91.00	01.73 / 02.00	34.20 / 61.33	10.51

TABLE IV: Adversarial robustness evaluation of ASR systems under PVP. Columns show inference configurations across different precisions. The final column reports distortion, while other entries show WER / SER. Results are evaluated on 100 C&W AEs generated from the Mozilla Common Voice test set. Bold marks the highest WER / SER under random sampling. †

ASR Model -	C&W vs.	Psy. vs.	Both (C&W and Psy.) vs. benign
Training precision	benign	benign	PVP	NF	TD	DistriBlock
CTC-FP32	0.92	0.93	0.92	0.89	0.94	0.99
CTC-FP16	0.91	0.95	0.95	0.89	0.93	0.99
CTC-BF16	0.86	0.91	0.91	0.87	0.94	0.99
seq2seq-FP32	0.90	0.93	0.91	0.70	0.83	0.96
seq2seq-FP16	0.97	0.98	0.97	0.73	0.81	0.97

TABLE V: Adversarial detection performance (AUROC) across different defense methods. Benign samples from the LibriSpeech test-clean set are evaluated against their AE counterparts (C&W and Psy). Bold marks the highest detection score. †

ASR Model	PVP	DistriBlock	$\text{Hybrid}_{\text{JS}}$	$\text{Hybrid}_{\text{MP}}$
CTC-FP32	0.92	0.95	0.99	0.95
CTC-FP16	0.94	0.96	0.98	0.95
CTC-BF16	0.86	0.96	0.98	0.96

TABLE VI: Adversarial detection performance (AUROC) across different defense methods.

100

Benign samples from the Common Voice German test set are evaluated against their C&W AE counterparts. Bold marks the highest detection score.

ASR	PVP	DistriBlock	$\text{Hybrid}_{\text{JS}}$	$\text{Hybrid}_{\text{MP}}$
Model	$\text{SNR}_{\text{seg}}$ /AUROC	$\text{SNR}_{\text{seg}}$ /AUROC	$\text{SNR}_{\text{seg}}$ /AUROC	$\text{SNR}_{\text{seg}}$ /AUROC
CTC-FP32	18.00 / 0.48	14.50 / 0.69	08.79 / 0.63	-00.54 / 0.47
CTC-FP16	17.39 / 0.48	15.34 / 0.68	10.09 / 0.63	03.57 / 0.50
CTC-BF16	18.79 / 0.47	15.01 / 0.70	08.37 / 0.64	01.40 / 0.53
† Bold marks the highest distortion among the generated AEs.

TABLE VII: Adaptive attack detection distortion (

\text{SNR}_{\text{seg}}

\mathrm{dB}

) and performance (AUROC) across defense methods. Evaluated using

100

benign samples from the Common Voice German test set against their adaptive AE counterparts. †

Adversarial Robustness

Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.

See all 463 papers

What Was That Again? Certified Robustness for Automatic Speech Recognition
Jun 26, 2026Andrew C. Cullen, Neil G. Marchant, Jiani Xie +2ASR Evaluation Neural Network Robustness
University of Melbourne · DST Group, Adelaide
Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition
Jun 4, 2026Yifan Liao, Zongmin Zhang, Zhen Sun +3Adversarial Attacks Transfer-Based Adversarial Attacks
The Hong Kong University of Science and Technology (Guangzhou) · Wuhan University
Breaking Adversarial Transferability in Fine-Tuned Speech Recognition
Oct 6, 2026Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi +3Adversarial Transferability Automatic Speech Recognition
Department of Computer Engineering Sharif University of Technology · Department of Computer Engineering University of Isfahan

Jun 26, 2026J/K move · Enter open · S save

What Was That Again? Certified Robustness for Automatic Speech Recognition

Andrew C. Cullen, Neil G. Marchant, Jiani Xie, Paul Montague+1

University of Melbourne · DST Group, Adelaide

ASR Evaluation Neural Network Robustness Automatic Speech Recognition Certified Robustness Adversarial Example Detection Robust ASR

Open paper AskPDF arXiv

Automatic Speech Recognition systems are notoriously both sensitive to adversarial and benign perturbations. While this has been repeatedly demonstrated using reference datasets, detecting such behaviors in deployed systems is incredibly challenging, due to the absence of oracle knowledge of the true transcription. We demonstrate that employing a certification-inspired mechanism can significantly decrease WER, increase recall, and decrease the Spearman correlation between confidence and WER. We achieve this through a dual-gate diagnostic pipeline: a Two-Sided Atomic Audit that accumulates statistical wealth to certify both token existence and adversarial exclusion, and a Rank-Based Tournament that selects the winning sequence. Our evaluations across four diverse architectures demonstrate up to a 55% relative reduction in Word Error Rate, while also providing granular word- and sentence-level certifications to enhance acoustic security.

Abstract

Adversarial Robustness

Explore similar work

What Was That Again? Certified Robustness for Automatic Speech Recognition

Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition

Breaking Adversarial Transferability in Fine-Tuned Speech Recognition