VoiceSHIELD-Small: Real-Time Malicious Speech Detection and Transcription

Sumit Ranjan; Sugandha Sharma; Ubaid Abbas; Puneeth N Ail

cs.SDMar 8, 2026

VoiceSHIELD-Small: Real-Time Malicious Speech Detection and Transcription

Authors: Sumit Ranjan, Sugandha Sharma, Ubaid Abbas, Puneeth N Ail

Organizations: Emvo

Abstract

Voice interfaces are quickly becoming a common way for people to interact with AI systems. This also brings new security risks, such as prompt injection, social engineering, and harmful voice commands. Traditional security methods rely on converting speech to text and then filtering that text, which introduces delays and can ignore important audio cues. This paper introduces VoiceSHIELD-Small, a lightweight model that works in real time. It can transcribe speech and detect whether it is safe or harmful, all in one step. Built on OpenAI's Whisper-small encoder, VoiceSHIELD adds a mean-pooling layer and a simple classification head. It takes just 90-120 milliseconds to classify audio on mid-tier GPUs, while transcription happens at the same time. Tested on a balanced set of 947 audio clips, the model achieved 99.16 percent accuracy and an F1 score of 0.9865. At the default setting, it missed 2.33 percent of harmful inputs. Cross-validation showed consistent performance (F1 standard deviation = 0.0026). The paper also covers the model's design, training data, performance trade-offs, and responsible use guidelines. VoiceSHIELD is released under the MIT license to encourage further research and adoption in voice AI security.

Figures & tables

Component	Parameters	Trainable?
Whisper Encoder	74M	Frozen
Whisper Encoder	14M	Frozen
Mean Pool + MLP	0.13M	Trainable
Total	88.6M

TABLE I: Model Parameter Summary

Split	Samples	Safe	Malicious
Train	4,416	3,016	1,400
Validation	947	647	300
Test	947	647	300
Total	6,310	4,310	2,000

TABLE II: Dataset Summary

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )
Learning rate	3e-5 (cosine decay to 0)
Batch size (effective)	32 (4 per GPU $\times$ 8 grad accum)
Max steps	3,000
Warmup steps	200
Weight decay	0.01

TABLE III: Training Hyperparameters

Metric	Mean	Std	Min	Max
F1	0.9879	$\pm$ 0.0026	0.9838	0.9912
Precision	0.9906	$\pm$ 0.0034	0.9854	0.9940
Recall	0.9853	$\pm$ 0.0062	0.9794	0.9941
ROC-AUC	0.9989	$\pm$ 0.0009	0.9976	0.9998

TABLE IV: Evaluation Metrics

	Predicted Safe	Predicted Malicious
Actual Safe	646	1
Actual Malicious	7	293

TABLE V: Confusion Matrix

Metric	Score	Interpretation
Accuracy	99.16%	Overall correct classifications
F1 Score	0.9865	Harmonic mean of precision and recall
Precision	0.9966	When model says "malicious", it is correct 99.66% of the time
Recall (Sensitivity)	0.9767	Model catches 97.67% of actual malicious clips
ROC-AUC	0.9948	Excellent separability between safe and malicious clips
False Negative Rate	2.33%	Proportion of malicious clips missed

TABLE VI: Metrics at Default Threshold 0.2

Hardware	Classification Latency	Full Pipeline (including transcript)
NVIDIA RTX 4090	45–60 ms	180–220 ms
NVIDIA RTX A4000 (mid-tier)	90–120 ms	280–350 ms
Apple M2 Pro (CPU)	250–300 ms	600–700 ms

TABLE VII: Inference Latency on Different Hardware

$\tau$	F1	Precision	Recall	FNR	FPR
0.10	0.9865	0.9886	0.9845	1.55%	0.45%
0.20	0.9882	0.9966	0.9800	2.00%	0.15%
0.35	0.9882	0.9966	0.9800	2.00%	0.15%
0.50	0.9865	0.9966	0.9767	2.33%	0.15%
0.70	0.9714	1.0000	0.9444	5.56%	0.00%

TABLE VIII: Evaluation Metrics at Different Thresholds

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 322 papers

AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models
Apr 10, 2026Mintong Kang, Chen Fang, Bo LiLLM Safety Benchmarks
University of Illinois Urbana-Champaign
SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models
Jul 17, 2026Jinwen Xin, Xixiang LvBackdoor Detection Automatic Speech Recognition
School of Cyber Engineering Xidian University Xi’an, China
Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription
Oct 9, 2026Ayush Kumar Bar, Promila GhoshSound Event Detection Audio Classification

Apr 10, 2026J/K move · Enter open · S save

AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models

Mintong Kang, Chen Fang, Bo Li

University of Illinois Urbana-Champaign

LLM Safety Benchmarks

Open paper AskPDF arXiv

Audio has rapidly become a primary interface for foundation models, powering real-time voice assistants. Ensuring safety in audio systems is inherently more complex than just "unsafe text spoken aloud": real-world risks can hinge on audio-native harmful sound events, speaker attributes (e.g., child voice), impersonation/voice-cloning misuse, and voice-content compositional harms, such as child voice plus sexual content. The nature of audio makes it challenging to develop comprehensive benchmarks or guardrails against this unique risk landscape. To close this gap, we conduct large-scale red teaming on audio systems, systematically uncover vulnerabilities in audio, and develop a comprehensive, policy-grounded audio risk taxonomy and AudioSafetyBench, the first policy-based audio safety benchmark across diverse threat models. AudioSafetyBench supports diverse languages, suspicious voices (e.g., celebrity/impersonation and child voice), risky voice-content combinations, and non-speech sound events. To defend against these threats, we propose AudioGuard, a unified guardrail consisting of 1) SoundGuard for waveform-level audio-native detection and 2) ContentGuard for policy-grounded semantic protection. Extensive experiments on AudioSafetyBench and four complementary benchmarks show that AudioGuard consistently improves guardrail accuracy over strong audio-LLM-based baselines with substantially lower latency.

Abstract

LLM Security

Explore similar work

AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models

SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models

Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription