Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

Kemal Altwlkany; Muamer Paric

cs.SDOct 19, 2026

Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

Authors: Kemal Altwlkany, Muamer Paric

Organizations: Infobip · Faculty of Science, University of Sarajevo

Abstract

Answering machine detection (AMD) is a fundamental task in telephony that classifies whether an answered call is picked up by a human or a voicemail (machine). Despite its widespread use across cloud communication platforms, existing approaches rely on proprietary systems with undisclosed architectures, or require computationally expensive models that cannot operate in real time. We propose a lightweight AMD system that combines pretrained audio embeddings with gradient boosting machines, trained to prioritize the correct detection of human-answered calls. We evaluate seven models across two datasets: three models that we propose, a proprietary industry model, two prior academic approaches, and a large language model baseline that relies on automatic speech recognition. Furthermore, we release OpenAMD, the first publicly available AMD dataset comprising of more than 500 recordings in 7 languages. Our best model achieves the highest precision across both datasets while maintaining low inference latency. We assess the robustness of models to additive noise and packet loss, and perform tests of statistical significance to discover per-language and per-gender model performance. Finally, we evaluate our model on an out-of-domain task of detecting robocalls, where it achieves 94.34% recall without any additional training, which is comparable to the industry standard.

Figures & tables

Figure 1: High-level overview of our proposed method. We compute the log-mel spectrogram from the input audio waveform and extract 512-dimensional clip-wise embeddings using PANNs CNN6 [ 20 ] . We train a GBM model on the binary classification task of AMD by using the CNN6 embeddings as inputs.

Model	Accuracy	Precision	Recall	Specificity	F1
RNN-YAMNet [ 2 ]	$83.35\pm 2.25$	$87.72\pm 3.04$	$74.40\pm 4.04$	$91.04\pm 2.43$	$80.49\pm 2.56$
AMD-Infobip [ 16 ]	$88.09\pm 1.86$	$92.09\pm 2.59$	$81.20\pm 3.55$	$94.01\pm 1.96$	$86.29\pm 2.44$
CNN6-LogReg (our)	$85.61\pm 2.38$	$91.55\pm 2.65$	$75.87\pm 4.43$	$93.99\pm 1.89$	$82.96\pm 2.73$
CNN6-GBM-balanced (our)	$85.91\pm 2.46$	$92.33\pm 2.95$	$75.83\pm 3.83$	$94.59\pm 2.27$	$83.25\pm 3.03$
CNN6-GBM (our)	$85.20\pm 2.10$	$\mathbf{94.70}\pm 2.37$	$72.02\pm 4.08$	$\mathbf{96.54}\pm 1.49$	$81.80\pm 2.81$
wav2vec-vm-fine [ 8 ]	$84.72\pm 2.21$	$87.77\pm 2.99$	$77.79\pm 3.87$	$90.68\pm 2.13$	$82.46\pm 2.83$

Table 1: Performance of AMD models on proprietary dataset.

Figure 2: Critical difference diagram showing the average ranks of the models across accuracy .

Figure 3: Critical difference diagram showing the average ranks of the models across specificity .

Model	Accuracy	Precision	Recall	Specificity	F1
RNN-YAMNet [ 2 ]	94.56 $\pm$ 2.06	91.69 $\pm$ 3.38	99.33 $\pm$ 1.06	88.41 $\pm$ 4.36	95.36 $\pm$ 1.88
AMD-Infobip [ 16 ]	91.74 $\pm$ 2.44	88.55 $\pm$ 3.64	98.00 $\pm$ 1.82	83.69 $\pm$ 4.98	93.04 $\pm$ 2.19
CNN6-LogReg (our)	88.74 $\pm$ 2.81	87.97 $\pm$ 3.73	92.67 $\pm$ 3.14	83.69 $\pm$ 4.72	90.26 $\pm$ 2.70
CNN6-GBM-balanced (our)	91.93 $\pm$ 2.44	92.41 $\pm$ 3.12	93.33 $\pm$ 2.94	90.13 $\pm$ 4.04	92.87 $\pm$ 2.26
CNN6-GBM (our)	92.50 $\pm$ 2.44	95.14 $\pm$ 2.64	91.33 $\pm$ 3.51	93.99 $\pm$ 3.12	93.20 $\pm$ 2.38
wav2vec-vm-finetune [ 8 ]	84.43 $\pm$ 3.00	95.02 $\pm$ 2.97	76.33 $\pm$ 4.72	94.85 $\pm$ 2.99	84.66 $\pm$ 3.23

Table 2: Performance of AMD models on OpenAMD dataset.

Figure 4: Accuracy of AMD models in noisy and poor network quality conditions. Left: audio is distorted by noise (x-axis indicates SNR). Right: audio is corrupted by packet loss. Shaded areas and black bars indicate the 95% CI.

Figure 5: Machine recall and human specificity, per gender and per model. Black bars indicate the 95% CI.

Model	Mean	Median	Std
CNN6-GBM	1.23	1.21	0.092
CNN6-GBM-balanced	1.36	1.33	0.358
CNN6-LogReg	1.19	1.18	0.062
wav2vec-vm-finetune	131.99	132.85	14.179

Table 3: Inference latency of AMD models (in milliseconds).

Parameter	Search space	Selected
Learning rate	{0.03, 0.1, 0.3}	0.1
Number of estimators	{50, 100, 200, 300, 400, 500}	200
Num leaves	{17, 31, 41, 51, 61, 127}	61
Min child samples	{5, 10, 20, 30, 50, 100, 200}	50

Table 4: CNN6-GBM hyperparameter search space and selected values.

Audio Classification

Audio classification assigns recordings to categories such as speech, music, animal calls, acoustic scenes, and health-related events.

See all 147 papers

Apr 2, 2026

Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

Outbound AI calling systems must distinguish voicemail greetings from live human answers in real time to avoid wasted agent interactions and dropped calls. We present a lightweight approach that extracts 15 temporal features from the speech activity pattern of a pre-trained neural voice activity detector (VAD), then classifies with a shallow tree-based ensemble. Across two evaluation sets totaling 764 telephony recordings, the system achieves a combined 96.1% accuracy (734/764), with 99.3% (139/140) on an expert-labeled test set and 95.4% (595/624) on a held-out production set. In production validation over 77,000 calls, it maintained a 0.3% false positive rate and 1.3% false negative rate. End-to-end inference completes in 46 ms on a commodity dual-core CPU with no GPU, supporting 380+ concurrent WebSocket calls. In our search over 3,780 model, feature, and threshold combinations, feature importance was concentrated in three temporal variables. Adding transcription keywords or beep-based features did not improve the best real-time configuration and increased latency substantially. Our results suggest that temporal speech patterns are a strong signal for distinguishing voicemail greetings from live human answers.

Kumar Saurav

CTO, ClearGrid

Speech Processing Audio Classification

Ask PDF

Jun 23, 2026

Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

Scam phone calls exploit vulnerable communities worldwide, yet research on detection has focused almost exclusively on English and other high-resource languages. In low-resource settings such as Turkish, detection is especially difficult, as annotated data is scarce and technological defenses remain limited. This research investigates how large language models (LLMs) can support scam detection in Turkish by introducing the first public multi-modal dataset of 100 aligned audio-transcript pairs of scam and benign conversations. We evaluate seven LLMs spanning three model families: Gemini 2.5 (Flash, Flash-Lite, Pro), GPT-4o, and Qwen (Max, Plus, Turbo), under three input conditions: raw audio, automatic speech-to-text transcripts, and transcripts refined by a native speaker. Our results suggest that transcript-based inputs consistently outperform direct audio processing, while human-corrected and uncorrected transcripts perform comparably. By centering a low-resource language and real world threat, this work highlights the urgent need for culturally and linguistically inclusive AI safety research and more robust multi-modal systems for fraud prevention.

Arda Eren, Micheal Cheung, Youqian Zhang +2

The Hong Kong Polytechnic University, Hong Kong · The Education University of Hong Kong, Hong Kong

Audio-Language Model Evaluation Financial Fraud Detection Low-Resource Language Processing

Ask PDF

Oct 14, 2026

Audio System One Model: Typed Probabilistic Perception of Emotion in Speech

Speech applications ask many questions about the same voice: which emotion a listener perceives, how aroused the speaker sounds, whether they sound frustrated. Classifiers answer fast, over one fixed label set; audio language models (ALMs) take any question in text, but answer by generation, one pass per question, with poorly calibrated probabilities. We present ASOM (Audio System One Model), a System One model for speech: choice, score and yes/no questions written at request time are answered in one pass, without generation, each with a probability distribution. Frozen speech and text encoders feed a 6.2M-parameter reader with no per-class parameters, trained on listener vote distributions. On MSP-Podcast, ASOM matches a classifier on the same features and data, leads the best of 6 zero-shot ALMs by 0.072 and an ALM fine-tuned on the same data by 0.026, and answers 20 questions in 13.1 ms, 17-41 times faster than the ALMs. It also answers reworded, reduced and new label sets, scales and gender, and on 21 unseen corpora in 13 languages it leads the best zero-shot ALM by 0.138 where the words carry no emotion; where they do, the ALMs mostly read.

Jaime Bellver-Soler, Iván Martín-Fernández, Anmol Guragain +1

Speech Technology and Machine Learning Group (THAU), Information Processing and Telecommunications Center, E.T.S.I. de Telecomunicación, Universidad Politécnica de Madrid, Spain

Speech Emotion Recognition Speech Foundation Models Audio-Language Model Evaluation

Ask PDF

Abstract

Audio Classification

Explore similar work

Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

Audio System One Model: Typed Probabilistic Perception of Emotion in Speech