Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

Kumar Saurav

cs.SDApr 2, 2026

Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

Authors: Kumar Saurav

Organizations: CTO, ClearGrid

Abstract

Outbound AI calling systems must distinguish voicemail greetings from live human answers in real time to avoid wasted agent interactions and dropped calls. We present a lightweight approach that extracts 15 temporal features from the speech activity pattern of a pre-trained neural voice activity detector (VAD), then classifies with a shallow tree-based ensemble. Across two evaluation sets totaling 764 telephony recordings, the system achieves a combined 96.1% accuracy (734/764), with 99.3% (139/140) on an expert-labeled test set and 95.4% (595/624) on a held-out production set. In production validation over 77,000 calls, it maintained a 0.3% false positive rate and 1.3% false negative rate. End-to-end inference completes in 46 ms on a commodity dual-core CPU with no GPU, supporting 380+ concurrent WebSocket calls. In our search over 3,780 model, feature, and threshold combinations, feature importance was concentrated in three temporal variables. Adding transcription keywords or beep-based features did not improve the best real-time configuration and increased latency substantially. Our results suggest that temporal speech patterns are a strong signal for distinguishing voicemail greetings from live human answers.

Figures & tables

#	Feature	Description
1	speech_ratio	Total speech duration divided by $W$
2	num_segments	Number of distinct speech segments $N$
3	mean_seg_ms	Mean segment duration: $\frac{1}{N}\sum_{i}(e_{i}-s_{i})$
4	longest_seg_ms	Maximum segment duration: $\max_{i}(e_{i}-s_{i})$
5	first_onset_ms	Time to first speech: $s_{1}$
6	first_seg_ms	Duration of first segment: $e_{1}-s_{1}$

Table 1: The 15 temporal speech activity features. All features are computed from VAD speech timestamps within the detection window

W

Dataset	Size	VM	NVM	Labeling Method
Training	24,812	$\sim$ 14,887	$\sim$ 9,925	Pseudo-labeled (cross-model agreement)
Eval-A (Test-140)	140	70	70	Hand-labeled by domain expert
Eval-B (Held-out 624)	624	434	190	41% expert-verified, 59% MLM-validated

Table 2: Dataset summary. All audio is 8 kHz stereo telephony.

Model	Feat.	$W$	$\tau$	Eval-A	Eval-B	Comb.	Lat.
Clf. A (boost, $d{=}2$ , $n{=}50$ )	VAD-15	5s	0.50	139/140	595/624	96.1%	46ms
Clf. B (bagged, $n{=}50$ )	VAD-15	3s	0.30	136/140	596/624	95.8%	29ms
Cross-channel (stereo)	Eng.-8	5s	—	137/140	—	—	$<$ 10ms
OR Ensemble (A $\lor$ C)	Mixed	5s	var.	129/140	508/624	83.4%	92ms
Clf. A + Transcript	VAD+KW	5s	—	138/140	608/624	97.6%	$\sim$ 500ms

Table 3: Top model configurations. Combined = weighted accuracy across both evaluation sets (764 files). Latency excludes the

W

-second audio buffer.

Rank	Feature	Importance	Cumulative
1	speech_first_half_ratio	54.6%	54.6%
2	first_seg_ms	20.0%	74.6%
3	first_onset_ms	11.0%	85.6%
4	speech_ratio	7.6%	93.2%
5	mean_seg_ms	5.0%	98.2%
6–15	(remaining 10 features)	1.8%	100%

Table 4: Feature importance for Classifier A (5-second window). Top 3 features account for 85.6% of total importance.

Feature Set	Best Test-140	Finding
VAD-15 (temporal only)	139/140 (99.3%)	Best overall
Beep-8 (frequency)	71/140 (50.7%)	Near-random; beeps on wrong channel
Combined-23 (VAD + beep)	139/140 (99.3%)	No improvement over VAD-15
VAD + Transcript-25	138/140; 97.4% HF	+2% HF but 10 $\times$ slower
Engineered-46	127/140 (90.7%)	Overfitting with more features

Table 5: Ablation: best accuracy per feature set (5-second window, optimal model and threshold).

∗ Estimated from vendor documentation [ 1 ] ; not independently verified.
Method	Accuracy	Time-to-Decision	False Positives	GPU Required
Ours (temporal VAD features)	96.1%	5.05 s	0.3%	No
Beep detection (DESA-2)	50.7%	10–30 s	23.2%	No
Industry AMD (silence heuristics)	70–85% ∗	3–5 s	10–15% ∗	No
Transcription keywords	97.6%	$\sim$ 5.5 s	$<$ 1%	Yes

Table 6: Our temporal approach vs. industry-standard beep detection.

Window	Best Model	Test-140	Time-to-Decision
3 s	Classifier B	136/140 (97.1%)	3.03 s
5 s	Classifier A	139/140 (99.3%)	5.05 s
7 s	Classifier A	139/140 (99.3%)	7.05 s

Table 7: Best accuracy by detection window.

Strategy	Test-140	HF-624	Latency
OR (predict VM if either predicts VM)	129/140 (92.1%)	81.6%	92 ms
Weighted average ( $w_{A}{=}0.3,w_{C}{=}0.4$ )	127/140 (79.5%)	—	92 ms
AND cascade (both must agree)	127/140 (78.4%)	—	92 ms
Single best (Classifier A alone)	139/140 (99.3%)	95.4%	46 ms

Table 8: Ensemble strategy comparison (5-second window).

Table 9: Confusion matrices for Classifier A (

W{=}5

\tau{=}0.50

Window	Accuracy	Precision	Recall	F1-Score
500 ms	95.97%	99.28%	96.12%	97.68%
1,000 ms	96.83%	99.50%	96.90%	98.18%
2,000 ms	97.73%	99.33%	98.09%	98.71%

Table 10: Early detection performance (large-scale evaluation, 12,000 samples).

Component	Time
Resample 8 kHz $\rightarrow$ 16 kHz	$\sim$ 1 ms
Neural VAD inference	$\sim$ 44 ms
Temporal feature extraction	$\sim$ 0.2 ms
Tree classifier prediction	$\sim$ 0.2 ms
Total processing	$\sim$ 46 ms
Total time-to-decision	5.05 s

Table 11: Processing latency for a single call (Classifier A,

W{=}5

s).

Concurrent	Throughput	P50 Wall Time	Accuracy	Failures
10	1.7/s	5.10 s	88.5%	0
50	7.5/s	5.13 s	87.5%	0
100	13.1/s	5.29 s	88.5%	0
200	20.3/s	6.12 s	87.5%	0
380	25.1/s	9.46 s	88.6%	0
500	25.0/s	14.2 s	87.6%	1

Table 12: Load test results on a 2-core cloud instance (c2d-standard-2, $76/month).

Metric	Value
Total calls evaluated	77,000
Voicemails detected	32,000 (41.6%)
False positive rate	0.3% (live humans misclassified as VM)
False negative rate	1.3% (voicemails missed)

Table 13: Production validation on 77,000 calls.

Metric	Without Detection	With Detection	Savings
Avg. call duration (VM)	30 s	5.05 s	83%
AI agent compute per VM	Full inference	None (terminated)	$\sim$ 100%
Telephony cost per VM	$0.02–0.05	$0.003	85–94%
Wasted agent-minutes (77K calls)	$\sim$ 16,000 min	$\sim$ 2,700 min	83%

Table 14: Cost impact of voicemail detection at production scale.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Rank	Model	Features	$W$	$\tau$	Test-140	HF-624
1	Boost ( $n{=}50$ , $d{=}2$ )	VAD-15	5 s	0.50	139/140	95.4%
2	Boost ( $n{=}50$ , $d{=}2$ )	VAD-15	5 s	0.45	139/140	95.2%
3	Boost ( $n{=}50$ , $d{=}2$ )	VAD-15	5 s	0.55	139/140	95.0%
4	Boost ( $n{=}50$ , $d{=}2$ )	Comb-23	5 s	0.50	139/140	94.9%
5	AdaBoost ( $n{=}100$ )	VAD-15	5 s	0.45	137/140	93.8%
6	Cross-channel	Energy-8	5 s	—	137/140	—

Appendix

Table 15: Top 10 configurations from the 3,780-experiment grid search, ranked by Test-140 accuracy then HF-624 accuracy.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines
Oct 19, 2026Kemal Altwlkany, Muamer ParicAudio Classification Audio Understanding
Infobip · Faculty of Science, University of Sarajevo
Real-Time Voice AI Hears but Does Not Listen
Jun 24, 2026Martijn Bartelds, Federico Bianchi, James ZouAudio-Language Model Evaluation Voice Agent Evaluation
Together AI · Stanford University
Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision
Sep 17, 2026Guangzhao Yang, Muhammad Huzaifah, Yu Pan +2Data Augmentation Speech Processing
R&D Team, Recho Inc., Tokyo, Japan

Oct 19, 2026J/K move · Enter open · S save

Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

Kemal Altwlkany, Muamer Paric

Infobip · Faculty of Science, University of Sarajevo

Audio Classification Audio Understanding

Open paper AskPDF arXiv

Answering machine detection (AMD) is a fundamental task in telephony that classifies whether an answered call is picked up by a human or a voicemail (machine). Despite its widespread use across cloud communication platforms, existing approaches rely on proprietary systems with undisclosed architectures, or require computationally expensive models that cannot operate in real time. We propose a lightweight AMD system that combines pretrained audio embeddings with gradient boosting machines, trained to prioritize the correct detection of human-answered calls. We evaluate seven models across two datasets: three models that we propose, a proprietary industry model, two prior academic approaches, and a large language model baseline that relies on automatic speech recognition. Furthermore, we release OpenAMD, the first publicly available AMD dataset comprising of more than 500 recordings in 7 languages. Our best model achieves the highest precision across both datasets while maintaining low inference latency. We assess the robustness of models to additive noise and packet loss, and perform tests of statistical significance to discover per-language and per-gender model performance. Finally, we evaluate our model on an out-of-domain task of detecting robocalls, where it achieves 94.34% recall without any additional training, which is comparable to the industry standard.

Abstract

Speech Processing

Explore similar work

Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

Real-Time Voice AI Hears but Does Not Listen

Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision