FEND: Far-End Detection in Outbound Voice AI Calls
Organizations: Observe.AI, India
Abstract
When a conversational Voice AI agent places an outbound call, the far end may be a live person, a voicemail greeting, an interactive voice response (IVR) system, or increasingly another AI system answering on the callee's behalf. Standard Answering Machine Detection (AMD) reports only whether a human or a machine picked up, a design meant for human call centres, where one machine is as uninteresting as another. That single bit does not tell a Voice AI agent what to do, since leaving a voicemail message, navigating an IVR and holding a short exchange with another AI are three different actions and a two-valued label cannot distinguish them. In 1,000 randomly sampled calls from our own outbound deployment, a live person answered 48.1%, a voicemail system 44.6%, an AI 6.9% and an IVR 0.4%. Our proposed framework, FEND, reads the far end's audio and its streaming transcript and runs one detector for each automated class (voicemail, IVR, AI callee), with human the default when nothing fires, returning a single label within 20 seconds. On a public benchmark the voicemail components reach 100% accuracy without ever misclassifying a live human, where Asterisk's AMD misclassifies 30 to 66% of them. For the AI callee, acoustic features do not transfer across recording conditions, but speech timing does, separating AI from human at 0.952 ROC-AUC, because synthesised speech leaves far shorter gaps between words. On production traffic, separating a live person from any automated far end reaches F1 0.979, and only 2.9% of live people are given an automated label, the error that matters most for an agent acting on the result.
Figures & tables
| Fire rate (%) | Binary | |||||
| Component | VM | AM | Hum | IVR | Acc | Rec |
| tuning half ( scored, ivr held out) | ||||||
| Tone (beep) | 70.0 | 100.0 | 0.0 | 0.0 | 0.900 | 0.842 |
| Keyphrase | 50.0 | 25.9 | 0.0 | 0.0 | 0.611 | 0.386 |
| LLM (voicemail) | 96.7 | 100.0 | 0.0 | 0.0 | 0.989 | 0.982 |
| Fusion | 96.7 | 100.0 | 0.0 | 0.0 | 0.989 | 0.982 |
| Cadence feature | AI | Human | AUC |
|---|---|---|---|
| Articulation rate (w s -1 ) | 3.68 | 2.51 | 0.844 |
| Inter-word gap, CV | 3.50 | 1.76 | 0.834 |
| Inter-word gap, p90 (s) | 0.000 | 0.138 | 0.687 |
| Word duration, CV | 0.373 | 0.447 | 0.706 |
| fused, held out ( , , 200 splits) | |||
| ROC-AUC | [0.902, 0.990] | ||
| Class | n | Prec | Rec | F1 | FP into |
|---|---|---|---|---|---|
| human | 241 | 0.987 | 0.971 | 0.979 | IVR 1, AI 1 |
| voicemail/AM | 223 | 0.912 | 0.982 | 0.946 | AI 18, H 3 |
| ivr | 2 | 0.333 | 0.500 | 0.400 | n too small |
| AI callee | 35 | 0.762 | 0.457 | 0.571 | VM 2, H 3 |
| human vs rest | 241 | 0.987 | 0.971 | 0.979 | 7 humans |
| Detector | Med. | p90 | Cost |
|---|---|---|---|
| Cadence (AI callee) | s | s | negligible |
| Keyphrase | s | s | negligible |
| Tone (beep) | s | s | negligible |
| LLM (voicemail) | s | s | tok |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| human | vm | am | ivr | |
|---|---|---|---|---|
| Clips | 114 | 102 | 92 | 92 |
| Parameter | Default | Role |
| STFT front-end | ||
| sampling rate | 8 kHz | telephony stream rate |
| window / hop | 64 / 32 ms | freq. resolution |
| FFT size | 512 | spectral detail |
| Detection thresholds (tuned on golden set) | ||
| dominant-peak margin | 20 dB | vs. selectivity |
| null calibration, train only ( ) | |||
|---|---|---|---|
| Metric | within p99 | cross | adm. |
| Formant-band conc. | 0.25 | 1.24 | no |
| Voicing continuity | 0.25 | 2.79 | no |
| Level dynamics | 0.21 | 1.11 | no |
| Background flatness | 0.28 | 3.95 | no |
| Segment structure | 0.23 | 0.26 | yes |
| Priority order | H F1 | H FP | VM F1 | AI F1 | Acc |
|---|---|---|---|---|---|
| vm ivr ai | 0.979 | 7 | 0.946 | 0.571 | 0.938 |
| vm ai ivr | 0.979 | 7 | 0.946 | 0.571 | 0.938 |
| ivr vm ai | 0.979 | 7 | 0.944 | 0.571 | 0.936 |
| ai vm ivr | 0.979 | 7 | 0.737 | 0.405 | 0.798 |
| ai ivr vm | 0.979 | 7 | 0.733 | 0.405 | 0.796 |
| ivr ai vm | 0.979 | 7 | 0.733 | 0.405 | 0.796 |
| Order | Acc | Prec H | Rec H | FP H | |
|---|---|---|---|---|---|
| VM IVR AI | 5 s | 94.3 | 0.848 | 1.000 | 0 |
| VM IVR AI | 15 s | 98.2 | 0.963 | 1.000 | 0 |
| VM IVR AI | 30 s | 98.2 | 0.963 | 1.000 | 0 |
| IVR VM AI | 30 s | 98.6 | 0.963 | 1.000 | 0 |
| first-firing (no priority) | 30 s | 98.9 | 0.963 | 1.000 | 0 |
Classification
Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.
See all 449 papers