The Last Mile of Deepfake Speech Detection: An Industry-Academia Experience Report
Organizations: Phonexia, Brno, Czech Republic
Abstract
Synthetic speech detection benchmarks now report sub-1% error rates on some in-domain evaluations, yet performance degrades under unseen attacks, channel mismatch, and distribution shift. Based on a three-year effort with Phonexia, a commercial speaker-recognition vendor, we report barriers encountered while building and deploying a detector. Many public benchmarks are not licensed for commercial model development. Real inputs are not four-second clean clips but long, codec-degraded, sometimes partially synthetic recordings. And when a calibrated system returns a log-likelihood ratio of 2.5, no one can tell the customer what it means for their decision. Rather than proposing a new model, we connect these barriers to concrete research and coordination proposals: shared standards for commercially usable datasets, realistic deployment benchmarks, and scores that non-experts can act on. These observations come from one project and should be tested in other settings.
Figures & tables
| System | Detection accuracy (seen attacks) | Detection accuracy (unseen attacks) | Balanced accuracy (seen attacks) | Balanced accuracy (unseen attacks) |
|---|---|---|---|---|
| [Company] v2 | 0.9135 | 0.9381 | 0.9383 | 0.9509 |
| [Company] v3 | 0.9723 | 0.9760 | 0.9721 | 0.9745 |
| Experience (Sec.) | Unresolved community-level question | Actions |
|---|---|---|
| Data sourcing & provenance ( 3.1 ) | Can a detector’s training data be verified and held to clean-sourcing standards when no one can audit a deployed model? | CR2, CO2 |
| Data composition & drift ( 3.1 ) | Which data does a detector actually need, how diverse must bona fide speech be, and how do we keep pace with drift? | CR1, CO1 |
| Generalization ( 3.3 ) | How do we know an evaluation tests genuinely unseen attacks and conditions, rather than familiar ones? | CM1 |
| Metrics & comparability ( 3.3 ) | What metrics and reporting make results meaningful at a deployment operating point, and comparable across conditions? | CM2, CO1 |
| Score communication ( 3.4 ) | How should a calibrated score become a decision, which a non-expert can act on and defend? | CM3 |
| Deployment & defense ( 3.5 , 3.6 ) | How should the field prepare for unseen attacks and define “fit for purpose” for consequential uses when customers lack representative data and deploy systems independently? | CO3, CO4 |