VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning
Organizations: Department of Computer Science, Northwestern University · Management and Organizations, Kellogg School of Management, Northwestern University
Abstract
Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.
Figures & tables
| Benchmark | Domain | # Speakers | # Stimuli | Human judgments | # Participants | AI-gen | Morphs | Target |
|---|---|---|---|---|---|---|---|---|
| VoxCeleb 1+2 | Voice | 7,363 | 1.28M | Speaker ID | ||||
| VoxBlink2 | Voice | 111,284 | 9.9M | Speaker ID | ||||
| ASVspoof 2021 | Voice | – | 1.4M | Real/synth | ||||
| WaveFake | Voice | 2 | 105K | Real/synth | ||||
| FakeAVCeleb | Voice+Vid | 500 | 20K | Real/synth | ||||
| DeepSpeak | Voice+Vid | 500 | 13,025 | Real/synth |
| Type | Comparison recording | What the type measures | Pairs | Judgments | Mean | Reliability |
|---|---|---|---|---|---|---|
| 1 | Same recording, different segment | Ceiling for “same” responses | 100 | 3,676 | 0.91 | 0.59 |
| 2 | Same speaker, different recording | Variation between recordings of a speaker | 400 | 15,988 | 0.69 | 0.87 |
| 3 | Same speaker, AI voice clone | Whether a clone is heard as its source speaker | 400 | 4,024 | 0.59 | 0.58 |
| 4 | Different speaker, real recording | Confusion between speakers of the same cell | 400 | 15,675 | 0.26 | 0.85 |
| 5 | Different speaker, AI voice clone | Confusion with the clone of another speaker | 400 | 4,027 | 0.27 | 0.52 |
| 6 | Morph of two speakers | Graded targets between two speakers | 8,100 | 81,486 | 0.38 | 0.66 |
| Model | Type | Training objective | Training data | Dim |
|---|---|---|---|---|
| x-vector [ 25 ] | Supervised | Softmax | VoxCeleb 1 and 2 | 512 |
| ECAPA-TDNN [ 26 ] | Supervised | AAM-Softmax | VoxCeleb 1 and 2 | 192 |
| RawNet3 [ 27 ] | Supervised | AAM-Softmax | VoxCeleb 1 and 2 | 192 |
| TitaNet [ 28 ] | Supervised | AAM-Softmax | VoxCeleb 1 and 2, Fisher, Switchboard, LibriSpeech, NIST SRE | 192 |
| resemblyzer [ 29 ] | Supervised | GE2E | LibriSpeech, VoxCeleb 1 and 2 | 256 |
| WeSpeaker [ 30 ] | Supervised | AAM-Softmax | VoxCeleb 2 | 256 |
| Task 1 | Task 2 | Task 3 | |||||
| Model | AUC | ECE (raw) | ECE (cal) | ||||
| Supervised | |||||||
| resemblyzer | 0.656 | 0.430 | 0.609 | 0.872 | 0.338 | 0.027 | 0.542 |
| ECAPA-TDNN | 0.646 | 0.417 | 0.591 | 0.862 | 0.047 | 0.018 | 0.339 |
| TitaNet | 0.638 | 0.407 | 0.577 | 0.861 | 0.057 | 0.019 | 0.379 |
| WeSpeaker | 0.627 | 0.393 | 0.558 | 0.851 | 0.054 | 0.012 | 0.389 |
| Model | Slope ratio | |||
| Supervised | ||||
| resemblyzer | 0.297 [0.242, 0.345] | 0.379 [0.346, 0.411] | 1.29 [1.21, 1.38] | 0.001 |
| ECAPA-TDNN | 0.282 [0.223, 0.332] | 0.362 [0.328, 0.395] | 1.13 [1.06, 1.19] | 0.001 |
| WeSpeaker | 0.268 [0.211, 0.317] | 0.335 [0.300, 0.371] | 1.15 [1.07, 1.22] | 0.001 |
| TitaNet | 0.266 [0.208, 0.317] | 0.351 [0.316, 0.385] | 1.16 [1.09, 1.24] | 0.001 |
| RawNet3 | 0.184 [0.114, 0.243] | 0.330 [0.293, 0.365] | 1.22 [1.15, 1.30] | 0.001 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Input | File | Used for |
|---|---|---|
| Speakers with sociophonetic group, gender, and age bracket | data/speakers.csv | Fairness analysis, CV folds, Task 3 |
| Pairs with stimulus type, metadata label, and vote counts | data/stimuli.csv | All four tasks, analyses by stimulus type |
| Exact morph weights and transcripts | data/stimuli_interpol.csv | Analyses along the morph weight |
| Individual judgments, familiarity flag, and questionnaire answers | data/participant_responses.csv | , noise ceiling, unfamiliar trials, human baseline |
| Embeddings of the eleven models | data/embeddings/{model}.npz | All four tasks |
| Embeddings at every layer for the SSL models and Whisper | data/embeddings/layers/{model}.npz | Layer selection, layer analysis |
| Model | Main set | Extension subset |
|---|---|---|
| Supervised | ||
| resemblyzer | 0.656 | 0.51 [0.37, 0.66] |
| ECAPA-TDNN | 0.646 | 0.50 [0.06, 0.79] |
| TitaNet | 0.638 | 0.49 [0.02, 0.84] |
| WeSpeaker | 0.627 | 0.49 [ , 0.86] |
| RawNet3 | 0.617 | 0.56 [0.17, 0.80] |
| Question | Answers |
|---|---|
| Gender | 837 female, 398 male, 23 “prefer not to say” or “intersex” |
| Age | 25 under 18, 451 aged 18 to 24, 407 aged 25 to 34, 192 aged 35 to 44, 101 aged 45 to 54, 57 aged 55 to 64, 6 aged 65 or older, 19 “prefer not to say” |
| First language | 899 English, 323 another language, 36 “prefer not to say” |
| Hearing | 17 poor, 104 fair, 352 good, 379 very good, 406 excellent |
| Task 1 Pearson | Task 2 AUC | ||||
|---|---|---|---|---|---|
| Model | All | Unfamiliar | Familiar | All | Unfamiliar |
| resemblyzer | 0.656 | 0.631 | 0.560 | 0.872 | 0.850 |
| ECAPA-TDNN | 0.646 | 0.613 | 0.576 | 0.862 | 0.840 |
| TitaNet | 0.638 | 0.606 | 0.572 | 0.861 | 0.837 |
| WeSpeaker | 0.627 | 0.594 | 0.567 | 0.851 | 0.828 |
| RawNet3 | 0.617 | 0.585 | 0.555 | 0.851 | 0.827 |
| Judgments | SD of | Split-half | Responses | Pairs with majority | Tied | |||
|---|---|---|---|---|---|---|---|---|
| Type | Pairs | per pair | correlation | Reliability | against label | against label | pairs | |
| 1 | 100 | 36.8 | 0.07 | 0.42 | 0.59 | 9.5% | 0.0% | 0.0% |
| 2 | 400 | 40.0 | 0.19 | 0.76 | 0.87 | 30.9% | 17.3% | 1.8% |
| 3 | 400 | 10.1 | 0.23 | 0.41 | 0.58 | 40.7% | 26.0% | 15.8% |
| 4 | 400 | 39.2 | 0.17 | 0.74 | 0.85 | 25.8% | 11.5% | 0.5% |
| 5 | 400 | 10.1 | 0.19 | 0.35 | 0.52 | 27.4% | 10.5% | 8.5% |
| Model | Directional | Symmetrized |
|---|---|---|
| Supervised | ||
| resemblyzer | 0.542 | 0.615 |
| x-vector | 0.479 | 0.505 |
| WeSpeaker | 0.389 | 0.439 |
| RawNet3 | 0.386 | 0.421 |
| TitaNet | 0.379 | 0.433 |
| Model | ||
|---|---|---|
| Supervised | ||
| resemblyzer | 0.082 [0.056, 0.116] | 0.56 [0.43, 0.69] |
| ECAPA-TDNN | 0.080 [0.050, 0.118] | 0.09 [0.05, 0.13] |
| WeSpeaker | 0.068 [0.040, 0.104] | 0.10 [0.05, 0.15] |
| TitaNet | 0.085 [0.055, 0.124] | 0.11 [0.07, 0.15] |
| RawNet3 | 0.146 [0.104, 0.195] | 0.15 [0.10, 0.20] |
| Model | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| wav2vec 2.0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| HuBERT | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 |
| WavLM | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| XLS-R | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Whisper | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| Model | Source | Checkpoint identifier |
|---|---|---|
| x-vector | SpeechBrain | speechbrain/spkrec-xvect-voxceleb |
| ECAPA-TDNN | SpeechBrain | speechbrain/spkrec-ecapa-voxceleb |
| RawNet3 | ESPnet | espnet/voxcelebs12_rawnet3 |
| TitaNet | NVIDIA NeMo | nvidia/speakerverification_en_titanet_large |
| resemblyzer | resemblyzer (pip) | default voice encoder (GE2E pretrained) |
| WeSpeaker | pyannote.audio 3.4 | pyannote/wespeaker-voxceleb-resnet34-LM |
| EER | AUC | |||||
| Model | Set A | Set B | Set C | Set A | Set B | Set C |
| Supervised | ||||||
| WeSpeaker | 1.5% | 3.5% | 3.6% | 0.998 | 0.994 | 0.995 |
| TitaNet | 1.6% | 3.0% | 2.7% | 0.999 | 0.995 | 0.997 |
| ECAPA-TDNN | 1.6% | 4.6% | 3.1% | 0.998 | 0.992 | 0.995 |
| RawNet3 | 1.9% | 4.0% | 3.5% | 0.997 | 0.992 | 0.995 |
| Cutoff from the CV | Cutoff from all pairs | |||
| Model | Accuracy | Listeners below | Accuracy | Listeners below |
| Supervised | ||||
| resemblyzer | 0.790 | 72.5% | 0.797 | 74.3% |
| ECAPA-TDNN | 0.784 | 70.1% | 0.788 | 71.7% |
| TitaNet | 0.782 | 69.4% | 0.782 | 69.6% |
| RawNet3 | 0.778 | 68.9% | 0.782 | 69.6% |
| Model | 95% interval of | Mean rank | 95% interval of the rank | |
|---|---|---|---|---|
| Supervised | ||||
| resemblyzer | 0.656 | [0.632, 0.679] | 1.26 | 1 to 3 |
| ECAPA-TDNN | 0.646 | [0.619, 0.672] | 1.87 | 1 to 3 |
| TitaNet | 0.638 | [0.611, 0.664] | 2.91 | 2 to 4 |
| WeSpeaker | 0.627 | [0.600, 0.653] | 4.06 | 3 to 5 |
| RawNet3 | 0.617 | [0.586, 0.645] | 4.96 | 4 to 6 |
| Model | All pairs | Real recordings | Without morphs | Equal weight | 10 judgments |
|---|---|---|---|---|---|
| Supervised | |||||
| resemblyzer | 0.656 | 0.859 | 0.804 | 0.506 | 0.650 |
| ECAPA-TDNN | 0.646 | 0.855 | 0.791 | 0.442 | 0.640 |
| TitaNet | 0.638 | 0.858 | 0.794 | 0.444 | 0.633 |
| WeSpeaker | 0.627 | 0.847 | 0.784 | 0.430 | 0.622 |
| RawNet3 | 0.617 | 0.843 | 0.774 | 0.381 | 0.612 |
| Same speaker | Different speaker | |||||
| Model | One recording | Two recordings | Clone | Real | Clone | Morph |
| Supervised | ||||||
| resemblyzer | 0.286 | 0.650 | 0.513 | 0.520 | 0.463 | 0.607 |
| ECAPA-TDNN | 0.348 | 0.581 | 0.453 | 0.365 | 0.306 | 0.598 |
| TitaNet | 0.246 | 0.612 | 0.498 | 0.409 | 0.316 | 0.585 |
| WeSpeaker | 0.341 | 0.550 | 0.407 | 0.375 | 0.332 | 0.572 |
| Model | All pairs | Outside VoxCeleb | With a VoxCeleb speaker |
|---|---|---|---|
| Supervised | |||
| resemblyzer | 0.656 | 0.664 | 0.642 |
| ECAPA-TDNN | 0.646 | 0.672 | 0.608 |
| TitaNet | 0.638 | 0.671 | 0.589 |
| WeSpeaker | 0.627 | 0.650 | 0.592 |
| RawNet3 | 0.617 | 0.635 | 0.592 |
| Model | |||
| Supervised | |||
| resemblyzer | 0.435 | 0.403 | |
| ECAPA-TDNN | 0.357 | 0.343 | |
| TitaNet | 0.354 | 0.353 | |
| RawNet3 | 0.341 | 0.336 | |
| WeSpeaker | 0.314 | 0.304 | |
| Model | Cosine | Quadratic | Pair features |
|---|---|---|---|
| Supervised | |||
| resemblyzer | 0.656 | 0.619 | 0.680 |
| ECAPA-TDNN | 0.646 | 0.559 | 0.624 |
| TitaNet | 0.638 | 0.574 | 0.601 |
| WeSpeaker | 0.627 | 0.570 | 0.614 |
| RawNet3 | 0.617 | 0.575 | 0.605 |
Speaker Verification
Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.
See all 68 papers