cs.SDNov 3, 2026

VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

Authors: Xudong Tang, Matthew Groh

Organizations: Department of Computer Science, Northwestern University · Management and Organizations, Kellogg School of Management, Northwestern University

Abstract

Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models

    Jan 31, 2026Ruinan Jin, Xinting Liao, Hanlin Yu +2Adversarial RobustnessAudio Generation

  2. Voice "Cloning" is Style Transfer

    May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech GenerationTrust in AI

  3. Your Voice Cloning System is Secretly a Voice Anonymizer

    Aug 27, 2026Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1Speaker AnonymizationPrivacy-Preserving Speech Processing