Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties

Tiantian Feng; Jihwan Lee; Anfeng Xu; Yoonjeong Lee; Thanathai Lertpetchpun; Xuan Shi; Helin Wang; Thomas Thebaud

cs.SDMay 20, 2025

Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties

Authors: Tiantian Feng, Jihwan Lee, Anfeng Xu, Yoonjeong Lee, Thanathai Lertpetchpun, Xuan Shi, Helin Wang, Thomas Thebaud, +4 more

Abstract

We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles spanning both static speaker traits (e.g., age, sex, accent, voice quality) and dynamic speech traits (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: https://github.com/tiantiaf0627/vox-profile-release.

Explore similar work

ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions
Date pendingThomas Thebaud, Junhyeok Lee, Laureano Moro-Velazquez +2Speech Synthesis Speaker
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · AMIAD, Palaiseau, France · Department of Signal Theory and Communications, Universidad Carlos III de Madrid (UC3M), Leganes, Spain
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
Jul 16, 2026David Ayllon, Alice Baird, Jeffrey Brooks +11Voice Agents Legal Natural Language Processing Benchmarks
Hume AI Research
ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood
May 28, 2026Tiantian Feng, Anfeng Xu, Xuan Shi +10Vocalizations Children
University of Southern California · The Ohio State University · University of California, Los Angeles +3

Date pending, eess.ASJ/K move · Enter open · S save

ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions

Thomas Thebaud, Junhyeok Lee, Laureano Moro-Velazquez, Jesus Villalba Lopez+1

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · AMIAD, Palaiseau, France · Department of Signal Theory and Communications, Universidad Carlos III de Madrid (UC3M), Leganes, Spain

Speech Synthesis Speaker Profile Synthetic Data Distributions

Open paper AskPDF arXiv

Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.

Abstract

Explore similar work

ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood