VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale
Authors: Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Gijs Wijngaard, Kourosh Nadi, Huu Nguyen, +2 more
Organizations: LAION e.V. · Scalable Learning & Multi-Purpose AI (SLAMPAI) Lab, Forschungszentrum Jülich GmbH · L3S Research Center · Black Forest Labs · Computer Science Department, TU Darmstadt · German Research Center for Artificial Intelligence (DFKI) · Hessian Center for AI (hessian.AI) · Leibniz Universität Hannover
Expressive speech synthesis has outpaced expressive speech perception: systems now render fine-grained vocal performances that no public benchmark can score. Most benchmarks for this inverse problem stop at six to nine basic emotion categories, largely on acted speech. This paper introduces VoiceNet, a human-annotated representation-level benchmark for voice performance understanding on permissively-licensed in-the-wild speech. VoiceNet has two subsets: VoiceNet-Emo applies a 40-emotion taxonomy with three expert ratings per item, and VoiceNet-Ext, a preliminary subset, scores 57 talking-style attributes including speaking rate, vocal tension, breathiness, and register. The paper also releases Emolia, an emotion-annotated version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter VoiceCLAP-Small for fast large-scale data filtering and a 7B VoiceCLAP-Large for state-of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on VoiceNet-Emo. On VoiceNet-Emo, VoiceCLAP-Large aligns more closely with the aggregate expert consensus than individual experts agree with one another: a comparison against the majority label rather than evidence of surpassing human emotion perception. All systems evaluated here are voice-text embedding models: VoiceNet scores representation-level attribute recognition and retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering uncurated speech corpora into subsets that span diverse talking styles and emotions remains an open challenge; VoiceCLAP embeddings offer a promising tool for this task. VoiceNet, Emolia, and VoiceCLAP are publicly available for research use.
Figures & tables
Model
VoiceNet-Emo bal@pp
VoiceNet-Ext bal@pp
VoiceNet-Emo ρ
VoiceNet-Ext ρ
Cacophony ( Zhu et al., 2024 )
0.5432
0.6089
− 0.0183
0.0188
LAION-CLAP ( Wu et al., 2023 )
0.5624
0.6184
0.0310
0.0600
CLSP ( Yang et al., 2026b )
0.5675
0.6221
0.0610
0.0937
M2D-CLAP-2025 ( Niizumi et al., 2025 )
0.5681
0.6361
0.0348
0.1003
MS-CLAP-2023 ( Elizalde et al., 2024 )
0.5754
0.6321
− 0.0084
0.1044
GLAP ( Dinkel et al., 2025 )
0.5765
0.6254
0.0667
0.0836
Table 1: Headline results on VoiceNet , grouped by model family and sorted within each group by VoiceNet-Emo per-prompt balanced accuracy ( bal@pp ). ρ is the mean per-prompt Spearman correlation between cosine similarity and human present-vote share. All numbers are computed on items with at least two human raters (7,986 of 7,988 VoiceNet-Emo items and 16,166 of 18,532 VoiceNet-Ext non- LANG items) so the labels reflect a multi-rater majority rather than a single annotator’s call; per-tag files in App. A.7 report both this filtered cut and the full-set numbers. Bootstrap CIs are in Tab. 6 . Human baselines and consistency references on the same task are discussed in the text. Best per column in bold; higher is better.
Model
EmoNet-Voice
IEMOCAP
RAVDESS
CREMA-D
LAION-CLAP ( Wu et al., 2023 )
0.028
0.201
0.173
0.366
MS-CLAP-2023 ( Elizalde et al., 2024 )
0.025
0.217
0.259
0.355
CLSP ( Yang et al., 2026b )
0.036
0.231
0.194
0.333
GLAP ( Dinkel et al., 2025 )
0.025
0.147
0.115
0.198
MGA-CLAP ( Li et al., 2024 )
0.029
0.183
0.105
0.349
Cacophony ( Zhu et al., 2024 )
0.024
0.181
0.181
0.342
Table 2: Zero-shot emotion classification accuracy on four public speech-emotion benchmarks. EmoNet-Voice is the synthetic 40-class v1 release ( Schuhmann et al., 2025a ) . VoiceCLAP-Large substantially outperforms other models on IEMOCAP and CREMA-D and is the runner-up on EmoNet-Voice and RAVDESS. Best per column in bold, runner-up underlined; higher is better.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Open Licence
Size (#Utts/Hours)
#Labels
#Spk.
Real / In-wild
Multilin.
IEMOCAP ( Busso et al., 2008 )
✗
10k / ∼ 12h
≤ 9
10 (5M/5F)
R / ✗
✗
RAVDESS ( Livingstone & Russo, 2018 )
✓
1.4k / ∼ 1h
≤ 8
24 (12M/12F)
R / ✗
✗
SAVEE ( Jackson & Haq, 2014 )
✗
480 / < 1h
≤ 7
4 (Male)
R / ✗
✗
EmoDB ( Burkhardt et al., 2005 )
✗
535 / < 1h
≤ 7
10 (5M/5F)
R / ✗
✗
CREMA-D ( Cao et al., 2014 )
✓
7.4k / ∼ 6h
≤ 6
91 (48M/43F)
R / ✗
✗
SERAB ( Scheidwasser-Clow et al., 2022 )
✗
9 corpora / var.
≤ 6
var.
R / ✗
✓
Appendix
Table 3: Comparison of speech-emotion benchmarks . #Labels count the size of the label set: emotion or affect categories for prior benchmarks and VoiceNet-Emo , talking-style attributes for VoiceNet-Ext . Real = recorded human speech; In-the-wild = unscripted, naturalistic. Open license means CC-BY or CC-BY-NC 4.0 or equivalent; var. means varies across pooled corpora.
Source dataset
Clips
Annotation calls (18 × )
Multilingual In The Wild
721k
12.98M
Majestrino
973k
17.51M
LAION’s Got Talent
1.69M
30.34M
Emolia-Balanced (balanced subset of Emolia )
5.26M
94.62M
Total
8.64M
155.46M
Appendix
Table 4: The four MOSS-Audio -annotated voice corpora. Each clip is queried with 18 prompt-groups; the per-corpus annotation-call counts follow.
Corpus
Clips
Caption key
LAION’s Got Talent
1,685,809
detailed_caption
Emolia-Balanced
5,256,683
emotion_caption
Majestrino
972,658
caption
Synthetic Vocal Bursts
325,548
text (in-house, YouTube)
Improved Synthetic Vocal Bursts
15,680
text (in-house, YouTube)
EARS
17,227
text (FCaps)
Appendix
Table 5: VoiceCLAP Data training corpus manifest (Multilingual In The Wild is annotated and released as part of the Emolia suite but excluded from training after ablation). Caption-key indicates which JSON field is used as the contrastive text.
Model
VoiceNet-Emo bal@pp
VoiceNet-Ext bal@pp
VoiceNet-Emo ρ
VoiceNet-Ext ρ
LAION-CLAP
0.5624 [0.5511, 0.5727]
0.6184 [0.6104, 0.6268]
0.0310 [0.0071, 0.0537]
0.0600 [0.0421, 0.0753]
MGA-CLAP
0.5897 [0.5789, 0.5998]
0.6208 [0.6128, 0.6287]
0.0947 [0.0717, 0.1186]
0.0819 [0.0653, 0.0962]
GLAP
0.5765 [0.5661, 0.5877]
0.6254 [0.6171, 0.6341]
0.0667 [0.0425, 0.0907]
0.0836 [0.0655, 0.0992]
M2D-CLAP-2025
0.5681 [0.5571, 0.5787]
0.6361 [0.6273, 0.6444]
0.0348 [0.0121, 0.0582]
0.1003 [0.0822, 0.1142]
CLSP
0.5675 [0.5563, 0.5791]
0.6221 [0.6140, 0.6303]
0.0610 [0.0380, 0.0840]
0.0937 [0.0763, 0.1072]
MS-CLAP-2023
0.5754 [0.5646, 0.5858]
0.6321 [0.6243, 0.6403]
− 0.0084 [ − 0.0303, 0.0153]
0.1044 [0.0867, 0.1186]
Appendix
Table 6: Paired clip-bootstrap 95% CIs for Tab. 1 (2,000 replicates). Entries are point estimate [lo,hi] . Per-prompt thresholds are held fixed; intervals therefore do not correct their same-set fitting. VoiceNet-Ext remains preliminary because this resampling also does not propagate annotation uncertainty.
emotion
VoiceCLAP Small
VoiceCLAP Large
LCO Omni-3B
LCO Omni-7B
LAION CLAP
MS-CLAP 2023
CLSP
GLAP
M2D CLAP
MGA CLAP
Cacophony
Anger
0.555
0.577
0.455
0.524
0.310
0.348
0.486
0.389
0.228
0.448
0.054
Impatience and Irritability
0.638
0.581
0.524
0.544
0.212
− 0.086
0.149
0.375
− 0.132
0.489
0.092
Contempt
0.398
0.441
0.433
0.497
0.230
− 0.008
0.149
0.254
0.186
0.394
0.007
Amusement
0.518
0.500
0.509
0.530
0.283
− 0.281
− 0.351
0.530
0.062
0.526
0.154
Distress
0.627
0.626
0.538
0.575
− 0.030
− 0.255
0.226
0.226
0.325
0.382
− 0.318
Contemplation
0.449
0.464
0.437
0.423
0.022
0.312
0.161
− 0.022
0.138
0.161
0.185
Appendix
Table 7: Per-emotion Spearman ρ on VoiceNet-Emo for VoiceCLAP , the Omni-Embedding bases, and all seven general-audio CLAP baselines. Rows are sorted by the mean ρ across the displayed model columns; the mean is used only for ordering and is not shown. Best per row is bold, runner-up underlined. Cells are coloured by ρ .
Code
Description
n
maj-yes
polar. match
Perceived speaker (2)
GEND
gender presentation
350
0.237
0.594
AGEV
age range
349
0.284
0.564
Affective dimensions (7)
VALN
valence
343
0.315
0.624
AROU
arousal
343
0.280
0.618
Appendix
Table 8: Per-attribute coverage and majority-vote statistics on VoiceNet-Ext . n is the number of (audio, attribute-level) pairs released for that attribute, aggregated across the seven rubric levels; per-(attribute, level) prompt counts are therefore roughly n/7 . maj-yes is the fraction of items where the rater majority voted “present”; polar. match is the fraction of items where the majority vote agrees with the prompt’s intended affirmative-vs.-contrastive polarity. Per-attribute Spearman ρ for each model is computed over the same n items per attribute and is released alongside the paper as a data artifact. Attribute codes follow the schema in App. A.4 ; rows are ordered to match that grouping.
Figure 1: Expert agreement on emotion presence for VoiceNet-Emo , per emotion, on the 7,984 pairs with three ratings. Stacked bars show the proportion of (audio, emotion) pairs by the number of experts who marked the emotion present (weakly or strongly): three ( 3:0 unanimous present ), two ( 2:1 ), one ( 1:2 ), or none ( 3:0 unanimous absent ). The text to the right of each bar gives the number of pairs n and the share of unanimous pairs. Rows are sorted by unanimity.
Figure 2: Scaling view of VoiceNet performance. Each point is one model. The y-axis is the unweighted mean of VoiceNet-Emo and VoiceNet-Ext per-prompt balanced accuracy from Table 1 . The red step line tracks the best score reached by any model up to each parameter count. VoiceCLAP-Small (110M) sits above the general-audio CLAP cluster and is competitive with the larger Omni bases, while the 7B VoiceCLAP-Large reaches the highest average score. Parameter counts are approximate (audio + text encoder, before LoRA adapters). Averages match Tab. 1 .
Figure 3: Instructions given to the human annotator for the expert annotation of VoiceNet-Emo .
Figure 4: UI of our expert annotation tool for VoiceNet-Emo .
Figure 5: UI of the second annotation interface, used for the emotion-presence binary confirmation task on VoiceNet-Emo . Annotators listen to a clip and confirm or reject whether the target emotion is present.
Figure 6: UI of the second annotation interface, used for the talking-style attribute binary confirmation task on VoiceNet-Ext . For each (dimension, level) bucket, annotators confirm or reject whether the audio matches the target rubric-level description.
Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio. The benchmark contains 273 human-recorded English WAV clips from 51 speaker accounts totaling 1.95 hours across seven labels: angry, disgusted, fearful, happy, neutral, sad, and surprised, with 39 clips per class. All baselines are evaluated from audio alone, without transcripts or contextual metadata. Across six released baselines, average accuracy is 35.5%. The strongest baseline, gemini_3_5_flash, reaches 46.5% on the seven-way task, above the 14.3% random baseline but far from robust emotion recognition. A secondary valence-bucket analysis maps labels into positive, neutral, and negative classes, excluding surprised because its valence is ambiguous. Aggregate accuracy under this coarser view is 50.9%. Performance is highly uneven across classes. By recall, neutral is identified most reliably at 75.6% averaged across baselines, while surprised and fearful reach only 10.7% and 15.4%, respectively. These results show that the evaluated baselines can extract some affective signal from speech, but discrete expressed-emotion recognition remains fragile, especially for non-neutral emotions that are often most important in voice agent workflows.
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.
Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA