VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale

Christoph Schuhmann; Robert Kaczmarczyk; Gollam Rabby; Felix Friedrich; Maurice Kraus; Gijs Wijngaard; Kourosh Nadi; Huu Nguyen

cs.SDSep 25, 2026

VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale

Authors: Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Gijs Wijngaard, Kourosh Nadi, Huu Nguyen, +2 more

Organizations: LAION e.V. · Scalable Learning & Multi-Purpose AI (SLAMPAI) Lab, Forschungszentrum Jülich GmbH · L3S Research Center · Black Forest Labs · Computer Science Department, TU Darmstadt · German Research Center for Artificial Intelligence (DFKI) · Hessian Center for AI (hessian.AI) · Leibniz Universität Hannover

Abstract

Expressive speech synthesis has outpaced expressive speech perception: systems now render fine-grained vocal performances that no public benchmark can score. Most benchmarks for this inverse problem stop at six to nine basic emotion categories, largely on acted speech. This paper introduces VoiceNet, a human-annotated representation-level benchmark for voice performance understanding on permissively-licensed in-the-wild speech. VoiceNet has two subsets: VoiceNet-Emo applies a 40-emotion taxonomy with three expert ratings per item, and VoiceNet-Ext, a preliminary subset, scores 57 talking-style attributes including speaking rate, vocal tension, breathiness, and register. The paper also releases Emolia, an emotion-annotated version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter VoiceCLAP-Small for fast large-scale data filtering and a 7B VoiceCLAP-Large for state-of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on VoiceNet-Emo. On VoiceNet-Emo, VoiceCLAP-Large aligns more closely with the aggregate expert consensus than individual experts agree with one another: a comparison against the majority label rather than evidence of surpassing human emotion perception. All systems evaluated here are voice-text embedding models: VoiceNet scores representation-level attribute recognition and retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering uncurated speech corpora into subsets that span diverse talking styles and emotions remains an open challenge; VoiceCLAP embeddings offer a promising tool for this task. VoiceNet, Emolia, and VoiceCLAP are publicly available for research use.

Figures & tables

Model	VoiceNet-Emo bal@pp	VoiceNet-Ext bal@pp	VoiceNet-Emo $\rho$	VoiceNet-Ext $\rho$
Cacophony ( Zhu et al., 2024 )	0.5432	0.6089	$-$ 0.0183	0.0188
LAION-CLAP ( Wu et al., 2023 )	0.5624	0.6184	0.0310	0.0600
CLSP ( Yang et al., 2026b )	0.5675	0.6221	0.0610	0.0937
M2D-CLAP-2025 ( Niizumi et al., 2025 )	0.5681	0.6361	0.0348	0.1003
MS-CLAP-2023 ( Elizalde et al., 2024 )	0.5754	0.6321	$-$ 0.0084	0.1044
GLAP ( Dinkel et al., 2025 )	0.5765	0.6254	0.0667	0.0836

Table 1: Headline results on VoiceNet , grouped by model family and sorted within each group by VoiceNet-Emo per-prompt balanced accuracy ( bal@pp ).

\rho

is the mean per-prompt Spearman correlation between cosine similarity and human present-vote share. All numbers are computed on items with at least two human raters (7,986 of 7,988 VoiceNet-Emo items and 16,166 of 18,532 VoiceNet-Ext non- LANG items) so the labels reflect a multi-rater majority rather than a single annotator’s call; per-tag files in App. A.7 report both this filtered cut and the full-set numbers. Bootstrap CIs are in Tab. 6 . Human baselines and consistency references on the same task are discussed in the text. Best per column in bold; higher is better.

Model	EmoNet-Voice	IEMOCAP	RAVDESS	CREMA-D
LAION-CLAP ( Wu et al., 2023 )	0.028	0.201	0.173	0.366
MS-CLAP-2023 ( Elizalde et al., 2024 )	0.025	0.217	0.259	0.355
CLSP ( Yang et al., 2026b )	0.036	0.231	0.194	0.333
GLAP ( Dinkel et al., 2025 )	0.025	0.147	0.115	0.198
MGA-CLAP ( Li et al., 2024 )	0.029	0.183	0.105	0.349
Cacophony ( Zhu et al., 2024 )	0.024	0.181	0.181	0.342

Table 2: Zero-shot emotion classification accuracy on four public speech-emotion benchmarks. EmoNet-Voice is the synthetic 40-class v1 release ( Schuhmann et al., 2025a ) . VoiceCLAP-Large substantially outperforms other models on IEMOCAP and CREMA-D and is the runner-up on EmoNet-Voice and RAVDESS. Best per column in bold, runner-up underlined; higher is better.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Open Licence	Size (#Utts/Hours)	#Labels	#Spk.	Real / In-wild	Multilin.
IEMOCAP ( Busso et al., 2008 )	✗	10k / $\sim$ 12h	$\leq$ 9	10 (5M/5F)	R / ✗	✗
RAVDESS ( Livingstone & Russo, 2018 )	✓	1.4k / $\sim$ 1h	$\leq$ 8	24 (12M/12F)	R / ✗	✗
SAVEE ( Jackson & Haq, 2014 )	✗	480 / $<$ 1h	$\leq$ 7	4 (Male)	R / ✗	✗
EmoDB ( Burkhardt et al., 2005 )	✗	535 / $<$ 1h	$\leq$ 7	10 (5M/5F)	R / ✗	✗
CREMA-D ( Cao et al., 2014 )	✓	7.4k / $\sim$ 6h	$\leq$ 6	91 (48M/43F)	R / ✗	✗
SERAB ( Scheidwasser-Clow et al., 2022 )	✗	9 corpora / var.	$\leq$ 6	var.	R / ✗	✓

Appendix

Table 3: Comparison of speech-emotion benchmarks . #Labels count the size of the label set: emotion or affect categories for prior benchmarks and VoiceNet-Emo , talking-style attributes for VoiceNet-Ext . Real = recorded human speech; In-the-wild = unscripted, naturalistic. Open license means CC-BY or CC-BY-NC 4.0 or equivalent; var. means varies across pooled corpora.

Source dataset	Clips	Annotation calls (18 $\times$ )
Multilingual In The Wild	721k	12.98M
Majestrino	973k	17.51M
LAION’s Got Talent	1.69M	30.34M
Emolia-Balanced (balanced subset of Emolia )	5.26M	94.62M
Total	8.64M	155.46M

Appendix

Table 4: The four MOSS-Audio -annotated voice corpora. Each clip is queried with 18 prompt-groups; the per-corpus annotation-call counts follow.

Corpus	Clips	Caption key
LAION’s Got Talent	1,685,809	detailed_caption
Emolia-Balanced	5,256,683	emotion_caption
Majestrino	972,658	caption
Synthetic Vocal Bursts	325,548	text (in-house, YouTube)
Improved Synthetic Vocal Bursts	15,680	text (in-house, YouTube)
EARS	17,227	text (FCaps)

Appendix

Table 5: VoiceCLAP Data training corpus manifest (Multilingual In The Wild is annotated and released as part of the Emolia suite but excluded from training after ablation). Caption-key indicates which JSON field is used as the contrastive text.

Model	VoiceNet-Emo bal@pp	VoiceNet-Ext bal@pp	VoiceNet-Emo $\rho$	VoiceNet-Ext $\rho$
LAION-CLAP	0.5624 [0.5511, 0.5727]	0.6184 [0.6104, 0.6268]	0.0310 [0.0071, 0.0537]	0.0600 [0.0421, 0.0753]
MGA-CLAP	0.5897 [0.5789, 0.5998]	0.6208 [0.6128, 0.6287]	0.0947 [0.0717, 0.1186]	0.0819 [0.0653, 0.0962]
GLAP	0.5765 [0.5661, 0.5877]	0.6254 [0.6171, 0.6341]	0.0667 [0.0425, 0.0907]	0.0836 [0.0655, 0.0992]
M2D-CLAP-2025	0.5681 [0.5571, 0.5787]	0.6361 [0.6273, 0.6444]	0.0348 [0.0121, 0.0582]	0.1003 [0.0822, 0.1142]
CLSP	0.5675 [0.5563, 0.5791]	0.6221 [0.6140, 0.6303]	0.0610 [0.0380, 0.0840]	0.0937 [0.0763, 0.1072]
MS-CLAP-2023	0.5754 [0.5646, 0.5858]	0.6321 [0.6243, 0.6403]	$-$ 0.0084 [ $-$ 0.0303, 0.0153]	0.1044 [0.0867, 0.1186]

Appendix

Table 6: Paired clip-bootstrap 95% CIs for Tab. 1 (2,000 replicates). Entries are point estimate

[\mathrm{lo},\mathrm{hi}]

. Per-prompt thresholds are held fixed; intervals therefore do not correct their same-set fitting. VoiceNet-Ext remains preliminary because this resampling also does not propagate annotation uncertainty.

emotion	VoiceCLAP Small	VoiceCLAP Large	LCO Omni-3B	LCO Omni-7B	LAION CLAP	MS-CLAP 2023	CLSP	GLAP	M2D CLAP	MGA CLAP	Cacophony
Anger	0.555	0.577	0.455	0.524	0.310	0.348	0.486	0.389	0.228	0.448	0.054
Impatience and Irritability	0.638	0.581	0.524	0.544	0.212	$-$ 0.086	0.149	0.375	$-$ 0.132	0.489	0.092
Contempt	0.398	0.441	0.433	0.497	0.230	$-$ 0.008	0.149	0.254	0.186	0.394	0.007
Amusement	0.518	0.500	0.509	0.530	0.283	$-$ 0.281	$-$ 0.351	0.530	0.062	0.526	0.154
Distress	0.627	0.626	0.538	0.575	$-$ 0.030	$-$ 0.255	0.226	0.226	0.325	0.382	$-$ 0.318
Contemplation	0.449	0.464	0.437	0.423	0.022	0.312	0.161	$-$ 0.022	0.138	0.161	0.185

Appendix

Table 7: Per-emotion Spearman

\rho

on VoiceNet-Emo for VoiceCLAP , the Omni-Embedding bases, and all seven general-audio CLAP baselines. Rows are sorted by the mean

\rho

across the displayed model columns; the mean is used only for ordering and is not shown. Best per row is bold, runner-up underlined. Cells are coloured by

\rho

Code	Description	$n$	maj-yes	polar. match
Perceived speaker (2)
GEND	gender presentation	350	0.237	0.594
AGEV	age range	349	0.284	0.564
Affective dimensions (7)
VALN	valence	343	0.315	0.624
AROU	arousal	343	0.280	0.618

Appendix

Table 8: Per-attribute coverage and majority-vote statistics on VoiceNet-Ext .

n

is the number of (audio, attribute-level) pairs released for that attribute, aggregated across the seven rubric levels; per-(attribute, level) prompt counts are therefore roughly

n/7

. maj-yes is the fraction of items where the rater majority voted “present”; polar. match is the fraction of items where the majority vote agrees with the prompt’s intended affirmative-vs.-contrastive polarity. Per-attribute Spearman

\rho

for each model is computed over the same

n

items per attribute and is released alongside the paper as a data artifact. Attribute codes follow the schema in App. A.4 ; rows are ordered to match that grouping.

Figure 1: Expert agreement on emotion presence for VoiceNet-Emo , per emotion, on the 7,984 pairs with three ratings. Stacked bars show the proportion of (audio, emotion) pairs by the number of experts who marked the emotion present (weakly or strongly): three ( 3:0 unanimous present ), two ( 2:1 ), one ( 1:2 ), or none ( 3:0 unanimous absent ). The text to the right of each bar gives the number of pairs $n$ and the share of unanimous pairs. Rows are sorted by unanimity.

Figure 2: Scaling view of VoiceNet performance. Each point is one model. The y-axis is the unweighted mean of VoiceNet-Emo and VoiceNet-Ext per-prompt balanced accuracy from Table 1 . The red step line tracks the best score reached by any model up to each parameter count. VoiceCLAP-Small (110M) sits above the general-audio CLAP cluster and is competitive with the larger Omni bases, while the 7B VoiceCLAP-Large reaches the highest average score. Parameter counts are approximate (audio + text encoder, before LoRA adapters). Averages match Tab. 1 .

Figure 3: Instructions given to the human annotator for the expert annotation of VoiceNet-Emo .

Figure 4: UI of our expert annotation tool for VoiceNet-Emo .

Figure 5: UI of the second annotation interface, used for the emotion-presence binary confirmation task on VoiceNet-Emo . Annotators listen to a clip and confirm or reject whether the target emotion is present.

Figure 6: UI of the second annotation interface, used for the talking-style attribute binary confirmation task on VoiceNet-Ext . For each (dimension, level) bucket, annotators confirm or reject whether the audio matches the target rubric-level description.

Explore similar work

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models
Aug 28, 2026Luc Debaupte, Tyler Baumgartner, Brandon Tai +3Audio-Language Model Evaluation Speech Emotion Recognition
Besimple AI, San Mateo, CA
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
Jul 16, 2026David Ayllon, Alice Baird, Jeffrey Brooks +11ASR Evaluation Voice Agent Evaluation
Hume AI Research
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Mar 9, 2026Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan +1Speech Emotion Recognition Speech Language Models
Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA

Aug 28, 2026J/K move · Enter open · S save

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

Luc Debaupte, Tyler Baumgartner, Brandon Tai, Candice Fan+2

Besimple AI, San Mateo, CA

Audio-Language Model Evaluation Speech Emotion Recognition

Open paper AskPDF arXiv

Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio. The benchmark contains 273 human-recorded English WAV clips from 51 speaker accounts totaling 1.95 hours across seven labels: angry, disgusted, fearful, happy, neutral, sad, and surprised, with 39 clips per class. All baselines are evaluated from audio alone, without transcripts or contextual metadata. Across six released baselines, average accuracy is 35.5%. The strongest baseline, gemini_3_5_flash, reaches 46.5% on the seven-way task, above the 14.3% random baseline but far from robust emotion recognition. A secondary valence-bucket analysis maps labels into positive, neutral, and negative classes, excluding surprised because its valence is ambiguous. Aggregate accuracy under this coarser view is 50.9%. Performance is highly uneven across classes. By recall, neutral is identified most reliably at 75.6% averaged across baselines, while surprised and fearful reach only 10.7% and 15.4%, respectively. These results show that the evaluated baselines can extract some affective signal from speech, but discrete expressed-emotion recognition remains fragile, especially for non-neutral emotions that are often most important in voice agent workflows.

Abstract

Explore similar work

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs