VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

Hezhao Zhang; Huang-Cheng Chou; Shrikanth Narayanan; Thomas Hain

cs.SDMar 9, 2026

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

Authors: Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan, Thomas Hain

Organizations: Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA

Abstract

Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.

Figures & tables

Dataset	Year	Aud. S.	Lab. S.	#Emos	#Spks	Selected #Utts	#Hrs	Langage
In-the-wild
EmotionTalk [ 25 ]	2025	Talkshow	Perc	7	30	19,250	23.6	Mandarin
MSP-Podcast [ 19 , 18 ]	2025	Podcast	Perc	8	3,711	212,801	324.1	English
BIIC-Podcast [ 26 ]	2023	Podcast	Perc	8	–	69,874	147.2	Taiwanese Mandarin
M3ED [ 27 ]	2022	TV	Perc	7	634	24,437	9.8	Mandarin
ASVP-ESD [ 28 ]	2020	Online media	Perc	14	132	13,964	18.0	English/Mandarin

Table 1: Dataset overview, grouped by acted vs in-the-wild recordings and sorted by year. Audio source ( Aud. S. ) reports recording origin for in-the-wild corpora, and scripted, spontaneous, mixed ( Script/Spont ) style for acted corpora. Label source indicates available label sources ( Lab. S. ): Expr (expressed/actor-intended), Perc (perceived/listener-annotator). Emo, #Utts, and #Hrs reflect the benchmark subset after dataset-specific filtering and label-set restriction (e.g., excluding no-agreement samples in MSP-Podcast and using a commonly adopted label subset for IEMOCAP).

	Qwen2-Audio	Audio Flamingo 3
Audio encoder	Whisper-large-v3	Whisper-large-v3
LLM backbone	Qwen-7B	Qwen2.5-7B
Training data	370k hrs, undisclosed	$>$ 9M QA pairs, released
IEMOCAP WA (%)	59.2	63.8

Table 2: Comparison of the two evaluated speech LLMs.

Zero-shot Prompt Template
Task description	You are a speech emotion recognition system. You are given an audio clip.
Emotion categories	Hard-label prompts: Choose exactly one emotion label from this closed set: {labels} . Distribution prompt: Consider the following emotion labels: {labels} .
Instruction blocks
+ Direct decision	Decide the emotion directly from the audio.
+ Distribution	Estimate how likely each emotion is based on the audio. Assign a probability (between 0.0 and 1.0) to every label. The probabilities must sum to 1.0. Also state which single label is most likely.
+ ASR transcript	Transcribe the spoken content from the audio as accurately as possible.

Table 3: Zero-shot prompt template used in VoxEmo. Prompt families are formed by adding instruction blocks to the shared template. For hard-label evaluation, FINAL_LABEL is parsed as the prediction.

Setting	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)
	AESDD (el, 5)			ASED (am, 5)			ASVP-ESD (en/zh, 14)			BIIC-Podcast (zh-tw, 8)
Q2A Direct	28.57	28.19	20.06	35.93	35.39	32.74	11.28	16.61	9.91	27.73	29.85	18.34
Q2A +T	28.18	28.19	27.08	24.59	25	18.34	7.84	11.27	10.38	\textcolor red!8528.66	\textcolor red!8533.7	\textcolor red!8519.96
Q2A +A	\textcolor red!8539.52	\textcolor red!8539.6	\textcolor red!8538.63	\textcolor red!8541.41	\textcolor red!8542.21	\textcolor red!8539.02	\textcolor red!8520.86	\textcolor red!8524.55	\textcolor red!8518.4	21.59	39.35	16.55
Q2A +T+A	27.38	27.52	24.67	29.11	29.87	25.16	9.45	12.47	10.17	20.51	36.98	15.29
Q2A +T+A+R	26.92	26.85	25.77	30.5	31.17	26.81	14.94	21.08	15.47	24.3	40.03	18.08

Table 4: VoxEmo zero-shot + SFT results. Q2A = Qwen2-Audio, AF3 = Audio Flamingo 3. Prompt abbreviations: T = ASR transcript, A = acoustic caption, R = reasoning. \textcolor red!85Bold red : best Q2A prompt per dataset; \textcolor red!85bold red : best AF3 prompt per dataset. \textcolor red!85Bold red : best score among Q2A SFT / AF3 SFT / EmoBox (by Macro-F1) per dataset. Part 1/2.

Setting	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)	UA(%)	WA(%)	F1(%)
	Oreau (fr, 7)			PAVOQUE (de, 5)			Polish (pl, 3)			RAVDESS (en, 8)
Q2A Direct	46.22	47.12	39.85	\textcolor red!8547.56	\textcolor red!8562.77	\textcolor red!8546.09	34.89	34.89	21.97	\textcolor red!8572.4	\textcolor red!8572.43	\textcolor red!8567.55
Q2A +T	39.84	41.35	37.53	27.72	53	29.07	34	34	27.91	49.15	45.76	44.54
Q2A +A	33.12	35.58	25.72	45.23	67.52	45.27	\textcolor red!8534.67	\textcolor red!8534.67	\textcolor red!8530.14	68.29	66.39	62.73
Q2A +T+A	41.89	44.23	41.82	31.14	59.61	33.85	35.11	35.11	25.8	49.8	47.01	48.57
Q2A +T+A+R	\textcolor red!8545.52	\textcolor red!8548.08	\textcolor red!8546.17	34.54	60.26	36.3	34.44	34.44	27.34	45.12	44.93	44

Table 5: VoxEmo zero-shot + SFT results. Q2A = Qwen2-Audio, AF3 = Audio Flamingo 3. Prompt abbreviations: T = ASR transcript, A = acoustic caption, R = reasoning. \textcolor red!85Bold red : best Q2A prompt per dataset; \textcolor red!85bold red : best AF3 prompt per dataset. \textcolor red!85Bold red : best score among Q2A SFT / AF3 SFT / EmoBox [ 13 ] (by MacroF1) per dataset. Part 2/2.

Dataset	Model	Prompt	Hard-decision Assessment			Distribution-aware Assessment
			Ma-F1 $\uparrow$	Mi-F1 $\uparrow$	Acc $\uparrow$	KLD $\downarrow$	JSD $\downarrow$	TVD $\downarrow$	SIM $\uparrow$	MSE $\downarrow$
BIIC-Podcast (zh-tw, 8)	Q2A	Direct	28.14	42.16	29.57	5.473	0.363	0.657	49.61	0.074
		+A	4.79	9.38	9.67	\textcolor red!852.440	0.402	0.749	45.78	0.073
		Ensemble	\textcolor red!8531.12	\textcolor red!8551.14	\textcolor red!8542.95	7.257	\textcolor red!850.309	\textcolor red!850.559	\textcolor red!8556.49	\textcolor red!850.070
	AF3	Direct	15.57	25.70	22.95	4.767	0.374	0.680	46.97	0.083
		+A	16.88	42.70	38.11	10.391	0.358	0.616	48.96	0.101

Table 6: Zero-shot soft-label evaluation results. We compare the single direct prompt ( Direct ), +A prompt, against the proposed prompt ensemble method ( Ensemble ). Performance is categorized into hard-decision metrics (Macro-F1, Micro-F1, and Top-1 Accuracy in %) and distribution-aware assessments , including Kullback–Leibler Divergence (KLD), Jensen–Shannon Divergence (JSD), Total Variation Distance (TVD), Cosine Similarity (Sim

\times 100

), and Mean Squared Error (MSE). \textcolor red!85Bold red denotes the superior setting per corpus.

\uparrow

(

\downarrow

) indicates that higher (lower) values are better.

Figure 1: Impact of zero-shot prompt complexity on performance and format adherence. The blue bars (left axis) denote the Average Macro-F1 across five evaluation corpora, while the red line (right axis) illustrates the Average Parse Failure Rate. As prompt constraints increase (from direct classification Direct to requesting intermediate acoustic captions +T ), both Q2A and AF3 struggle to follow the structured text-output format, leading to a severe degradation in overall predictive performance. This zero-shot stochasticity necessitates our proposed distribution-aware prompt ensemble methodology.

Dataset	Model / Framework	Ma-F1
BIIC-Podcast (zh-tw, 8)	[ 64 ]	35.5
	[ 6 ]	28.3
	This paper (Q2A-Ensemble)	31.1
	This paper (AF3-Ensemble)	21.0
CREMA-D (en, 6)	[ 21 ]	71.0
	[ 64 ]	70.6

Table 7: Comparison of best zero-shot speech LLMs against recent fully supervised state-of-the-art benchmarks on Hard-label Macro-F1 (%).

Source $\backslash$ Target	CREMA-D	IEMOCAP	MELD	MSP-Podcast 2
Qwen2-Audio
CREMA-D	91.1 (+21.7)	79.9 (+20.5)	30.7 (+2.6)	25.8 (+5.5)
IEMOCAP	76.8 (+7.4)	82.4 (+23.0)	24.7 (-3.4)	31.5 (+11.1)
MELD	86.6 (+17.1)	81.6 (+22.2)	39.3 (+11.2)	34.1 (+13.7)
MSP-Podcast 2	78.6 (+9.1)	68.1 (+8.7)	30.8 (+2.8)	29.2 (+8.8)
AudioFlamingo-3

Table 8: Cross-scenario transfer (Macro-F1). Each cell shows the score when fine-tuning on the source (row) and evaluating on the target (column), with

\Delta

over best zero-shot in parentheses. Diagonal ( bold ): in-domain SFT. MSP-Podcast uses test1.

Explore similar work

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
Sep 30, 2026Hezhao Zhang, Thomas HainConcept Bottleneck Models Emotion Recognition in Conversations
School of Computer Science, University of Sheffield Sheffield, United Kingdom
Toward Human-Aligned Judgement of Speech Emotion Similarity
Sep 26, 2026Yun-Shao Tsai, Yi-Cheng Lin, Chih-Kai Yang +7Speech Generation Evaluation Emotional Speech Synthesis
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Independent Researcher · National Taiwan University, Taiwan +2
Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition
Sep 17, 2026Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4Speech Emotion Recognition Speech Language Models
Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, USA & India +1

Sep 30, 2026J/K move · Enter open · S save

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

Hezhao Zhang, Thomas Hain

School of Computer Science, University of Sheffield Sheffield, United Kingdom

Concept Bottleneck Models Emotion Recognition in Conversations Speech Emotion Recognition Speech Language Models

Open paper AskPDF arXiv

Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.

Abstract

Explore similar work

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

Toward Human-Aligned Judgement of Speech Emotion Similarity

Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition