Organizations: Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Independent Researcher · National Taiwan University, Taiwan · National Taiwan University of Science and Technology, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan
Evaluating emotion preservation in expressive speech generation involves assessing how closely generated speech matches a reference in emotion. Human listening tests assess this similarity, but their cost motivates automatic measures aligned with human judgments. To support the development and evaluation of such measures, we introduce SES-Bench, a speech emotion similarity benchmark built from human comparisons of two candidate utterances against a shared reference. These comparisons record which candidate listeners find emotionally closer to the reference and the strength of their preference. Using these annotations, we train SES-Judge to score emotion similarity between two utterances. SES-Judge significantly outperforms embedding cosine similarity and prompted large audio-language models in preference accuracy and correlation with human ratings that capture both preference direction and strength.
Figures & tables
Figure 1: Annotation schematic. An annotator compares two candidates with a reference and rates their relative emotional similarity on a seven-point scale.
Figure 2: Distributions of agreement, reference emotion labels, and generation systems in SES-Bench, with Train and Test shown side by side for each group. System proportions count candidate occurrences separately for TTS and VC.
Model / Encoder
Training data
Training task / loss
Scoring input
Scoring rule
EmotionRankCLAP
—
—
Projected audio embedding
Cosine similarity
WavLM-SER
—
—
SER head-hidden embedding
Cosine similarity
emotion2vec
—
—
Final encoder-layer embedding
Cosine similarity
WavLM-SER emotion2vec
MSP-Podcast with / without synthetic speech
Emotion classification Cross-entropy
Fused embedding
Cosine similarity
Head-hidden embedding
Cosine similarity
Class probabilities
Negative KL divergence
Table 1: Baseline configurations. Training data and objectives refer to additional training in this work. Each listed scoring input defines a separate evaluation. Fused embeddings combine layer representations; head-hidden embeddings are taken after the downstream hidden layer. Classification is trained separately with and without synthetic speech.
Method
Spearman
Accuracy (%)
SES-Judge
0.5297∗
74.96∗
Embedding cosine similarity
EmotionRankCLAP
0.3537∗
63.46∗
WavLM-SER
0.3591∗
64.25∗
emotion2vec
0.3064∗
65.51∗
WavLM-SER + downstream training
Table 2: Baselines grouped by encoder and training objective. Fused and hidden cosine use embeddings before and after the downstream hidden layer. VAD distance is the negative Euclidean distance between predicted VAD vectors. ∗ indicates a significant difference from SES-Judge (paired triplet bootstrap for Spearman; exact two-sided McNemar for accuracy, α=0.05 ).
Method
Spearman
Accuracy (%)
SES-Judge
0.5297∗
74.96∗
WavLM-SER + downstream training
Classification / Fused cosine
0.3559∗
66.61∗
Classification / Hidden cosine
0.1783∗
57.80∗
Classification / Negative KL
0.1330∗
56.38∗
emotion2vec + downstream training
Table 3: Comparison with classification baselines trained on MSP-Podcast and synthetic speech from SES-Bench Train. Fused and hidden cosine use embeddings before and after the downstream hidden layer. ∗ indicates a significant difference from SES-Judge (paired triplet bootstrap for Spearman; exact two-sided McNemar for accuracy, α=0.05 ).
Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring emotional prosody transfer. To quantify this, the field widely relies on emotion similarity between reference and generated samples. This approach computes cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations. We challenge this assumption through controlled adversarial tasks and human alignment tests. Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation. Representational limitations cause linguistic and speaker interference to overshadow emotional features, degrading discriminative ability. Consequently, the metric misaligns with human perception. This acoustic vulnerability reveals it rewards acoustic mimicry over genuine emotional synthesis.
Yun-Shao Tsai, Yi-Cheng Lin, Huang-Cheng Chou +5
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, USA · Gilbert AI Lab, USA +2
Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.
Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio. The benchmark contains 273 human-recorded English WAV clips from 51 speaker accounts totaling 1.95 hours across seven labels: angry, disgusted, fearful, happy, neutral, sad, and surprised, with 39 clips per class. All baselines are evaluated from audio alone, without transcripts or contextual metadata. Across six released baselines, average accuracy is 35.5%. The strongest baseline, gemini_3_5_flash, reaches 46.5% on the seven-way task, above the 14.3% random baseline but far from robust emotion recognition. A secondary valence-bucket analysis maps labels into positive, neutral, and negative classes, excluding surprised because its valence is ambiguous. Aggregate accuracy under this coarser view is 50.9%. Performance is highly uneven across classes. By recall, neutral is identified most reliably at 75.6% averaged across baselines, while surprised and fearful reach only 10.7% and 15.4%, respectively. These results show that the evaluated baselines can extract some affective signal from speech, but discrete expressed-emotion recognition remains fragile, especially for non-neutral emotions that are often most important in voice agent workflows.