Organizations: Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Independent Researcher · National Taiwan University, Taiwan · National Taiwan University of Science and Technology, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan
Evaluating emotion preservation in expressive speech generation involves assessing how closely generated speech matches a reference in emotion. Human listening tests assess this similarity, but their cost motivates automatic measures aligned with human judgments. To support the development and evaluation of such measures, we introduce SES-Bench, a speech emotion similarity benchmark built from human comparisons of two candidate utterances against a shared reference. These comparisons record which candidate listeners find emotionally closer to the reference and the strength of their preference. Using these annotations, we train SES-Judge to score emotion similarity between two utterances. SES-Judge significantly outperforms embedding cosine similarity and prompted large audio-language models in preference accuracy and correlation with human ratings that capture both preference direction and strength.
Figures & tables
Figure 1: Annotation schematic. An annotator compares two candidates with a reference and rates their relative emotional similarity on a seven-point scale.
Figure 2: Distributions of agreement, reference emotion labels, and generation systems in SES-Bench, with Train and Test shown side by side for each group. System proportions count candidate occurrences separately for TTS and VC.
Model / Encoder
Training data
Training task / loss
Scoring input
Scoring rule
EmotionRankCLAP
—
—
Projected audio embedding
Cosine similarity
WavLM-SER
—
—
SER head-hidden embedding
Cosine similarity
emotion2vec
—
—
Final encoder-layer embedding
Cosine similarity
WavLM-SER emotion2vec
MSP-Podcast with / without synthetic speech
Emotion classification Cross-entropy
Fused embedding
Cosine similarity
Head-hidden embedding
Cosine similarity
Class probabilities
Negative KL divergence
Table 1: Baseline configurations. Training data and objectives refer to additional training in this work. Each listed scoring input defines a separate evaluation. Fused embeddings combine layer representations; head-hidden embeddings are taken after the downstream hidden layer. Classification is trained separately with and without synthetic speech.
Method
Spearman
Accuracy (%)
SES-Judge
0.5297∗
74.96∗
Embedding cosine similarity
EmotionRankCLAP
0.3537∗
63.46∗
WavLM-SER
0.3591∗
64.25∗
emotion2vec
0.3064∗
65.51∗
WavLM-SER + downstream training
Table 2: Baselines grouped by encoder and training objective. Fused and hidden cosine use embeddings before and after the downstream hidden layer. VAD distance is the negative Euclidean distance between predicted VAD vectors. ∗ indicates a significant difference from SES-Judge (paired triplet bootstrap for Spearman; exact two-sided McNemar for accuracy, α=0.05 ).
Method
Spearman
Accuracy (%)
SES-Judge
0.5297∗
74.96∗
WavLM-SER + downstream training
Classification / Fused cosine
0.3559∗
66.61∗
Classification / Hidden cosine
0.1783∗
57.80∗
Classification / Negative KL
0.1330∗
56.38∗
emotion2vec + downstream training
Table 3: Comparison with classification baselines trained on MSP-Podcast and synthetic speech from SES-Bench Train. Fused and hidden cosine use embeddings before and after the downstream hidden layer. ∗ indicates a significant difference from SES-Judge (paired triplet bootstrap for Spearman; exact two-sided McNemar for accuracy, α=0.05 ).
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, USA · Gilbert AI Lab, USA +2
Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA