EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions

Xin Jing; Andreas Triantafyllopoulos; Jiadong Wang; Shahin Amiriparian; Jun Luo; Björn Schuller

cs.SDMar 10, 2026

EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions

Authors: Xin Jing, Andreas Triantafyllopoulos, Jiadong Wang, Shahin Amiriparian, Jun Luo, Björn Schuller

Organizations: CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany · MCML - Munich Center for Machine Learning, Munich, Germany · Huawei, Netherlands · GLAM, Imperial College London, UK

Abstract

Recent advancements in speech captioning models have enabled the generation of rich, fine-grained captions for emotional speech. However, the evaluation of such captions remains a critical bottleneck: traditional N-gram metrics fail to capture semantic nuances, while LLM judges often suffer from reasoning inconsistency and context-collapse when processing long-form descriptions. In this work, we propose EmoSURA, a novel evaluation framework that shifts the paradigm from holistic scoring to atomic verification. EmoSURA decomposes complex captions into Atomic Perceptual Units, which are self-contained statements regarding vocal or emotional attributes, and employs an audio-grounded verification mechanism to validate each unit against the raw speech signal. Furthermore, we address the scarcity of standardized evaluation resources by introducing SURABench, a carefully balanced and stratified benchmark. Our experiments show that EmoSURA achieves a positive correlation with human judgments, offering a more reliable assessment for long-form captions compared to traditional metrics, which demonstrated negative correlations due to their sensitivity to caption length.

Figures & tables

Type	Version	Caption Text
A	GT Caption	A male speaker delivers the line… in a bold, vibrant voice brimming with confident enthusiasm . His tone is commanding and energetic , projecting authority while radiating positivity …
A	Sabotaged	A male speaker delivers the line… in a weak, shaky voice brimming with anxious dread . His tone is fearful and hesitant , projecting vulnerability while radiating negativity …
B	GT Caption	A male speaker introduces himself … in a gruff, commanding manner. Despite the low, rumbling pitch … his voice carries an intense, forceful edge… The texture is rough and raspy …
B	Sabotaged	A female speaker introduces herself … in a soft, gentle manner. Despite the high, delicate pitch … her voice carries a light, airy edge… The texture is smooth and clear …
C	GT Caption	A female speaker delivers the line "…" with a lively, upbeat energy and confident presence . Her voice is clear and smooth , projecting loudly and rapidly , giving her words a sense of animated assurance …
C	Sabotaged	A female speaker sings the line "…" with a melodic, operatic energy . Her voice is musical and rhythmic , projecting loudly in a song-like manner , giving her words a sense of artistic performance …

Table 1: Case examples of the four hallucination categories. The injected hallucinations and altered acoustic descriptions are highlighted in bold red to illustrate the contrast against the ground-truth (GT) captions.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: The framework of EmoSURA. It consists of three steps: (1) Decomposition of captions into Atomic Perceptual Units (APUs) using LLMs; (2) Verification of generated APUs against the raw audio using an ALM; and (3) Matching generated APUs with benchmark to assess comprehensiveness.

Figure 2: The emotional distribution of SURABench in the Valence-Arousal space. Point colors represent Dominance. The marginal histograms (top and right) demonstrate the uniformity of the dataset.

Figure 3: Distribution plots (top) and scatter plots against human ratings (bottom) for baseline metrics versus EMOSURA. Unlike Bleu 4, ROUGE_L, and CIDEr, which show negative correlations (red lines), EMOSURA demonstrates a positive linear relationship (green line) with human ground truth, indicating higher reliability.

Rule-Based Metrics
Metric	PCC ( $\rho$ ) $\uparrow$	Kd $\tau$ $\uparrow$	Sp $\tau$ $\uparrow$
BLEU-4 [ 27 ]	-0.6419	-0.4494	-0.6916
ROUGE-L [ 28 ]	-0.7017	-0.4606	-0.6916
METEOR [ 29 ]	-0.5813	-0.4541	-0.6559
CIDEr [ 30 ]	-0.6640	-0.3732	-0.6175
SPIDER [ 31 ]	-0.6679	-0.4481	-0.7019

Appendix

Table 1: Correlation between automated evaluation metrics and human ratings. Performance is measured using Pearson correlation coefficient (PCC), Kendall’s rank correlation coefficient (Kd

\tau

) and sample-wise

\tau

(Sp

\tau

). All p-values

<

0.001.

Perturbation	Injected	Detected	Rate (%)
Acoustics Feat.	120	112	93.33
– Gender	40	39	97.50
– Pitch, Tempo, Vol., etc.	80	73	91.25
Emotion	40	33	82.50
Vocal Event	40	24	60.00

Appendix

Table 2: Detection performance of EmoSURA across different hallucination categories. The system demonstrates exceptional sensitivity to demographic and acoustic factual errors.

Audio Captioning

Audio captioning turns sound into natural-language descriptions of events, attributes, speakers, music, and their timing.

See all 23 papers

AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis
Apr 5, 2026Tianhua Qi, Wenming Zheng, Björn W. Schuller +2Emotional Speech Synthesis Speech Emotion Recognition
School of Biological Science and Medical Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education, China · Chair of Health Informatics (CHI), Technical University of Munich, 80333 Munich, Germany +3
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Mar 9, 2026Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan +1Speech Emotion Recognition Speech Language Models
Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Aug 10, 2026Junyu Wang, Siyuan Zhang, Peiyuan Jiang +11Speech Language Models
Tianjin Key Laboratory of Cognitive Computing and Application Tianjin University Tianjin, China · Tianjin University Tianjin, China · Fuzhou University Fuzhou, China +2

Apr 5, 2026J/K move · Enter open · S save

AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis

Tianhua Qi, Wenming Zheng, Björn W. Schuller, Zhaojie Luo+1

School of Biological Science and Medical Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education, China · Chair of Health Informatics (CHI), Technical University of Munich, 80333 Munich, Germany+3

Emotional Speech Synthesis Speech Emotion Recognition

Open paper AskPDF arXiv

Emotion is essential in spoken communication, yet most existing frameworks in speech emotion modeling rely on predefined categories or low-dimensional continuous attributes, which offer limited expressive capacity. Recent advances in speech emotion captioning and synthesis have shown that textual descriptions provide a more flexible and interpretable alternative for representing affective characteristics in speech. However, progress in this direction is hindered by the lack of an emotional speech dataset aligned with reliable and fine-grained natural language annotations. To tackle this, we introduce AffectSpeech, a large-scale corpus of human-recorded speech enriched with structured descriptions for fine-grained emotion analysis and generation. Each utterance is characterized across six complementary dimensions, including sentiment polarity, open-vocabulary emotion captions, intensity level, prosodic attributes, prominent segments, and semantic content, enabling multi-granular modeling of vocal expression. To balance annotation quality and scalability, we adopt a human-LLM collaborative annotation pipeline that integrates algorithmic pre-labeling, multi-LLM description generation, and human-in-the-loop verification. Furthermore, these annotations are reformulated into diverse descriptive styles to enhance linguistic diversity and reduce stylistic bias in downstream modeling. Experimental results on speech emotion captioning and synthesis demonstrate that models trained on AffectSpeech consistently achieve superior performance across multiple evaluation settings.

Abstract

Audio Captioning

Explore similar work

AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models