AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis
Authors: Tianhua Qi, Wenming Zheng, Björn W. Schuller, Zhaojie Luo, Haizhou Li
Organizations: School of Biological Science and Medical Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education, China · Chair of Health Informatics (CHI), Technical University of Munich, 80333 Munich, Germany · Group on Language, Audio, & Music (GLAM), Imperial College London, SW7 2AZ London, U.K. · Shenzhen Loop Area Institute, Shenzhen 518045, China · Shenzhen Research Institute of Big Data, School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen 518172, China
Emotion is essential in spoken communication, yet most existing frameworks in speech emotion modeling rely on predefined categories or low-dimensional continuous attributes, which offer limited expressive capacity. Recent advances in speech emotion captioning and synthesis have shown that textual descriptions provide a more flexible and interpretable alternative for representing affective characteristics in speech. However, progress in this direction is hindered by the lack of an emotional speech dataset aligned with reliable and fine-grained natural language annotations. To tackle this, we introduce AffectSpeech, a large-scale corpus of human-recorded speech enriched with structured descriptions for fine-grained emotion analysis and generation. Each utterance is characterized across six complementary dimensions, including sentiment polarity, open-vocabulary emotion captions, intensity level, prosodic attributes, prominent segments, and semantic content, enabling multi-granular modeling of vocal expression. To balance annotation quality and scalability, we adopt a human-LLM collaborative annotation pipeline that integrates algorithmic pre-labeling, multi-LLM description generation, and human-in-the-loop verification. Furthermore, these annotations are reformulated into diverse descriptive styles to enhance linguistic diversity and reduce stylistic bias in downstream modeling. Experimental results on speech emotion captioning and synthesis demonstrate that models trained on AffectSpeech consistently achieve superior performance across multiple evaluation settings.
Figures & tables
Dataset
# Samples
# Emo.
Elicitation
Desc. Form
Sent.
Emo*
Pros.
Int.
Seg.
Sem.
# Desc.
Anno. by
Categorical
SAVEE [ 20 ]
480
7
Acted
Label
✗
✗
✗
✗
✗
✗
480
Human
EMO-DB [ 19 ]
535
7
Acted
Label
✗
✗
✗
✗
✗
✗
535
Human
eNTERFACE [ 34 ]
1,263
6
Acted
Label
✗
✗
✗
✗
✗
✗
1,263
Human
JL-Corpus [ 35 ]
2,400
5
Acted
Label
✗
✗
✗
✗
✗
✗
2,400
Human
TESS [ 36 ]
2,800
7
Acted
Label
✗
✗
✗
✗
✗
✗
2,800
Human
TABLE I: Comparison of representative emotional speech datasets based on their provided annotation types and data characteristics.
Fig. 1: The construction and annotation pipeline of the AffectSpeech dataset.
Fig. 2: Composition and scale of the AffectSpeech dataset categorized by original data sources.
Fig. 3: Illustration of diverse speech emotion captioning styles.
Fig. 4: Quantitative distributions of speaker gender, sentiment polarity, intensity level, and prosodic attributes (pitch, tempo, and energy).
Verification Stage
Count
Proportion [%]
Direct Consensus a
164,747
64.9
Manual Selection b
73,468
28.9
Manual Correction c
15,584
6.2
Total
253,799
100.0
a Descriptions from multiple MLLMs exhibited high keypoint consistency.
b Human experts identified one valid candidate despite keypoint discrepancies.
TABLE II: Statistical distribution of LLM adjudication and human verification outcomes.
Fig. 5: Text length distributions across six textual description styles.
Fig. 6: Empirical cumulative distribution function of utterance duration.
Fig. 7: Word cloud of the top-100 most frequent emotion-related words.
Model
Emotion Accuracy [%] ↑
Prosody Accuracy [%] ↑
Description Quality
Category
Intensity
Key Segment
Pitch
Tempo
Energy
Distinct-2
Self-BLEU
Avg. Word
GPT-4o mini Audio
58.88
36.75
32.63
37.50
64.00
31.75
0.4435
0.5936
112.81
GPT-4o Audio
66.50
57.13
37.25
41.75
64.25
36.38
0.3439
0.6975
101.50
Step-Audio 2 mini [ 60 ]
66.63
45.00
35.75
33.38
64.75
38.25
0.1867
0.8598
84.00
Audio Flamingo 3 [ 61 ]
70.50
52.75
38.25
31.13
60.50
35.63
0.1562
0.8639
51.57
MiniCPM-o 2.6
64.25
56.38
36.00
30.50
60.38
35.50
0.1280
0.9239
85.13
TABLE III: Performance of Speech Emotion Captioning across different models and supervised Fine-tuning datasets.
Fig. 8: Subjective preference test results, which compare SFT variants trained on EmoVoice-DB, TextrolSpeech [ 17 ] , and the proposed AffectSpeech across three perceptual dimensions.
Model
Output
Qwen2-Audio [ 46 ]
The emotional tone is neutral. The speech emotion is calm. The emotional intensity is low. The pitch level is normal. the speech tempo is normal. the energy level is low. the key segment of emotional expression occurs in the entire part.
Qwen2.5-Omni [ 47 ]
The overall emotional tone is neutral; the speech emotion is calm with no particular intensity noted; the pitch level is normal; the speech tempo is normal; the energy level is moderate; the key segment of emotional expression occurs throughout the entire speech as there’s no specific emphasis on any part.
Qwen2.5-Omni-sft
EmoVoice-DB [ 16 ]
Expressing a neutral and balanced vocal quality.
TextrolSpeech [ 17 ]
The man speaks with a regular pitch, conversing at an average pace with standard energy.
AffectSpeech (proposed)
The audio has a neutral emotional tone and low emotional intensity. The speaker maintains a normal pitch, rate, and volume throughout, with no noticeable emotional fluctuations, conveying a calm and composed manner.
TABLE IV: Qualitative comparison of representative captions generated by different models and supervised Fine-tuning datasets.
Model
Objective Evaluation
Subjective Evaluation
WER [%] ↓
DDUR [s] ↓
Emo_Sim [%] ↑
Emo_Div [ ×10−4 ] ↑
Audio Quality
Intelligibility
Naturalness
GPT-4o mini TTS
4.72
0.27
66.85
15.69
3.97 ± 0.15
4.03 ± 0.21
4.08 ± 0.25
TTS-1
4.28
0.23
85.95
14.51
3.59 ± 0.15
3.79 ± 0.20
3.62 ± 0.21
TTS-1 HD
1.11
0.21
80.07
14.07
3.72 ± 0.26
3.83 ± 0.21
3.31 ± 0.26
PromptTTS [ 15 ]
15.73
0.34
64.13
6.85
2.97 ± 0.17
2.95 ± 0.12
3.15 ± 0.23
Salle [ 17 ]
10.97
0.23
70.14
13.75
3.49 ± 0.23
3.36 ± 0.28
3.22 ± 0.27
TABLE V: Quantitative and Qualitative Comparisons of Emotional Speech Synthesis with Textual Description.
Fig. 9: Perceptual attribute analysis of synthesized speech, illustrating subjective ratings (1–7 Likert scale) for prosodic fluency, emotion accuracy, and emotion diversity.
Erik Jonsson School of Engineering and Computer Science, University of Texas at Dallas, Richardson, TX 75080 USA · Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA-15213 USA · AT&T Labs Research, Bedminster, NJ USA