Emotion is a core paralinguistic feature in voice interaction. It is widely believed that emotion understanding models learn fundamental representations that transfer to synthesized speech, making emotion understanding results a plausible reward or evaluation metric for assessing emotional expressiveness in speech synthesis. In this work, we critically examine this assumption by systematically evaluating Speech Emotion Recognition (SER) on synthesized speech across datasets, discriminative and generative SER models, and diverse synthesis models. We find that current SER models can not generalize to synthesized speech, largely because speech token prediction during synthesis induces a representation mismatch between synthesized and human speech. Moreover, generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues. Overall, our findings suggest that existing SER models often exploit non-robust shortcuts rather than capturing fundamental features, and paralinguistic understanding in SLMs remains challenging.
Figures & tables
Figure 1: Speech emotion recognition accuracy on TESS and CREMA-D dataset. Synthetic speech results represent the agreement with human, as audio samples with indistinct emotional expressions were manually excluded. SER results utilizing Emotion2vec highlight a clear gap between human and synthesized speech.
Figure 2: The confusion matrix for speech emotion recognition is shown. The vertical axis represents the ground truth, and the horizontal axis represents the model’s predictions. Sub-figures (a) and (b) show SER results on human speech, TESS and CREMA-D. Sub-figures (c) and (d) show SER results for speech synthesized by two TTS models from LLM-generated text. Sub-figures (e) and (f) represent the same results as (c) and (d) after filtering out weak emotional expression. Sub-figures (g) and (h) show the same as (e) and (f), but with synthesis by S2S LLMs.
Figure 3: To mitigate the impact of text distribution generated by LLMs, we investigated speech emotion recognition performance on identical text datasets.
Figure 4: Confusion matrix of speech emotion recognition results for synthetic speech based on TESS text. The vertical axis represents the ground truth, while the horizontal axis represents the model’s predictions. Sub-figs (a) and (b) show SER results for two TTS models, while (c) and (d) show SER results for two S2S models.
Human
Synthesized
IEM.
RAV.
TESS
IEM.
TESS
I. Emotion2vec Train on Human Speech
Ang
77.23
81.25
97.50
63.27
50.00
Hap
64.14
56.25
100.00
80.30
0.00
Neu
71.98
100.00
100.00
43.24
0.00
Sad
75.68
50.00
100.00
24.59
0.00
Table 1: SER accuracy on real or synthesized speech while trained with human or synthesized speech data. The IEMOCAP dataset contains only 4 emotion labels.
Figure 5: Speech emotion recognition accuracy of SLMs. Solid colors and shaded areas represent the SER results of Qwen3-omni and GPT-4o Audio respectively.
Human
Synthesized
IEM.
RAV.
TESS
IEM.
TESS
Emotion2vec
71.28
77.88
99.64
49.58
15.31
C2SER
78.64
53.85
47.50
74.24
54.59
Gemini 2.5 Pro
65.10
30.77
22.14
72.02
26.53
Table 2: Emotion recognition accuracy of top-tier open-source SER model and commercial S2S LLM.
Figure 6: Confusion matrix of the speech emotion recognition results of Google Gemini 2.5 pro. All samples are synthesized by GPT-4o TTS and GPT-4o Audio.
Figure 7: Visualization of the Emotion2vec embedding space, distinguished by synthesis type, vocoder, and emotion.
Metric(%)
Syn.
Voc.
Mod.
Emo.
Balanced Acc
99.81
96.25
75.01
59.86
Macro-F1
99.81
96.25
76.74
59.19
Table 3: Linear probing results on Emotion2vec representation space. Target probing categories including synthesis / human (Syn.), vocoder name (Voc.), model name (Mol.), and emotion label (Emo.).
Human Speech
Synthesized Speech
Method
In Domain
In Domain
OOD
IEMOCAP
RAVDESS
TESS
MELD
CREMA-D
IEMOCAP
TESS
CREMA-D †
CREMA-D ‡
MLP
71.13
66.35
99.64
50.15
78.43
89.20
88.27
32.77
49.38
DANN_syn
69.22
70.19
98.21
47.85
72.10
84.21
84.18
47.30
50.21
DANN_vocoder
69.07
69.23
97.14
47.24
71.70
84.49
85.20
51.35
46.88
DANN_model
69.37
68.27
98.21
47.97
71.96
81.99
83.16
41.22
43.87
Table 4: Under different fine-tuning strategies, the SER models fail to demonstrate generalization, both to cases where training data includes real CREMA-D and speech synthesis by CosyVoice and Kimi-audio but lacks CREMA-D synthesis with these models (OOD CREMA-D † ), and to unseen synthesis models (OOD CREMA-D ‡ ). The asterisk (*) indicates that positive and negative samples are balanced within each mini-batch.
Figure 8: Domain adversarial fine-tuning. An MLP further extracts features and is used for classification in understanding tasks. Meanwhile, a GRL is employed to prevent shortcut learning through artifact features.
Figure 9: The impact of three synthesis stages. The horizontal axis represents emotion recognition accuracy and deep red arrows indicate the error at each stage. The prompt audios or speech tokens utilize (1) the human speech itself, (2) truncated human speech with half length, (3) cross-speaker within TESS dataset, and (4) out-of-domain speaker in RAVDESS dataset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Human Speech
Synthesized Speech
Method
In Domain
In Domain
OOD
IEMOCAP
RAVDESS
TESS
MELD
CREMA-D
IEMOCAP
TESS
CREMA-D †
CREMA-D ‡
Emotion2vec
71.13
66.35
99.64
50.15
78.43
89.20
88.27
32.77
49.38
Qwen2.5-omni-3B
84.24
79.81
99.64
56.44
82.24
96.12
96.43
68.92
57.07
Appendix
Table 5: Qwen2.5-omni-3B results training with mixed datasets.
Figure 10: Confusion matrix of speech emotion recognition results. All samples are synthesized by GPT-4o TTS and GPT-4o Audio.
Figure 11: The impact of three synthesis stages of IndexTTS2.
Figure 12: The horizontal axis represents the proportion of data where humans perceive significant emotional expression, while the vertical axis indicates the Emotion2vec SER accuracy.