Emotion is a core paralinguistic feature in voice interaction. It is widely believed that emotion understanding models learn fundamental representations that transfer to synthesized speech, making emotion understanding results a plausible reward or evaluation metric for assessing emotional expressiveness in speech synthesis. In this work, we critically examine this assumption by systematically evaluating Speech Emotion Recognition (SER) on synthesized speech across datasets, discriminative and generative SER models, and diverse synthesis models. We find that current SER models can not generalize to synthesized speech, largely because speech token prediction during synthesis induces a representation mismatch between synthesized and human speech. Moreover, generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues. Overall, our findings suggest that existing SER models often exploit non-robust shortcuts rather than capturing fundamental features, and paralinguistic understanding in SLMs remains challenging.
Figures & tables
Figure 1: Speech emotion recognition accuracy on TESS and CREMA-D dataset. Synthetic speech results represent the agreement with human, as audio samples with indistinct emotional expressions were manually excluded. SER results utilizing Emotion2vec highlight a clear gap between human and synthesized speech.
Figure 2: The confusion matrix for speech emotion recognition is shown. The vertical axis represents the ground truth, and the horizontal axis represents the model’s predictions. Sub-figures (a) and (b) show SER results on human speech, TESS and CREMA-D. Sub-figures (c) and (d) show SER results for speech synthesized by two TTS models from LLM-generated text. Sub-figures (e) and (f) represent the same results as (c) and (d) after filtering out weak emotional expression. Sub-figures (g) and (h) show the same as (e) and (f), but with synthesis by S2S LLMs.
Figure 3: To mitigate the impact of text distribution generated by LLMs, we investigated speech emotion recognition performance on identical text datasets.
Figure 4: Confusion matrix of speech emotion recognition results for synthetic speech based on TESS text. The vertical axis represents the ground truth, while the horizontal axis represents the model’s predictions. Sub-figs (a) and (b) show SER results for two TTS models, while (c) and (d) show SER results for two S2S models.
Human
Synthesized
IEM.
RAV.
TESS
IEM.
TESS
I. Emotion2vec Train on Human Speech
Ang
77.23
81.25
97.50
63.27
50.00
Hap
64.14
56.25
100.00
80.30
0.00
Neu
71.98
100.00
100.00
43.24
0.00
Sad
75.68
50.00
100.00
24.59
0.00
Table 1: SER accuracy on real or synthesized speech while trained with human or synthesized speech data. The IEMOCAP dataset contains only 4 emotion labels.
Figure 5: Speech emotion recognition accuracy of SLMs. Solid colors and shaded areas represent the SER results of Qwen3-omni and GPT-4o Audio respectively.
Human
Synthesized
IEM.
RAV.
TESS
IEM.
TESS
Emotion2vec
71.28
77.88
99.64
49.58
15.31
C2SER
78.64
53.85
47.50
74.24
54.59
Gemini 2.5 Pro
65.10
30.77
22.14
72.02
26.53
Table 2: Emotion recognition accuracy of top-tier open-source SER model and commercial S2S LLM.
Figure 6: Confusion matrix of the speech emotion recognition results of Google Gemini 2.5 pro. All samples are synthesized by GPT-4o TTS and GPT-4o Audio.
Figure 7: Visualization of the Emotion2vec embedding space, distinguished by synthesis type, vocoder, and emotion.
Metric(%)
Syn.
Voc.
Mod.
Emo.
Balanced Acc
99.81
96.25
75.01
59.86
Macro-F1
99.81
96.25
76.74
59.19
Table 3: Linear probing results on Emotion2vec representation space. Target probing categories including synthesis / human (Syn.), vocoder name (Voc.), model name (Mol.), and emotion label (Emo.).
Human Speech
Synthesized Speech
Method
In Domain
In Domain
OOD
IEMOCAP
RAVDESS
TESS
MELD
CREMA-D
IEMOCAP
TESS
CREMA-D †
CREMA-D ‡
MLP
71.13
66.35
99.64
50.15
78.43
89.20
88.27
32.77
49.38
DANN_syn
69.22
70.19
98.21
47.85
72.10
84.21
84.18
47.30
50.21
DANN_vocoder
69.07
69.23
97.14
47.24
71.70
84.49
85.20
51.35
46.88
DANN_model
69.37
68.27
98.21
47.97
71.96
81.99
83.16
41.22
43.87
Table 4: Under different fine-tuning strategies, the SER models fail to demonstrate generalization, both to cases where training data includes real CREMA-D and speech synthesis by CosyVoice and Kimi-audio but lacks CREMA-D synthesis with these models (OOD CREMA-D † ), and to unseen synthesis models (OOD CREMA-D ‡ ). The asterisk (*) indicates that positive and negative samples are balanced within each mini-batch.
Figure 8: Domain adversarial fine-tuning. An MLP further extracts features and is used for classification in understanding tasks. Meanwhile, a GRL is employed to prevent shortcut learning through artifact features.
Figure 9: The impact of three synthesis stages. The horizontal axis represents emotion recognition accuracy and deep red arrows indicate the error at each stage. The prompt audios or speech tokens utilize (1) the human speech itself, (2) truncated human speech with half length, (3) cross-speaker within TESS dataset, and (4) out-of-domain speaker in RAVDESS dataset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Human Speech
Synthesized Speech
Method
In Domain
In Domain
OOD
IEMOCAP
RAVDESS
TESS
MELD
CREMA-D
IEMOCAP
TESS
CREMA-D †
CREMA-D ‡
Emotion2vec
71.13
66.35
99.64
50.15
78.43
89.20
88.27
32.77
49.38
Qwen2.5-omni-3B
84.24
79.81
99.64
56.44
82.24
96.12
96.43
68.92
57.07
Appendix
Table 5: Qwen2.5-omni-3B results training with mixed datasets.
Figure 10: Confusion matrix of speech emotion recognition results. All samples are synthesized by GPT-4o TTS and GPT-4o Audio.
Figure 11: The impact of three synthesis stages of IndexTTS2.
Figure 12: The horizontal axis represents the proportion of data where humans perceive significant emotional expression, while the vertical axis indicates the Emotion2vec SER accuracy.
Large speech language models have demonstrated strong capabilities in unified cross-modal understanding and generation, yet paralinguistic cues, especially emotion, remain difficult to preserve. Existing systems typically rely on entangled acoustic representations, which allow the underlying language model to depend excessively on recovered lexical content instead of grounding its behavior in acoustic-prosodic evidence. We address this limitation with EMODE, an emotion-aware speech language model built around Dynamic Para-Semantic Experts (DPSE). DPSE decomposes continuous speech features into semantic and paralinguistic pathways, routes them dynamically, and fuses them before integration into the language model. To turn this structural decomposition into functional specialization, EMODE is trained with a three-stage curriculum consisting of semantic warm-up, paralinguistic activation, and joint refinement, guided by Orthogonal Expert Guidance (OEG), Semantic-to-Acoustic Alignment (SAA), and Gating Diversity Regularization (GDR). Experiments on SER test, empathetic response evaluation, and the newly constructed bilingual MEPA benchmark show that EMODE improves the balance between lexical fidelity and emotional sensitivity, strengthens affect-grounded response generation, and exposes the value of explicit para-semantic factorization for robust cross-corpus emotion understanding.
Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.
Kuang Wang, Lai Wei, Ping Lin +8
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.