Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment
Authors: Kuang Wang, Lai Wei, Ping Lin, Qibing Bai, Wenkai Fang, Li Zhou, Feng Jiang, Zhongjie Jiang, +3 more
Organizations: The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University · Zhejiang University · Shenzhen University of Advanced Technology
Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.
Figures & tables
Figure 1: Comparison of context-aware expressive speech generation paradigms, categorized by their expressive control signals . Controllable TTS depends on external style references, whereas Vanilla SLMs naively reuse semantic hidden states hi to modulate text tokens ti into speech si . Unlike Emotional SLMs, which inject explicit emotion labels L prior to synthesis, SASLM self-distills intrinsic expressive intent zi directly from hi , eliminating proxy supervision.
Figure 3: Overview of the SASLM framework. Left: Intent-aware architecture with VIB-driven expressive modulation. Right: Three-stage realization-aware training paradigm, progressing from acoustic bootstrapping to closed-loop self-reward optimization.
Table 1: Datasets used in different training stages.
Model Type
Model Name
Utterance-Level Objective Metrics
Conversation-Level Subjective Metrics
WER% ↓
F0-Var ↑
EmoAlign% ↑
CE / CU / PC / PQ / Aes. ↑
Emo. ↑
Pro. ↑
Nat. ↑
Ovr. ↑
Controllable TTS (w/ Oracle Prompts)
CosyVoice2
3.80
51.20
39.07
6.21 / 7.09 / 1.42 / 7.67 / 5.60
4.15
4.25
4.34
4.23
GPT-4o-mini-TTS
7.63
70.44
45.04
6.22 / 7.37 / 1.41 / 8.00 / 5.75
4.27
4.43
4.58
4.41
Commercial SLM
Doubao
1.12
68.58
42.95
6.21 / 7.38 / 1.53 / 7.94 / 5.72
4.22
4.50
4.59
4.44
Vanilla SLM
Kimi Audio
6.99
46.98
22.64
5.31 / 5.98 / 1.49 / 6.87 / 4.91
3.77
4.02
4.15
3.97
Step Audio2 Mini
4.87
50.92
22.92
6.12 / 7.06 / 1.44 / 7.68 / 5.58
4.00
4.19
4.43
4.15
Table 2: Comparison of context-aware expressive speech generation. Bold and underline denote the best and second-best results among open-source SLMs , excluding commercial SLMs and controllable TTS systems.
Figure 4: Evaluation accuracy dynamics during UAPO training with different reward models.
Figure 5: t-SNE visualization of emotion clustering across token embeddings ( e ), LLM hidden states ( h ), and distilled VIB latent variables ( z ). Linear-probe accuracies, computed on the original representations rather than the t-SNE projections, are 18.1%, 41.4%, and 64.5%, respectively.
M(eyi,hyi)
RG
WER% ↓
Emo. ↑
Pro. ↑
Nat. ↑
Ovr. ↑
(A) Ablation on Modulation Strategy
π1
Acoustic
3.61
4.06
4.22
4.30
4.17
⊕
Acoustic
8.29
4.04
4.28
4.31
4.20
AdaLN
Acoustic
6.50
4.15
4.28
4.34
4.22
(B) Ablation on Context Grounding
VIB+AdaLN
∅
4.19
4.10
4.23
4.33
4.21
Table 3: Ablation study on the modulation strategy M(eyi,hyi) and context grounding RG in Eq. 9 .
Strategy
RM Size
WER(%) ↓
Emo. ↑
Pro. ↑
Nat. ↑
Ovr. ↑
SA-SLM(SFT)
—
4.74
4.21
4.34
4.45
4.31
+ Self-Reward
3B
4.56
4.24
4.36
4.49
4.33
+ Oracle-Reward
30B
4.21
4.30
4.41
4.54
4.38
Table 4: Comparison of reward strategies on Intelligibility (WER) and expressive speech quality.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model Name
Audio Understanding
Empathetic Response
Sound
Music
Speech
Avg.
CF
RN
Col.
IR
Qwen2.5-Omni-3B
30.24
49.25
39.04
39.51
4.54
4.66
4.51
3.24
Qwen2.5-Omni-7B
78.10
65.90
70.60
71.50
4.59
4.58
4.51
3.28
SASLM-3B (Ours)
29.94
49.25
39.63
39.60
4.56
4.65
4.40
3.24
Appendix
Table 5: Comparison of different speech language models on audio understanding (MMAU) and empathetic response ability (EchoMind). Bold = best; underline = second-best.
Model Type
Architecture Design
Intell.
Expressiveness
M(eyi,hyi)
RG
WER(%) ↓
F0-Var ↑
Emo. ↑
Pro. ↑
Nat. ↑
Ovr. ↑
Conversational TTS
π1
∅
3.53
55.88
4.04
4.20
4.30
4.17
Semantic
9.62
61.16
4.02
4.18
4.28
4.16
Acoustic
3.61
60.42
4.06
4.22
4.30
4.17
Naive Addition
⊕
∅
7.91
57.00
4.06
4.22
4.32
4.19
Semantic
15.40
57.61
4.11
4.24
4.35
4.21
Appendix
Table 6: Ablation study on expressive-aware speech generation across two independent design choices in Eq. 9 . (1) Intent Modulation Strategy M(eyi,hyi) : how semantic intent is injected for expressive generation, where π1 : M(eyi,hyi)=eyi discards SLM semantic intent (conversational TTS-style baseline); ⊕ : naive element-wise addition eyi+hyi Xu et al. (2025a) ; AdaLN Wang et al. (2025a) : adaptive modulation without VIB; VIB+AdaLN : our proposed disentangled modulation. (2) Context Grounding RG : how context is constructed during synthesis, where ∅ denotes no context grounding, Semantic denotes perception-pathway representations RG=EX , and Acoustic denotes generation-encoder acoustic features RG=ϕG(XS) .
Figure 6: Pairwise human Win/Tie/Loss between our model and each baseline (GPT-4o-mini-TTS, Qwen3-Omni-30B) on the three perceptual dimensions, together with an aggregated Overall row per baseline. Win (green) indicates that our model is preferred, Loss (red) indicates that the baseline is preferred, and Tie (gray) indicates perceptually comparable.
Figure 7: Human Win/Tie/Loss validation of reward models. Each row corresponds to a (model, dimension) configuration. Win (green) indicates that the RM-high sample is preferred by human raters, Loss (red) indicates that the RM-low sample is preferred, and Tie (gray) indicates that the two samples are judged perceptually comparable.
Figure 8: Prompt for evaluating emotion delivery in synthesized speech.
Figure 9: Prompt for text-level emotional QA completion. The figure illustrates the “answer-to-question” direction; other directions follow the same template by swapping the observed and generated sides.
Figure 10: Prompt for evaluating emotion and intensity in synthesized speech.
Figure 11: Annotation guideline shown to human evaluators in both the MOS study and the reward-model validation study.
Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.
Yizhong Geng, Yanliang Li, Jinghan Yang +4
Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA +2
Emotion is a core paralinguistic feature in voice interaction. It is widely believed that emotion understanding models learn fundamental representations that transfer to synthesized speech, making emotion understanding results a plausible reward or evaluation metric for assessing emotional expressiveness in speech synthesis. In this work, we critically examine this assumption by systematically evaluating Speech Emotion Recognition (SER) on synthesized speech across datasets, discriminative and generative SER models, and diverse synthesis models. We find that current SER models can not generalize to synthesized speech, largely because speech token prediction during synthesis induces a representation mismatch between synthesized and human speech. Moreover, generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues. Overall, our findings suggest that existing SER models often exploit non-robust shortcuts rather than capturing fundamental features, and paralinguistic understanding in SLMs remains challenging.
Yuan Ge, Haishu Zhao, Aokai Hao +10
Northeastern University, China · Meituan · NiuTrans Research +1
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.
Wenqian Cui, Xiao-Hui Li, Daxin Tan +2
The Chinese University of Hong Kong · Huawei Technologies