Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

Kuang Wang; Lai Wei; Ping Lin; Qibing Bai; Wenkai Fang; Li Zhou; Feng Jiang; Zhongjie Jiang

cs.CLApr 13, 2026

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

Authors: Kuang Wang, Lai Wei, Ping Lin, Qibing Bai, Wenkai Fang, Li Zhou, Feng Jiang, Zhongjie Jiang, +3 more

Organizations: The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University · Zhejiang University · Shenzhen University of Advanced Technology

Abstract

Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.

Figures & tables

Figure 1: Comparison of context-aware expressive speech generation paradigms, categorized by their expressive control signals . Controllable TTS depends on external style references, whereas Vanilla SLMs naively reuse semantic hidden states $h_{i}$ to modulate text tokens $t_{i}$ into speech $s_{i}$ . Unlike Emotional SLMs, which inject explicit emotion labels $L$ prior to synthesis, SASLM self-distills intrinsic expressive intent $z_{i}$ directly from $h_{i}$ , eliminating proxy supervision.

Figure 2: Illustration of existing end-to-end SLM architectures: (a) Modular multi-head architecture; (b) Unified single-head architecture.

Figure 3: Overview of the SASLM framework. Left: Intent-aware architecture with VIB-driven expressive modulation. Right: Three-stage realization-aware training paradigm, progressing from acoustic bootstrapping to closed-loop self-reward optimization.

Dataset	Stage	#Samples	Hours
InstructS2S Fang et al. (2025a)	I	200k	$\sim$ 2000 h
Genshin Dataset 2 2 2 https://www.modelscope.cn/datasets/aihobbyist/Genshin_Dataset	II	220k	$\sim$ 603 h
EmoVoice-DB Yang et al. (2025)	II	66k	$\sim$ 192 h
EmoNet Schuhmann et al. (2025)	III	200k	$\sim$ 263 h

Table 1: Datasets used in different training stages.

Model Type	Model Name	Utterance-Level Objective Metrics				Conversation-Level Subjective Metrics
Model Type	Model Name	WER% $\downarrow$	F0-Var $\uparrow$	EmoAlign% $\uparrow$	CE / CU / PC / PQ / Aes. $\uparrow$	Emo. $\uparrow$	Pro. $\uparrow$	Nat. $\uparrow$	Ovr. $\uparrow$
Controllable TTS (w/ Oracle Prompts)	CosyVoice2	3.80	51.20	39.07	6.21 / 7.09 / 1.42 / 7.67 / 5.60	4.15	4.25	4.34	4.23
Controllable TTS (w/ Oracle Prompts)	GPT-4o-mini-TTS	7.63	70.44	45.04	6.22 / 7.37 / 1.41 / 8.00 / 5.75	4.27	4.43	4.58	4.41
Commercial SLM	Doubao	1.12	68.58	42.95	6.21 / 7.38 / 1.53 / 7.94 / 5.72	4.22	4.50	4.59	4.44
Vanilla SLM	Kimi Audio	6.99	46.98	22.64	5.31 / 5.98 / 1.49 / 6.87 / 4.91	3.77	4.02	4.15	3.97
	Step Audio2 Mini	4.87	50.92	22.92	6.12 / 7.06 / 1.44 / 7.68 / 5.58	4.00	4.19	4.43	4.15

Table 2: Comparison of context-aware expressive speech generation. Bold and underline denote the best and second-best results among open-source SLMs , excluding commercial SLMs and controllable TTS systems.

Figure 4: Evaluation accuracy dynamics during UAPO training with different reward models.

Figure 5: t-SNE visualization of emotion clustering across token embeddings (

\mathbf{e}

), LLM hidden states (

\mathbf{h}

), and distilled VIB latent variables (

\mathbf{z}

). Linear-probe accuracies, computed on the original representations rather than the t-SNE projections, are 18.1%, 41.4%, and 64.5%, respectively.

$\mathcal{M}(\mathbf{e}_{y_{i}},\mathbf{h}_{y_{i}})$	$\mathbf{R}_{G}$	WER% $\downarrow$	Emo. $\uparrow$	Pro. $\uparrow$	Nat. $\uparrow$	Ovr. $\uparrow$
(A) Ablation on Modulation Strategy
$\pi_{1}$	Acoustic	3.61	4.06	4.22	4.30	4.17
$\oplus$	Acoustic	8.29	4.04	4.28	4.31	4.20
$\mathrm{AdaLN}$	Acoustic	6.50	4.15	4.28	4.34	4.22
(B) Ablation on Context Grounding
$\mathrm{VIB+AdaLN}$	$\varnothing$	4.19	4.10	4.23	4.33	4.21

Table 3: Ablation study on the modulation strategy

\mathcal{M}(\mathbf{e}_{y_{i}},\mathbf{h}_{y_{i}})

and context grounding

\mathbf{R}_{G}

in Eq. 9 .

Strategy	RM Size	WER(%) $\downarrow$	Emo. $\uparrow$	Pro. $\uparrow$	Nat. $\uparrow$	Ovr. $\uparrow$
SA-SLM(SFT)	—	4.74	4.21	4.34	4.45	4.31
+ Self-Reward	3B	4.56	4.24	4.36	4.49	4.33
+ Oracle-Reward	30B	4.21	4.30	4.41	4.54	4.38

Table 4: Comparison of reward strategies on Intelligibility (WER) and expressive speech quality.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model Name	Audio Understanding				Empathetic Response
Model Name	Sound	Music	Speech	Avg.	CF	RN	Col.	IR
Qwen2.5-Omni-3B	30.24	49.25	39.04	39.51	4.54	4.66	4.51	3.24
Qwen2.5-Omni-7B	78.10	65.90	70.60	71.50	4.59	4.58	4.51	3.28
SASLM-3B (Ours)	29.94	49.25	39.63	39.60	4.56	4.65	4.40	3.24

Appendix

Table 5: Comparison of different speech language models on audio understanding (MMAU) and empathetic response ability (EchoMind). Bold = best; underline = second-best.

Model Type	Architecture Design		Intell.	Expressiveness
Model Type	$\mathcal{M}(\mathbf{e}_{y_{i}},\mathbf{h}_{y_{i}})$	$\mathbf{R}_{G}$	WER(%) $\downarrow$	F0-Var $\uparrow$	Emo. $\uparrow$	Pro. $\uparrow$	Nat. $\uparrow$	Ovr. $\uparrow$
Conversational TTS	$\pi_{1}$	$\varnothing$	3.53	55.88	4.04	4.20	4.30	4.17
		Semantic	9.62	61.16	4.02	4.18	4.28	4.16
		Acoustic	3.61	60.42	4.06	4.22	4.30	4.17
Naive Addition	$\oplus$	$\varnothing$	7.91	57.00	4.06	4.22	4.32	4.19
		Semantic	15.40	57.61	4.11	4.24	4.35	4.21

Appendix

Table 6: Ablation study on expressive-aware speech generation across two independent design choices in Eq. 9 . (1) Intent Modulation Strategy

\mathcal{M}(\mathbf{e}_{y_{i}},\mathbf{h}_{y_{i}})

: how semantic intent is injected for expressive generation, where

\pi_{1}

\mathcal{M}(\mathbf{e}_{y_{i}},\mathbf{h}_{y_{i}})=\mathbf{e}_{y_{i}}

discards SLM semantic intent (conversational TTS-style baseline);

\oplus

: naive element-wise addition

\mathbf{e}_{y_{i}}+\mathbf{h}_{y_{i}}

Xu et al. (2025a) ;

\mathrm{AdaLN}

Wang et al. (2025a) : adaptive modulation without VIB;

\mathrm{VIB{+}AdaLN}

: our proposed disentangled modulation. (2) Context Grounding

\mathbf{R}_{G}

: how context is constructed during synthesis, where

\varnothing

denotes no context grounding, Semantic denotes perception-pathway representations

\mathbf{R}_{G}=\mathbf{E}_{X}

, and Acoustic denotes generation-encoder acoustic features

\mathbf{R}_{G}=\phi_{G}(X^{S})

Figure 6: Pairwise human Win/Tie/Loss between our model and each baseline (GPT-4o-mini-TTS, Qwen3-Omni-30B) on the three perceptual dimensions, together with an aggregated Overall row per baseline. Win (green) indicates that our model is preferred, Loss (red) indicates that the baseline is preferred, and Tie (gray) indicates perceptually comparable.

Figure 7: Human Win/Tie/Loss validation of reward models. Each row corresponds to a (model, dimension) configuration. Win (green) indicates that the RM-high sample is preferred by human raters, Loss (red) indicates that the RM-low sample is preferred, and Tie (gray) indicates that the two samples are judged perceptually comparable.

Panel of Figure 8 — Figure 8: Prompt for evaluating emotion delivery in synthesized speech.

Panel of Figure 9 — Figure 9: Prompt for text-level emotional QA completion. The figure illustrates the “answer-to-question” direction; other directions follow the same template by swapping the observed and generated sides.

Panel of Figure 10 — Figure 10: Prompt for evaluating emotion and intensity in synthesized speech.

Panel of Figure 11 — Figure 11: Annotation guideline shown to human evaluators in both the MOS study and the reward-model validation study.

Explore similar work

Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
Apr 10, 2026Yizhong Geng, Yanliang Li, Jinghan Yang +4TTS Synthesis Speech Language Models
Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA +2
On the Emotion Understanding of Synthesized Speech
Mar 17, 2026Yuan Ge, Haishu Zhao, Aokai Hao +10Speech Emotion Recognition Emotion Representation in Language Models
Northeastern University, China · Meituan · NiuTrans Research +1
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
May 7, 2026Wenqian Cui, Xiao-Hui Li, Daxin Tan +2Audio Understanding Speech Prosody
The Chinese University of Hong Kong · Huawei Technologies

Apr 10, 2026J/K move · Enter open · S save

Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

Yizhong Geng, Yanliang Li, Jinghan Yang, Tianhan Jiang+3

Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA+2

TTS Synthesis Speech Language Models Low-Resource TTS Synthesis Speech Prosody Voice Cloning

Open paper AskPDF arXiv

Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.

Abstract

Explore similar work

Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

On the Emotion Understanding of Synthesized Speech

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM