Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

Yizhong Geng; Yanliang Li; Jinghan Yang; Tianhan Jiang; Boxun An; Ya Li; Xiaoyu Shen

cs.CLApr 10, 2026

Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

Authors: Yizhong Geng, Yanliang Li, Jinghan Yang, Tianhan Jiang, Boxun An, Ya Li, Xiaoyu Shen

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA · Northwestern University, USA · Eastern Institute of Technology, Ningbo, China

Abstract

Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.

Figures & tables

Figure 1: Scaling behavior of objective metrics as synthetic data ratio $\alpha$ increases. WER decreases monotonically, indicating improved stability. In contrast, token entropy $H_{p}$ , repetition rate, NMOS, and SMOS exhibit non-monotonic trends—peaking around $\alpha\approx 50\%$ before degrading, revealing the Synthetic Erosion phenomenon. Notably, $H_{p}$ tracks NMOS, supporting its use as a lightweight proxy for prosodic diversity.

Figure 2: Disentanglement-Guided Self-Alignment (DGSA). Flow-Matching SLMs separate prosody (Text-Speech LM) from timbre (Flow-Matching Transformer). Enabling the style token produces expressive output $y^{\text{expr}}$ ; disabling it yields stable but flat output $y^{\text{stab}}$ . DGSA aligns both toward real speech $y^{\text{real}}$ via dual preference objectives.

Figure 3: Temperature-Driven Self-Critique (TDSC) For each input, the model generates candidates at temperatures $T\in\{low,mid,high\}$ , spanning conservative (stable) to exploratory (expressive) outputs. The Judge Model filters candidates by WER, length, and repetition criteria, yielding accepted ( $\mathcal{G}$ ) and rejected ( $\mathcal{R}$ ) sets for preference-based refinement.

Syn.	$\alpha$	WER $\downarrow$	$H_{p}$	Rep. $\downarrow$	NMOS $\uparrow$	SMOS $\uparrow$
(hours)	(%)	(%)	(bits)	(%)	(Mean $\pm$ CI)	(Mean $\pm$ CI)
10	3	75.0	10.419	2.96	$3.82\pm 0.09$	$4.31\pm 0.08$
30	9	68.5	10.477	2.83	$4.01\pm 0.08$	$4.52\pm 0.07$
53	15	58.2	10.455	2.62	$4.08\pm 0.08$	$4.54\pm 0.06$
100	25	52.0	10.423	2.48	$4.13\pm 0.07$	$4.55\pm 0.06$
200	40	49.2	10.498	2.21	$4.38\pm 0.07$	$4.60\pm 0.06$

Table 1: Scaling behavior across synthetic data ratios. Best expressivity metrics (

H_{p}

, NMOS, SMOS) occur at

\alpha\approx 50\%

. The

\alpha=100\%

row (pure synthetic, 0h real) confirms severe Synthetic Erosion. Subjective metrics are reported with 95% CI.

Figure 4: Stability-Expressivity trade-off space. Each point represents a model trained with different synthetic data ratios. Lower WER (rightward) indicates better stability; higher $H_{p}$ (upward) indicates better expressivity. The 300h configuration achieves the best balance, while excessive synthetic data (1200h, 1500h) sacrifices expressivity for marginal stability gains.

Metric	High $H_{p}$	Low $H_{p}$	$\Delta$
WER (%) $\downarrow$	38.2	38.8	–0.6 (n.s.)
F0 Std. (Hz) $\uparrow$	42.6	35.8	+6.8
F0 Range (Hz) $\uparrow$	128.4	109.7	+18.7
F0 Corr. $\uparrow$	0.68	0.52	+0.16
F0 RMSE (Hz) $\downarrow$	28.3	39.1	–10.8
Energy Std. (dB) $\uparrow$	7.4	6.1	+1.3

Table 2: Controlled comparison of high vs. low

H_{p}

samples paired by identical text and matched WER. Improvements in acoustic features and perceived expressivity (

p<0.01

) validate

H_{p}

as a prosodic indicator.

Method	WER $\downarrow$	$H_{p}$ $\uparrow$	Rep. $\downarrow$	NMOS $\uparrow$	SMOS $\uparrow$
SFT Baseline	38.9	10.36	6.51	$3.61\pm 0.09$	$3.54\pm 0.10$
Standard DPO	45.2	10.49	4.08	$3.92\pm 0.08$	$3.81\pm 0.09$
Rejection Sampling	40.5	10.41	5.18	$3.75\pm 0.08$	$3.66\pm 0.09$
DGSA (Ours)	38.9	10.52	2.82	$\mathbf{4.42\pm 0.07}$	$\mathbf{4.53\pm 0.06}$

Table 3: Alignment methods comparison at

\alpha=80\%

. DGSA simultaneously achieves high expressivity and stability. Intervals denote 95% CI.

Figure 5: Dynamic weight scheduling and $H_{p}$ recovery. Below $\alpha^{*}=50\%$ , $\lambda_{e}=0$ (no correction needed). Beyond $\alpha^{*}$ , $\lambda_{e}$ activates and $\Delta H_{p}$ scales proportionally.

Figure 6: TDSC iteration dynamics over 5 refinement rounds. WER decreases steadily while $H_{p}$ rises in later iterations as $T_{\max}$ expands. Pass rate increases from 23% to 62%, indicating progressive quality improvement.

Method	WER $\downarrow$	$H_{p}$ $\uparrow$	Rep. $\downarrow$	NMOS $\uparrow$
	(%)	(bits)	(%)	(Mean $\pm$ CI)
SFT Baseline	38.5	10.08	7.62	$3.12\pm 0.09$
Self-Training	35.2	10.15	6.94	$3.31\pm 0.08$
Rejection Sampling	36.8	10.11	7.28	$3.24\pm 0.09$
TDSC (Ours)	29.8	10.42	4.15	$\mathbf{3.94\pm 0.07}$

Table 4: TDSC evaluation on Lao. TDSC significantly outperforms alternative self-improvement methods (

p<0.05

Language	Method	WER $\downarrow$ (%)	NMOS $\uparrow$ (Mean $\pm$ CI)
Thai	Open-Source
	PyThaiTTS	78.4	$2.91\pm 0.10$
	MMS-TTS	53.5	$3.24\pm 0.09$
	Typhoon2-Audio	50.9	$3.72\pm 0.08$
	Seamless-M4T-v2	47.8	$3.55\pm 0.09$
	Commercial

Table 5: Standard TTS comparison on Thai and Lao. Our system achieves the best expressivity (NMOS) with statistical significance (

p<0.05

) against baselines.

Lang.	Method	WER $\downarrow$	SIM $\uparrow$	NMOS $\uparrow$	SMOS $\uparrow$
		(%)	(0-1)	(Mean $\pm$ CI)	(Mean $\pm$ CI)
Thai	ElevenLabs-v3	42.3	0.78	$4.21\pm 0.07$	$4.23\pm 0.07$
Thai	Ours (DGSA)	38.9	0.84	$\mathbf{4.42\pm 0.06}$	$\mathbf{4.51\pm 0.06}$
Lao	Other Systems	Not Supported
Lao	Ours (TDSC)	29.8	0.81	$\mathbf{3.94\pm 0.07}$	$\mathbf{4.32\pm 0.06}$

Table 6: Zero-shot voice cloning comparison. Our method outperforms baselines in speaker similarity and naturalness.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	WER $\downarrow$ (%)	$H_{p}$ $\uparrow$	NMOS $\uparrow$	$\Delta$ NMOS
Full DGSA	38.9	10.52	4.4	—
w/o Expressivity Obj.	38.2	10.38	3.7	$-$ 0.7
w/o Identity Pairs	42.8	10.41	3.9	$-$ 0.5
w/o Stability Obj.	46.5	10.55	4.0	$-$ 0.4
w/o Dynamic Scaling	40.2	10.45	4.2	$-$ 0.2

Appendix

Table 7: Ablation study of DGSA components at

\alpha=80\%

. Each row removes one component from the full system. All components contribute to final performance; removing the Expressivity Objective causes the largest degradation (

\Delta

NMOS =

-

0.7).

Variant	WER $\downarrow$ (%)	$H_{p}$ $\uparrow$ (bits)	NMOS $\uparrow$	$\Delta$ NMOS
Full TDSC	29.8	10.42	3.9	—
w/o DPO Loss	35.8	10.21	3.4	$-$ 0.5
w/o Multi-Temperature	34.1	10.05	3.5	$-$ 0.4
w/o Length Filter	33.5	10.35	3.5	$-$ 0.4
w/o Temp. Curriculum	32.6	10.18	3.6	$-$ 0.3
w/o Repetition Filter	31.2	10.28	3.6	$-$ 0.3

Appendix

Table 8: Ablation study of TDSC components on Lao. Each row removes one component from the full system. The DPO loss contributes most significantly (

\Delta

NMOS =

-

0.5).

Temp. $T$	WER $\downarrow$ (%)	$H_{p}$ $\uparrow$ (bits)	Rep. $\downarrow$ (%)	Pass (%)	Contrib.
0.7	26.8	9.85	3.42	78.5	42%
1.0	32.4	10.38	4.65	61.2	35%
1.3	41.6	10.82	6.18	38.4	23%
Filtered $\mathcal{G}$	29.8	10.42	4.15	—	100%

Appendix

Table 9: Candidate quality across temperatures at iteration

k=5

. Each temperature regime contributes complementary samples to the filtered set

\mathcal{G}

Component	Specification	Parameters
Speech Tokenizer	S3Tokenizer with FSQ, 25Hz	— (frozen)
	Codebook size: 6,561
Text-Speech LM	Qwen2.5-0.5B backbone	500M
	24 layers, 896 hidden, 14 heads
Flow-Matching	CFM with causal attention	300M
	12 layers, 512 hidden

Appendix

Table 10: Model configuration details.

Hyperparameter	SFT	DGSA	TDSC
Learning Rate	$1\times 10^{-5}$	$1\times 10^{-6}$	$1\times 10^{-5}$
Total Steps / Iterations	38k	10k	5 iter.
Batch Size	Dynamic (max 2,000 frames/GPU)
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.999$ )
Precision	bfloat16
Hardware	8 $\times$ NVIDIA RTX 4090

Appendix

Table 11: General training hyperparameters across different stages.

Category	Hyperparameter	Value
Filtering Criteria	WER Threshold ( $\tau_{w}$ )	$<40\%$
	Repetition Threshold ( $\tau_{r}$ )	$<10\%$
	Min Length Ratio ( $\gamma_{\min}$ )	$0.5\times\|x\|$
	Max Length Ratio ( $\gamma_{\max}$ )	$2.0\times\|x\|$
Sampling Strategy	Temperature Set ( $\mathcal{T}$ )	$\{0.7,1.0,T_{\max}^{(k)}\}$
	Candidates per Input	$N=12$ (4 per temp)

Appendix

Table 12: Detailed filtering thresholds and sampling configurations for the TDSC pipeline. These specific constraints ensure stable self-improvement.

Metric	Thai	Lao
ASR (for WER)	Whisper-large-v3	Dolphin-small
Speaker Similarity	WavLM-Large (layer 12, cosine)
MOS Evaluators	20 native speakers	20 native speakers
MOS Samples	200 samples per rater (8,000 total scores)

Appendix

Table 13: Evaluation setup and tools.

Language	Source	Hours	License
Thai	Real (Common Voice + TSynC-2)	300h	CC-0
Thai	Synthetic (MMS / Seamless / Typhoon2)	1,200h	Various OSS
Lao	Synthetic (MMS-TTS)	1,500h	CC-BY-NC

Appendix

Table 14: Final dataset statistics after quality filtering.

Explore similar work

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment
Apr 13, 2026Kuang Wang, Lai Wei, Ping Lin +8Speech Language Models Text-to-Speech
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
May 7, 2026Wenqian Cui, Xiao-Hui Li, Daxin Tan +2Audio Understanding Speech Prosody
The Chinese University of Hong Kong · Huawei Technologies
From Reliable Text to Real Voices: Trust-Aware Progressive Adaptation for Low-Resource TTS
Sep 22, 2026Jiayi Lu, Yizhong Geng, Jinghan Yang +4Synthetic-to-Real Domain Adaptation Low-Resource TTS Synthesis
Beijing Logic Intelligence Technology · University of Washington · Beijing University of Posts and Telecommunications +2

Apr 13, 2026J/K move · Enter open · S save

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

Kuang Wang, Lai Wei, Ping Lin, Qibing Bai+7

The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University+2

Speech Language Models Text-to-Speech Controllable Speech Generation Emotional Speech Synthesis

Open paper AskPDF arXiv

Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.

Abstract

Explore similar work

Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

From Reliable Text to Real Voices: Trust-Aware Progressive Adaptation for Low-Resource TTS