Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
Authors: Yizhong Geng, Yanliang Li, Jinghan Yang, Tianhan Jiang, Boxun An, Ya Li, Xiaoyu Shen
Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Beijing Logic Intelligence Technology, Beijing, China · University of California, USA · Northwestern University, USA · Eastern Institute of Technology, Ningbo, China
Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.
Figures & tables
Figure 1: Scaling behavior of objective metrics as synthetic data ratio α increases. WER decreases monotonically, indicating improved stability. In contrast, token entropy Hp , repetition rate, NMOS, and SMOS exhibit non-monotonic trends—peaking around α≈50% before degrading, revealing the Synthetic Erosion phenomenon. Notably, Hp tracks NMOS, supporting its use as a lightweight proxy for prosodic diversity.
Figure 2: Disentanglement-Guided Self-Alignment (DGSA). Flow-Matching SLMs separate prosody (Text-Speech LM) from timbre (Flow-Matching Transformer). Enabling the style token produces expressive output yexpr ; disabling it yields stable but flat output ystab . DGSA aligns both toward real speech yreal via dual preference objectives.
Figure 3: Temperature-Driven Self-Critique (TDSC) For each input, the model generates candidates at temperatures T∈{low,mid,high} , spanning conservative (stable) to exploratory (expressive) outputs. The Judge Model filters candidates by WER, length, and repetition criteria, yielding accepted ( G ) and rejected ( R ) sets for preference-based refinement.
Syn.
α
WER ↓
Hp
Rep. ↓
NMOS ↑
SMOS ↑
(hours)
(%)
(%)
(bits)
(%)
(Mean ± CI)
(Mean ± CI)
10
3
75.0
10.419
2.96
3.82±0.09
4.31±0.08
30
9
68.5
10.477
2.83
4.01±0.08
4.52±0.07
53
15
58.2
10.455
2.62
4.08±0.08
4.54±0.06
100
25
52.0
10.423
2.48
4.13±0.07
4.55±0.06
200
40
49.2
10.498
2.21
4.38±0.07
4.60±0.06
Table 1: Scaling behavior across synthetic data ratios. Best expressivity metrics ( Hp , NMOS, SMOS) occur at α≈50% . The α=100% row (pure synthetic, 0h real) confirms severe Synthetic Erosion. Subjective metrics are reported with 95% CI.
Figure 4: Stability-Expressivity trade-off space. Each point represents a model trained with different synthetic data ratios. Lower WER (rightward) indicates better stability; higher Hp (upward) indicates better expressivity. The 300h configuration achieves the best balance, while excessive synthetic data (1200h, 1500h) sacrifices expressivity for marginal stability gains.
Metric
High Hp
Low Hp
Δ
WER (%) ↓
38.2
38.8
–0.6 (n.s.)
F0 Std. (Hz) ↑
42.6
35.8
+6.8
F0 Range (Hz) ↑
128.4
109.7
+18.7
F0 Corr. ↑
0.68
0.52
+0.16
F0 RMSE (Hz) ↓
28.3
39.1
–10.8
Energy Std. (dB) ↑
7.4
6.1
+1.3
Table 2: Controlled comparison of high vs. low Hp samples paired by identical text and matched WER. Improvements in acoustic features and perceived expressivity ( p<0.01 ) validate Hp as a prosodic indicator.
Method
WER ↓
Hp↑
Rep. ↓
NMOS ↑
SMOS ↑
SFT Baseline
38.9
10.36
6.51
3.61±0.09
3.54±0.10
Standard DPO
45.2
10.49
4.08
3.92±0.08
3.81±0.09
Rejection Sampling
40.5
10.41
5.18
3.75±0.08
3.66±0.09
DGSA (Ours)
38.9
10.52
2.82
4.42±0.07
4.53±0.06
Table 3: Alignment methods comparison at α=80% . DGSA simultaneously achieves high expressivity and stability. Intervals denote 95% CI.
Figure 5: Dynamic weight scheduling and Hp recovery. Below α∗=50% , λe=0 (no correction needed). Beyond α∗ , λe activates and ΔHp scales proportionally.
Figure 6: TDSC iteration dynamics over 5 refinement rounds. WER decreases steadily while Hp rises in later iterations as Tmax expands. Pass rate increases from 23% to 62%, indicating progressive quality improvement.
Method
WER ↓
Hp↑
Rep. ↓
NMOS ↑
(%)
(bits)
(%)
(Mean ± CI)
SFT Baseline
38.5
10.08
7.62
3.12±0.09
Self-Training
35.2
10.15
6.94
3.31±0.08
Rejection Sampling
36.8
10.11
7.28
3.24±0.09
TDSC (Ours)
29.8
10.42
4.15
3.94±0.07
Table 4: TDSC evaluation on Lao. TDSC significantly outperforms alternative self-improvement methods ( p<0.05 ).
Language
Method
WER ↓ (%)
NMOS ↑ (Mean ± CI)
Thai
Open-Source
PyThaiTTS
78.4
2.91±0.10
MMS-TTS
53.5
3.24±0.09
Typhoon2-Audio
50.9
3.72±0.08
Seamless-M4T-v2
47.8
3.55±0.09
Commercial
Table 5: Standard TTS comparison on Thai and Lao. Our system achieves the best expressivity (NMOS) with statistical significance ( p<0.05 ) against baselines.
Lang.
Method
WER ↓
SIM ↑
NMOS ↑
SMOS ↑
(%)
(0-1)
(Mean ± CI)
(Mean ± CI)
Thai
ElevenLabs-v3
42.3
0.78
4.21±0.07
4.23±0.07
Ours (DGSA)
38.9
0.84
4.42±0.06
4.51±0.06
Lao
Other Systems
Not Supported
Ours (TDSC)
29.8
0.81
3.94±0.07
4.32±0.06
Table 6: Zero-shot voice cloning comparison. Our method outperforms baselines in speaker similarity and naturalness.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
WER ↓ (%)
Hp↑
NMOS ↑
Δ NMOS
Full DGSA
38.9
10.52
4.4
—
w/o Expressivity Obj.
38.2
10.38
3.7
− 0.7
w/o Identity Pairs
42.8
10.41
3.9
− 0.5
w/o Stability Obj.
46.5
10.55
4.0
− 0.4
w/o Dynamic Scaling
40.2
10.45
4.2
− 0.2
Appendix
Table 7: Ablation study of DGSA components at α=80% . Each row removes one component from the full system. All components contribute to final performance; removing the Expressivity Objective causes the largest degradation ( Δ NMOS = − 0.7).
Variant
WER ↓ (%)
Hp↑ (bits)
NMOS ↑
Δ NMOS
Full TDSC
29.8
10.42
3.9
—
w/o DPO Loss
35.8
10.21
3.4
− 0.5
w/o Multi-Temperature
34.1
10.05
3.5
− 0.4
w/o Length Filter
33.5
10.35
3.5
− 0.4
w/o Temp. Curriculum
32.6
10.18
3.6
− 0.3
w/o Repetition Filter
31.2
10.28
3.6
− 0.3
Appendix
Table 8: Ablation study of TDSC components on Lao. Each row removes one component from the full system. The DPO loss contributes most significantly ( Δ NMOS = − 0.5).
Temp. T
WER ↓ (%)
Hp↑ (bits)
Rep. ↓ (%)
Pass (%)
Contrib.
0.7
26.8
9.85
3.42
78.5
42%
1.0
32.4
10.38
4.65
61.2
35%
1.3
41.6
10.82
6.18
38.4
23%
Filtered G
29.8
10.42
4.15
—
100%
Appendix
Table 9: Candidate quality across temperatures at iteration k=5 . Each temperature regime contributes complementary samples to the filtered set G .
Component
Specification
Parameters
Speech Tokenizer
S3Tokenizer with FSQ, 25Hz
— (frozen)
Codebook size: 6,561
Text-Speech LM
Qwen2.5-0.5B backbone
500M
24 layers, 896 hidden, 14 heads
Flow-Matching
CFM with causal attention
300M
12 layers, 512 hidden
Appendix
Table 10: Model configuration details.
Hyperparameter
SFT
DGSA
TDSC
Learning Rate
1×10−5
1×10−6
1×10−5
Total Steps / Iterations
38k
10k
5 iter.
Batch Size
Dynamic (max 2,000 frames/GPU)
Optimizer
AdamW ( β1=0.9,β2=0.999 )
Precision
bfloat16
Hardware
8 × NVIDIA RTX 4090
Appendix
Table 11: General training hyperparameters across different stages.
Category
Hyperparameter
Value
Filtering Criteria
WER Threshold ( τw )
<40%
Repetition Threshold ( τr )
<10%
Min Length Ratio ( γmin )
0.5×∣x∣
Max Length Ratio ( γmax )
2.0×∣x∣
Sampling Strategy
Temperature Set ( T )
{0.7,1.0,Tmax(k)}
Candidates per Input
N=12 (4 per temp)
Appendix
Table 12: Detailed filtering thresholds and sampling configurations for the TDSC pipeline. These specific constraints ensure stable self-improvement.
Metric
Thai
Lao
ASR (for WER)
Whisper-large-v3
Dolphin-small
Speaker Similarity
WavLM-Large (layer 12, cosine)
MOS Evaluators
20 native speakers
20 native speakers
MOS Samples
200 samples per rater (8,000 total scores)
Appendix
Table 13: Evaluation setup and tools.
Language
Source
Hours
License
Thai
Real (Common Voice + TSynC-2)
300h
CC-0
Synthetic (MMS / Seamless / Typhoon2)
1,200h
Various OSS
Lao
Synthetic (MMS-TTS)
1,500h
CC-BY-NC
Appendix
Table 14: Final dataset statistics after quality filtering.
Speech Language Models (SLMs) exhibit strong semantic understanding, yet often fail to translate this capacity into expressive acoustic realization, producing speech with flattened prosody and misaligned emotion. We identify this mismatch as the semantic understanding-acoustic realization gap. Existing approaches typically rely on externally specified proxies, such as emotion labels or style prompts, which require annotations and struggle to capture dynamically evolving expressive intent throughout dialogue. To overcome these limitations, we propose SASLM (Self-Aware Speech Language Model), a proxy-free framework that bridges what the model thinks and how it speaks through self-aware intent-realization alignment: (1) Intent-Aware Bridging self-distills expressive intent from the model's own evolving semantic generation states via a Variational Information Bottleneck (VIB), thereby guiding expressive speech realization without external expressive supervision; while (2) Realization-Aware Alignment reflectively aligns generated acoustics with intended expression through self-reward optimization, progressively improving intent-realization consistency during speech generation. Despite using only 3B parameters and 800 hours of expressive speech data, SASLM achieves state-of-the-art performance on EchoMind among open-source systems, surpassing models over 10 times larger and approaching commercial systems.
Kuang Wang, Lai Wei, Ping Lin +8
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.
Wenqian Cui, Xiao-Hui Li, Daxin Tan +2
The Chinese University of Hong Kong · Huawei Technologies
Low-resource text-to-speech (TTS) adaptation is constrained by scarce paired data and costly manual transcription. Existing fixed-voice TTS systems can provide relatively accurate pronunciation, but their synthetic speech offers limited speaker diversity and may exhibit flat prosody. Real recordings provide natural prosody and diverse voices, yet their automatic speech recognition (ASR) pseudo-labels may contain transcription errors. We find that supervision order affects content accuracy and speaker similarity. We propose trust-aware progressive adaptation: synthetic-to-real adaptation first establishes text-speech correspondences, then restores reference-speaker control using real speech. Transcript-agreement weighting uses agreement between two fixed ASR systems as a proxy for pseudo-label reliability to limit noisy supervision. Experiments with FireRedTTS3 on Burmese and Lao and OmniVoice on Burmese show improved content accuracy with high naturalness and competitive speaker similarity. Jointly considering supervision order and pseudo-label reliability when combining synthetic and real speech offers a practical path to zero-shot voice cloning in low-resource languages with less manual transcription. Audio demos are available at https://insiderx-pro.github.io/S2R-Adaptation-TTS/
Jiayi Lu, Yizhong Geng, Jinghan Yang +4
Beijing Logic Intelligence Technology · University of Washington · Beijing University of Posts and Telecommunications +2