SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS

Walaa Othman; Baher Mohammad; Stamatios Lefkimmiatis; Ammar Ali

cs.LGOct 19, 2026

SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS

Authors: Walaa Othman, Baher Mohammad, Stamatios Lefkimmiatis, Ammar Ali

Organizations: ITMO University · SPC RAS · MWS AI

Abstract

Expanding the vocabulary of conditional text-to-speech (TTS) models typically requires extensive fine-tuning with paired audio, creating a major bottleneck for low-resource and cross-lingual adaptation. To address this, we introduce SEAM, a zero-audio initialization method that constructs embeddings for unseen tokens from known subtokens. Unlike standard pooling strategies that ignore sequential structure, SEAM applies a spectral filter to preserve temporal phonetic order and uses closed-form ridge regression to align the new embeddings with the pretrained conditioning manifold. Across three languages and two TTS architectures, SEAM approaches or matches baseline performance using up to four times less training data, reducing sample complexity by 75% compared to random initialization and mean pooling. Our approach provides a lightweight, reproducible, and theoretically grounded solution for vocabulary expansion in low-resource speech synthesis.

Figures & tables

Language	Budget	Init	CER	WER
Russian	5 h	Random	20.8	42.49
		Mean	17.53	36.73
		SEAM	14.88	34.41
	25 h	Random	17.66	27.18
		Mean	10.72	24.58
		SEAM	8.50	20.01

Table 1: CER (%) after training on fixed data budgets. Lower is better. SEAM achieves comparable performance to baselines with up to 4

\times

less data for some languages.

Panel of Figure 1 — Figure 1: Side-by-Side human preference results: SEAM-initialized model (25h training) vs. mean-pooling baseline (100h training). Green indicates preference for SEAM despite using 4 $\times$ less data. Results are shown for Russian, French, and Kazakh.

Figure 2: CER versus vocabulary expansion size. SEAM maintains robust performance as vocabulary size increases, while mean and random initialization degrade

Initialization Method	CER (%)	WER (%)
Random	20.80	42.49
Mean Pooling	17.53	36.73
Position-Aware Pooling	23.37	54.70
Attention Pooling	20.14	50.03
Learned Attention	15.30	42.86
SEAM (Ours)	14.88	34.41

Table 2: Extended baseline comparison on Russian (5h fine-tuning). Lower is better.

Method	CER (%)	WER (%)
SEAM (Original)	14.88	34.41
SEAM (Reversed)	19.67	51.99
SEAM (Shuffled)	22.91	53.66

Table 3: Order sensitivity ablation on Russian (5h).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Spectral energy decay in subtoken embedding sequences. Cumulative energy (blue) exceeds 99% by

k=3

, confirming bandlimited compositionality. The Gaussian filter

\sigma=2.0

(green) suppresses high-frequency noise while retaining bins

k\in\{0,1,2,3\}

(red dashed line). This empirically validates

(K=4,\sigma=2.0)

without hyperparameter tuning.

Language	CER (%)	WER (%)
Russian	4.88	14.68
French	0.63	3.30
Kazakh	0.98	5.68

Appendix

Table 4: ASR baseline performance on original human-recorded audio. Lower is better. These values define the error floor for TTS evaluation.

Language	Initialization	Hours	CER (%)
Russian	Random	5	9.26
Russian	Average	5	8.34
Russian	SEAM (Ours)	5	6.14

Appendix

Table 5: CER (%) for VibeVoice fine-tuned on Russian (5 hours) under different embedding initialization strategies. Lower is better.

Variant	Initial Loss	Final Loss
SEAM (Zero-Audio)	10.2	$\sim$ 4.0
Audio-Guided	7.2	$\sim$ 10.0

Appendix

Table 6: Training loss dynamics: Zero-audio SEAM vs. Audio-Guided variant (Russian, 5h). The audio-guided variant exhibits severe optimization instability.

Lang.	SEAM % (Non-Tie)	95% CI	$p$ -val
Russian	61.4%	[50.4, 71.4]	0.041
Kazakh	65.1%	[55.6, 73.6]	0.002
French	70.6%	[60.2, 79.2]	<.001

Appendix

Table 7: SBS preference statistics (

N=125

per language; 5 annotators

\times

25 samples). “SEAM % (Non-Tie)” represents the preference rate excluding ties. 95% Wilson confidence intervals and exact binomial

p

-values (against

p=0.5

) demonstrate statistical significance across all evaluated languages.

Text-to-Speech

Text-to-speech converts written language into spoken audio, improving intelligibility, naturalness, voice control, multilingual coverage, and efficiency.

See all 67 papers

SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations
Jun 10, 2026Peijie Chen, Wenhao Guan, Weijie Wu +7Variational Autoencoders Audio Representation Learning
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis
Mar 4, 2026Youngwon Choi, Jinwoo Oh, Hwayeon Kim +1Personalized Speech Recognition TTS Synthesis
Maum AI Inc., Republic of Korea · Humelo Inc., Republic of Korea
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Test-Time Adaptation Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Tencent

Jun 10, 2026J/K move · Enter open · S save

SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations

Peijie Chen, Wenhao Guan, Weijie Wu, Kaidi Wang+6

School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China

Variational Autoencoders Audio Representation Learning Speech Processing Zero-Shot TTS Text-to-Speech

Open paper AskPDF arXiv

Zero-shot text-to-speech (TTS) relies on robust speech representations. However, current speech tokenizers face a fundamental trade-off: acoustic codecs preserve high-fidelity audio but lack linguistic constraints, causing content errors during generation, whereas semantic tokens from self-supervised learning (SSL) models ensure precise text alignment but discard some acoustic information. To bridge this gap, we propose SARA, a dual-stream VAE that directly fuses a frozen SSL semantic anchor with a dedicated residual acoustic encoder. This effectively mitigates the dilemma, creating an efficient and compact latent space without relying on complex regularizers. SARA achieves superior reconstruction quality over strong baselines. Furthermore, in downstream zero-shot TTS tasks, it yields highly natural and expressive synthesis quality, and maintains robust generation performance even under accelerated inference, offering a favorable trade-off between synthesis speed and computational cost.

Abstract

Text-to-Speech

Explore similar work

SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations

ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation