Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Kirill Borodin; Vasiliy Kudryavtsev; Maxim Maslov; Nikita Vasiliev; Mikhail Gorodnichev; Grach Mkrtchian

cs.SDMar 31, 2026

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Authors: Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov, Nikita Vasiliev, Mikhail Gorodnichev, Grach Mkrtchian

Organizations: MTUCI, Moscow, Russia

Abstract

Text encoders in neural text-to-speech (TTS) see only linguistic content, so prosody must be inferred from paired data and synthesised intonation is often flat. Contrastive text-audio pretraining (CLAPSpeech) injects prosodic knowledge by contrasting acoustic realisations of the same phoneme, but it never trains the encoder to tell different phonemes apart, and its interaction with masked language modelling (MLM) is unexplored. We study a curriculum for a speaker-conditioned phoneme-and-subword encoder: MLM, contrastive alignment over batches of different phonemes, and CLAPSpeech-style same-phoneme refinement. On Russian conversational speech, with each variant conditioning an identical Grad-TTS model, MLM followed by mixed-phoneme alignment gives the lowest word error rate (17.6% versus 32.8% for same-phoneme training alone) and the best quality and intonation ratings. Adding same-phoneme refinement improves same-phoneme retrieval yet yields the worst synthesis: embedding-space retrieval does not predict usefulness for generation.

Figures & tables

Figure 1: Prosody-aware text encoder. The BPE and phoneme streams are embedded and encoded separately; the projected speaker embedding is added to the token embeddings and modulates every transformer block through AdaLN-Zero (dashed). BPE states are averaged within each word and repeated for every phoneme of that word, summed with the phoneme states, and passed through a shared encoder to give one embedding per phoneme.

Figure 2: Text–audio similarity matrices of the two batch types (four utterances shown). Subscripts denote occurrences of the same phoneme in different utterances.

Stages	R@1-sim	R@5-sim	R@1-diff	R@5-diff
3 †	0.692	0.903	0.779	0.919
1+3	0.665	0.890	0.766	0.922
2	0.690	0.911	0.933	0.986
1+2	0.746	0.937	0.926	0.983
2+3	0.782	0.946	0.871	0.958
1+2+3	0.770	0.935	0.893	0.974

Table 1: Text–audio retrieval of encoder variants (batch of 1024). -sim: same-phoneme batches (prosodic sensitivity); -diff: mixed-phoneme batches (phoneme discrimination). † CLAPSpeech-style same-phoneme training only. Best in bold, second best underlined.

Stages	R@1-sim	R@1-diff	IntMOS	MOS	UTMOS	NISQA	WER (%)	CER (%)	SIM-o
	Encoder retrieval		Listening tests		Predicted quality		Intelligibility, speaker
3 †	0.692	0.779	$1.615\pm 0.285$	$1.605\pm 0.165$	1.43	2.28	32.8	20.8	0.798
1+3	0.665	0.766	$1.855\pm 0.270$	$1.755\pm 0.160$	1.64	2.49	31.0	15.5	0.838
2	0.690	0.933	$1.890\pm 0.280$	$\underline{1.895\pm 0.190}$	1.65	2.72	29.0	14.2	0.854
1+2	0.746	0.926	$\mathbf{2.185\pm 0.255}$	$\mathbf{1.980\pm 0.172}$	1.62	2.80	17.6	6.7	0.862
2+3	0.782	0.871	$\underline{2.126\pm 0.278}$	$1.840\pm 0.160$	1.66	2.73	23.4	12.3	0.843

Table 2: Grad-TTS conditioned on each encoder variant. R@1 values are repeated from Table 1 to show that retrieval does not predict synthesis. MOS and IntMOS: mean

\pm

95% confidence interval on the anchored 0–5 scales of Sec. 3 . † CLAPSpeech-style same-phoneme training only. Best in bold, second best underlined.

Figure 3: WER of Grad-TTS against the retrieval accuracy of the encoder that conditions it; labels give the stages used ( † CLAPSpeech-style). Neither retrieval measure ranks the systems.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity
Jun 13, 2026Zhenwei Mou, Liping Chen, Yajun Hu +3TTS Synthesis Speech Prosody
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China
LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
Mar 27, 2026Xiaoyu Fan, Huizhi Xie, Wei Zou +1TTS Synthesis Zero-Shot TTS
BRVoice Team, Bairong, Inc., China
Revise-TTS: A Non-Masking Refinement Framework for Masked Diffusion Models in Text-to-Speech
Oct 14, 2026Wanli Sun, Dan Su, Anton RagniTTS Synthesis Zero-Shot TTS
School of Computer Science, University of Sheffield, Sheffield, UK · HojoAI, Beijing, China

Jun 13, 2026J/K move · Enter open · S save

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

Zhenwei Mou, Liping Chen, Yajun Hu, Zhen-Hua Ling+2

University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China

TTS Synthesis Speech Prosody Voice Cloning

Open paper AskPDF arXiv

Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.

Abstract

TTS Synthesis

Explore similar work

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling

Revise-TTS: A Non-Masking Refinement Framework for Masked Diffusion Models in Text-to-Speech