Probing Human Articulatory Constraints in End-to-End TTS with Reverse and Mismatched Speech-Text Directions

Parth Khadse; Sunil Kumar Kopparapu

cs.SDFeb 16, 2026

Probing Human Articulatory Constraints in End-to-End TTS with Reverse and Mismatched Speech-Text Directions

Authors: Parth Khadse, Sunil Kumar Kopparapu

Organizations: TCS Research, Tata Consultancy Services Limited, Mumbai, India

Abstract

An end-to-end (e2e) text-to-speech (TTS) system is a deep architecture that learns to associate a text string with acoustic speech patterns from a curated dataset. It is expected that all aspects associated with speech production, such as phone duration, speaker characteristics, and intonation among other things are captured in the trained TTS model to enable the synthesized speech to be natural and intelligible. Human speech is complex, involving smooth transitions between articulatory configurations (ACs). Due to anatomical constraints, some ACs are challenging to mimic or transition between. In this paper, we experimentally study if the constraints imposed by human anatomy have an implication on training an e2e-TTS systems. We experiment with two e2e-TTS architectures, namely, Tacotron-2 an autoregressive model and VITS-TTS a non-autoregressive model. In this study, we build TTS systems using (a) forward text, forward speech (conventional, e2e-TTS), (b) reverse text, reverse speech (r-e2e-TTS), and (c) reverse text, forward speech (rtfs-e2e-TTS). Experiments demonstrate that e2e-TTS systems are purely data-driven. Interestingly, the generated speech by r-e2e-TTS systems exhibits better fidelity, better perceptual intelligibility, and better naturalness

Figures & tables

Figure 1: High level block diagram of e2e-TTS architecture. We train the seq2seq block while using a pre-trained vocoder.

Panel of Figure 2 — Figure 2: Alignment plots for both e2e-TTS and r-e2e-TTS show no observable differences.

Model	Generated (#)	EoS issues (#)
e2e-TTS	1715	10
r-e2e-TTS	1715	4

Table 1: Synthesized audio seq2seq EoS problem.

	Model	WER (%)	CER (%)
Whisper ( small.en )	e2e-TTS	17.3	7.5
Whisper ( small.en )	r-e2e-TTS	10.8	4.0
WhisperX ( small )	e2e-TTS	16.3	6.9
WhisperX ( small )	r-e2e-TTS	10.7	3.9

Table 2: WER and CER (lower values better) calculated on

1701

text samples, using Tacotron-2 architecture.

Score	Naturalness	Intelligibility
1	awkward, robotic	cannot understand speech
2	strange/ not smooth	understandable; mostly unclear
3	like human voice	understandable; hard to follow
4	smooth and natural	easy to understand
5	sounds very natural	completely understandable

Table 3: Reference provided to subjects to mark naturalness and intelligibility.

	e2e-TTS	r-e2e-TTS
Naturalness	$3.40\pm 0.11$	3.75 $\pm$ 0.22
Intelligibility	$3.38\pm 0.27$	3.58 $\pm$ 0.60

Table 4: Mean Opinion Scores (Naturalness and Intelligibility).

	Sub# 1	Sub# 2	Sub# 3	Sub# 4	Sub# 5
Pair# 1	$0$	$1$	$1$	$1$	$1$
Pair# 2	$1$	$1$	$1$	$1$	$1$
Pair# 3	$1$	$0$	$1$	$1$	$1$
Pair# 4	$1$	$1$	$1$	$1$	$1$
Pair# 5	$1$	$1$	$1$	$1$	$1$

Table 5: Paired audio perceptual test. A

1

represents the instance when r-e2e-TTS generated audio was chosen as being better.

Figure 3: Scatter plot of duration of the audio for the same text produced by e2e-TTS (y-axis) and r-e2e-TTS (x-axis). The duration of audio generated by e2e-TTS is overall longer than the audio generated by r-e2e-TTS .

Tokenizer (# of tokens)	Tokens
character (34)	t h e _ s t a r s _ t w i n k l e _ a n d _ s h i n e _ b r i g h t
BPE (21)	the, _, st, ar, s, _t, w, in, k, le, _, and, _s, h, in, e, _b, ri, g, h, t
BPE (token length $\leq 2$ ) (24)	_t, he, _s, t, ar, s, _t, w, in, k, l, e, _a, nd, _s, h, in, e, _b, r, i, g, h, t

Table 6: Tokenization for the sentence ‘‘The stars twinkle and shine bright’’ . We use character tokenizer for e2e-TTS and r-e2e-TTS while we use Byte Pair Encoding (BPE) for BPE- e2e-TTS and BPE- r-e2e-TTS .

Panel of Figure 4 — Figure 4: Alignment plots for both BPE- e2e-TTS and BPE- r-e2e-TTS show no observable differences.

Model	Generated (#)	EoS issues (#)
BPE- e2e-TTS	1715	4
BPE- r-e2e-TTS	1715	3

Table 7: Synthesized audio seq2seq EoS problem using BPE- e2e-TTS and BPE- r-e2e-TTS .

	Model	WER (%)	CER (%)
Whisper ( small.en )	BPE- e2e-TTS	9.6	3.4
Whisper ( small.en )	BPE- r-e2e-TTS	9.4	3.3
WhisperX ( small )	BPE- e2e-TTS	9.5	3.4
WhisperX ( small )	BPE- r-e2e-TTS	9.1	3.2

Table 8: WER and CER (lower values better) calculated on

1708

text samples on audio generated by BPE- e2e-TTS and BPE- r-e2e-TTS .

	Model	WER (%)	CER (%)
Whisper ( small.en )	e2e-TTS	12.36	6.37
Whisper ( small.en )	r-e2e-TTS	11.80	5.90
WhisperX ( small )	e2e-TTS	12.11	6.18
WhisperX ( small )	r-e2e-TTS	11.69	5.80

Table 9: WER and CER (lower values better) calculated on

1715

text samples using VITS-TTS architecture.

Panel of Figure 5 — Figure 5: Alignment plots for training rtfs-e2e-TTS . (a) Tacotron-2 , (b) VITS-TTS .

	Model	WER(%)	CER (%)
Whisper ( small.en )	rtfs-e2e-TTS	9	3
WhisperX ( small )	rtfs-e2e-TTS	8.31	3.73

Table 10: WER and CER (

\downarrow

better) calculated on 1711 text samples on audio generated by rtfs-e2e-TTS . Using Tacotron-2 .

TTS Arch	Text Embed	Model type	Whisper		WhisperX
			WER	CER	WER	CER
Tacotron-2	char	e2e-TTS	17.32	3.44	16.33	6.94
		r-e2e-TTS	10.79	3.28	10.69	3.9
		rtfs-e2e-TTS	9	3	8.31	3.73
VITS-TTS	char	e2e-TTS	12.36	6.37	12.11	6.18
		r-e2e-TTS	11.83	5.96	11.69	5.8

Table 11: Summary of Results from all the experiments.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Aspect	Tacotron-2 (2018)	VITS-TTS (2021)
Architecture	Two-stage (autoregressive)	Single-stage (non-autoregressive)
Structure	Encoder-decoder with attention (text to spectrogram)	Integrated prior encoder, posterior encoder, decoder
Structure	WaveGlow vocoder for raw audio from spectrogram	Duration predictor + discriminator
Efficiency	Slower (frame-wise generation)	Faster (parallel generation)
Alignment	Location-sensitive attention	Monotonic Alignment Search
Alignment	—	Normalizing flow framework

Appendix

Table 12: Comparison between Tacotron-2 and VITS-TTS models

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

End-to-End Training for Discrete Token LLM based TTS System
Jun 8, 2026Changfeng Gao, Yong Ren, Jun Yuan +3TTS Synthesis Speech Processing
Tencent Yuanbao, China
Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech
Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS Synthesis TTS Synthesis
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Oct 9, 2026Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1TTS Synthesis Expressive Speech Synthesis
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China

Jun 8, 2026J/K move · Enter open · S save

End-to-End Training for Discrete Token LLM based TTS System

Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai+2

Tencent Yuanbao, China

TTS Synthesis Speech Processing Audio Tokenization Discrete Speech Representation Learning

Open paper AskPDF arXiv

Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.

Abstract

TTS Synthesis

Explore similar work

End-to-End Training for Discrete Token LLM based TTS System

Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS