Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

Jian Chen; You Zhang; Mark Vinton

eess.ASSep 29, 2026

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

Authors: Jian Chen, You Zhang, Mark Vinton

Organizations: Dolby Laboratories

Abstract

TTS systems with autoregressive semantic modeling have demonstrated strong zero-shot voice cloning performance and rich expressive variation, but their sequential decoding incurs substantial latency. Non-autoregressive alternatives offer much faster generation, yet often rely on more restrictive reference conditioning, such as requiring transcripts of the reference speech during inference. We present Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. Our model replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, introduces training-free acoustic length estimation, and accelerates the flow-matching renderer through ReFlow distillation. Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality while generating speech over 10x faster than IndexTTS2 for utterances longer than 5 seconds. Its transcript-free conditioning further supports cross-lingual and non-lexical references. We validate this capability using references from eight other languages, infant babble, and synthetic gibberish, where transcript-dependent systems often degrade or fail due to unreliable ASR transcripts.

Figures & tables

Figure 1 : Efficiency–speaker similarity trade-off of Tacit-TTS and baselines. Marker size indicates the number of generation-module parameters.

Figure 2 : Overview of the Tacit-TTS pipeline. Given a reference audio and target text, the encoders extract the conditioning representations, the text-to-semantic model generates semantic tokens via masked-LM generation, and the semantic-to-mel model converts them into a mel spectrogram for waveform synthesis by the vocoder.

	LibriSpeech test-clean		SeedTTS test-en		SeedTTS test-zh		AISHELL-1 test
Model	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	CER $\downarrow$	SS $\uparrow$	CER $\downarrow$
Autoregressive
CosyVoice2	0.843	5.999	0.794	3.277	0.846	1.451	0.834	1.967
SparkTTS	0.756	8.843	0.755	1.543	0.683	2.636	0.593	1.743
Non-autoregressive
MaskGCT	0.790	7.759	0.824	2.530	0.807	2.447	0.598	4.930

Table 1 : Zero-shot performance on the four datasets in speaker similarity (SS) and WER/CER. Bold : best per column among deployable systems (teacher and Ground Truth excluded). Results of all baselines are taken from the IndexTTS2 paper ( Zhou et al., 2026b ) .

	English targets			Chinese targets
Model	SS $\uparrow$	WER $\downarrow$	Fail $\downarrow$	SS $\uparrow$	CER $\downarrow$	Fail $\downarrow$
Transcript-dependent
F5-TTS	0.663 $\pm$ 0.120	24.56 $\pm$ 22.28	0.0 $\pm$ 0.0	0.622 $\pm$ 0.100	25.01 $\pm$ 20.89	0.0 $\pm$ 0.0
MaskGCT †	0.701 $\pm$ 0.065	20.16 $\pm$ 14.68	33.3 $\pm$ 47.1	0.700 $\pm$ 0.033	28.26 $\pm$ 14.32	33.3 $\pm$ 47.1
CosyVoice2	0.692 $\pm$ 0.081	45.95 $\pm$ 42.64	0.0 $\pm$ 0.0	0.680 $\pm$ 0.093	62.41 $\pm$ 47.33	0.0 $\pm$ 0.0
SparkTTS	0.611 $\pm$ 0.091	47.27 $\pm$ 42.23	20.0 $\pm$ 21.5	0.578 $\pm$ 0.092	60.42 $\pm$ 48.06	12.9 $\pm$ 16.9

Table 2 : Results of 6 methods using cross-lingual references, reading English and Chinese targets. Results are averaged over eight reference languages, with

\pm

indicating the standard deviation across languages.

\dagger

MaskGCT fails on all Russian and Arabic references, so we report averages over the six languages with successful generation.

	Infant babble				Synthetic gibberish
	English targets		Chinese targets		English targets		Chinese targets
Model	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	CER $\downarrow$	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	CER $\downarrow$
Transcript-dependent
SparkTTS	0.278	99.00	0.310	96.00	0.648	18.00	0.639	20.30
CosyVoice2	0.442	31.03	0.441	87.53	0.721	49.24	0.792	86.94
MaskGCT †	0.184	75.63	0.217	45.67	0.743	46.84	0.729	32.81

Table 3 : Results of 6 methods using non-lexical references, reading English and Chinese targets. † MaskGCT results are averaged over the 10 utterances that completed.

	Latency (ms)					Speed ( $\times$ real-time) $\uparrow$
Model	Cond.	ASR	T2S	S2A	Total	Cond.	Gen.	Total
Transcript-dependent
CosyVoice2	1082	918	3214	604	5818	3.03	1.56	$1.03$
MaskGCT	125	918	2428	2359	5830	5.88	1.27	$1.04$
SparkTTS ‡	57	918	7041	32	8049	6.67	0.92	$0.806$
F5-TTS †	599	315	1122		2036	5.88	4.76	$2.63$

Table 4 : Per-stage latency (ms) and speed in seconds of audio generated per second of compute (s/s, the inverse of real-time factor) for condition computation, ASR, generation (T2S

+

S2A), and the overall pipeline. † F5-TTS is single-stage; its T2S/S2A are not separable. ‡ SparkTTS means are over the 31/32 items that generated audio—on one AISHELL item its decoder emitted no semantic tokens.

Figure 3 : End-to-end generation time (including reference conditioning) vs. output duration for IndexTTS2 and Tacit-TTS, using one reference clip with a fixed passage repeated to lengthen the output. Dashed line: real time. Numbers above the Tacit-TTS curve are its speedup over the teacher; the inset zooms in on Tacit-TTS.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt recording	Pseudo-word sequence (target text)
voice_01	kalene kuguva ge yu mu nola vi pesa fa zula hukogu zuma ni na loze zela
voice_02	tuneho nide so gese wohele lu fine wadape vugofu yiba likive biye kiyume
voice_03	tuya sobazi de gawape wutiyu mumi zoyo ne sa nunaga ga guni wepuwe veka
voice_04	vivoha kafike mu poti nahu du ge vo mode bolo vuwe re di du yuta geguge
voice_05	kunu dugoyi mini pihi bo kamule gezi hu ruyara la pi mimi widavi bi le
voice_06	huvuba pu zu popari ze nazuvu peteva so nitafa nehi nebe sigi yosa le

Appendix

Table B.1 : The 10 synthetic-gibberish text prompt.

	LibriSpeech test-clean		SeedTTS test-en		SeedTTS test-zh		AISHELL-1 test
Model	UTMOS $\uparrow$	DNSMOS $\uparrow$	UTMOS $\uparrow$	DNSMOS $\uparrow$	UTMOS $\uparrow$	DNSMOS $\uparrow$	UTMOS $\uparrow$	DNSMOS $\uparrow$
Autoregressive
CosyVoice2	4.344	3.348	4.115	3.253	3.395	3.382	2.980	3.251
SparkTTS	4.017	3.065	3.903	3.133	3.271	3.266	2.864	3.081
Non-autoregressive
MaskGCT	3.903	3.296	3.484	3.106	2.549	3.241	2.343	3.218

Appendix

Table C.1 : Zero-shot performance on the four datasets in UTMOS and DNSMOS.

T2S	S2Mel	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	WER $\downarrow$	SS $\uparrow$	CER $\downarrow$	SS $\uparrow$	CER $\downarrow$
		English				Chinese				$1/\mathrm{RTF}$ $\uparrow$
		LibriSpeech test-clean		SeedTTS test-en		SeedTTS test-zh		AISHELL-1 test
8	1	0.710	7.11	0.682	2.32	0.570	3.25	0.530	3.37	17.9
8	4	0.860	6.56	0.837	2.05	0.800	3.12	0.784	3.24	18.5
8	8	0.872	6.64	0.849	2.13	0.825	3.14	0.799	3.17	14.3
12	1	0.711	5.78	0.684	2.36	0.575	1.81	0.535	2.39	19.2

Appendix

Table D.1 : SS, WER/CER, and efficiency on LibriSpeech test-clean (mean duration:

7.1

,s) across T2S/S2Mel sampling steps. The gray row denotes the default configuration. Efficiency is

1/\mathrm{RTF}

(

1\times

= real time), excluding one-time conditioning computation.

Figure F.1 : Generated vs. ground-truth speech duration across the four zero-shot test sets. Each panel reports the pooled Pearson correlation $r$ for one model.

	Japanese						Korean
	EN			ZH			EN			ZH
Model	SS $\uparrow$	WER $\downarrow$	Fail	SS $\uparrow$	CER $\downarrow$	Fail	SS $\uparrow$	WER $\downarrow$	Fail	SS $\uparrow$	CER $\downarrow$	Fail
Transcript-dependent
F5-TTS	0.408	38.86	0.0	0.406	40.99	0.0	0.648	31.33	0.0	0.643	23.59	0.0
MaskGCT	0.635	30.07	0.0	0.656	30.27	0.0	0.637	25.28	0.0	0.677	30.01	0.0
CosyVoice2	0.629	25.04	0.0	0.603	78.62	0.0	0.746	3.81	0.0	0.819	8.21	0.0

Appendix

Table G.1 : Per-language results of 6 methods using cross-lingual references, reading English (EN) and Chinese (ZH) targets. Fail: percentage of utterances with no generated speech.

\times

: no speech generated for that language. Bold : best SS and WER/CER per column.

Explore similar work

Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning
Sep 14, 2026Qingyu Liu, Rixi Xu, Yushen Chen +11Cross-Lingual Voice Cloning F5-Tts
Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China +3
Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder
Aug 12, 2026Huaxuan Wang, Huimin Wang, Ruiyu Zhang +2Cross-Lingual Voice Cloning Flow-Matching Text-To-Speech
NetEase Youdao, Beijing, China
X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning
May 7, 2026Rixi Xu, Qingyu Liu, Haitao Li +10Cross-Lingual Voice Cloning Multilingual Automatic Speech Recognition
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4

Sep 14, 2026, cs.SDJ/K move · Enter open · S save

Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning

Qingyu Liu, Rixi Xu, Yushen Chen, Zhikang Niu+10

Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China+3

Cross-Lingual Voice Cloning F5-Tts Speaker Similarity Transcript Unified Framework

Open paper AskPDF arXiv

Zero-shot text-to-speech (TTS) can clone a speaker's voice from a short audio prompt, yet most TTS systems still require the audio prompt transcript during inference. This dependency prevents cross-lingual voice cloning when the audio prompt transcript is unavailable, particularly for unseen languages. Cross-Lingual F5-TTS removes this dependency and enables transcript-free cross-lingual voice cloning, but it prepares its training data with forced alignment. Forced alignment is sensitive to boundary errors, and its cost grows as more languages are covered. Its speaking rate predictor is also unreliable at estimating duration when the audio prompt begins or ends with silence. In this paper, we present Cross-Lingual F5-TTS 2, a simplified framework for transcript-free cross-lingual voice cloning without forced alignment. Instead of using forced alignment to segment real utterances, we build same-speaker prompt and target pairs using a pretrained F5-TTS model and fine-tune the same model on these constructed pairs. This simplifies data preparation and preserves the acoustic modeling capability of the pretrained model, enabling adaptation with only a short fine-tuning stage. We further make the syllable-level speaking rate predictor robust to leading and trailing silence through silence-aware augmentation. Experiments show that Cross-Lingual F5-TTS 2 reaches higher speaker similarity than F5-TTS and Cross-Lingual F5-TTS while maintaining intelligibility. All related resources are publicly available.

Abstract

Explore similar work

Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning

Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning