DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift

Mohammad Nur Hossain Khan; Subrata Biswas; Bashima Islam

cs.SDOct 2, 2026

DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift

Authors: Mohammad Nur Hossain Khan, Subrata Biswas, Bashima Islam

Organizations: University of Massachusetts Amherst, MA, USA · Worcester Polytechnic Institute, MA, USA

Abstract

Few-step neural text-to-speech models often rely on short- ened diffusion or flow-matching schedules, or on distillation from pretrained multi-step teachers. To avoid these depen- dencies, we present DriftTTS, a few-step mel-spectrogram generator trained without a generative teacher, distillation, or adversarial discrimination. DriftTTS uses a distribution- matching drift objective in a mel-domain feature space defined by raw mels and a frozen masked-autoencoder encoder pretrained on the same LJSpeech training split. On-policy rollout trains the decoder on its own interme- diate states and supports inference up to the trained roll- out depth. On LJSpeech, DriftTTS at NFE=4 achieves 3.87 dB MCD and 3.7% WER, compared with 3.85 dB and 3.4% for Matcha-TTS. In a fully paired blind listen- ing test, DriftTTS obtains 4.18 MOS, compared with 3.96 for Matcha-TTS and 4.22 for ground truth. These results demonstrate competitive few-step synthesis without a pre- trained generative teacher. Code can be found at https: //github.com/BASHLab/driftTTS.git

Figures & tables

Figure 1: Overview of DriftTTS. Text-conditioned Gaussian noise is iteratively refined by a few-step drift decoder to generate mel-spectrograms, which are converted to waveform by HiFi-GAN. During training, the decoder is optimized with an on-policy distribution-matching drift objective computed in a mel-domain feature space using real and generated samples.

System	NFE	WER $\downarrow$	MCD $\downarrow$	UTMOS $\uparrow$	MOS $\uparrow$
Ground truth	–	0.029	0.00	4.37	$4.22$
GT mel $\rightarrow$ vocoder	–	–	2.14	4.20	–
Matcha-TTS	1	0.042	4.79	4.25	–
Matcha-TTS	4	0.034	3.85	4.36	$3.96$
Grad-TTS	10	0.043	4.17	3.76	$3.55$
DriftTTS	1	0.036	4.89	3.16	$3.00$

Table 1: Evaluation results on LJSpeech. All synthesized systems use the same HiFi-GAN vocoder, and WER is computed using Whisper-medium. Listening-test statistics are reported in Table 2 .

System	NFE	MOS $\pm$ 95% CI
Ground truth	–	$4.22\pm 0.17$
DriftTTS	4	$\mathbf{4.18\pm 0.16}$
Matcha-TTS	4	$3.96\pm 0.18$
Grad-TTS	10	$3.55\pm 0.15$
DriftTTS	1	$3.00\pm 0.21$
Paired comparisons

Table 2: Blind naturalness MOS and paired comparisons. Two-sided paired t-tests are performed both across subjects and across utterances. CI was measured over subjects’ mean rating.

Configuration	MCD $\downarrow$	MCD Std. $\downarrow$	Worst MCD $\downarrow$
MelMAE + raw mel, drift	3.873	0.457	4.78
Raw mel only, drift	4.065	0.575	5.84
MelMAE + raw mel, L2	4.309	0.616	6.91

Table 3: Ablation of the drift objective and feature space. All models use the same architecture,

K=4

rollout, and 60k training updates. Lower is better for all metrics.

Trained $K$	MCD $\downarrow$	WER $\downarrow$	UTMOS $\uparrow$
2	4.76	0.0705	2.818
4	3.87	0.0373	4.253

Table 4: Effect of training rollout depth on synthesis quality and learned dynamics. Evaluated at

NFE=K

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting
Oct 16, 2026Jaehyun Ahn, Dohyun Kim, Heeseung KimTTS Synthesis One-Step Generative Modeling
Department of AI, University of Seoul, Seoul, South Korea
Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech
Sep 28, 2026Yentl Collin, Evan Dufraisse, Amr Mohamed +3Flow Matching TTS Synthesis
Institute of Foundation Models (IFM), MBZUAI
Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis
Jul 7, 2026Ho-Lam Chung, Kuan-Po Huang, Bo-Ru Lu +1Flow Matching TTS Synthesis
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan · Graduate Institute of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · Amazon

Oct 16, 2026J/K move · Enter open · S save

DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting

Jaehyun Ahn, Dohyun Kim, Heeseung Kim

Department of AI, University of Seoul, Seoul, South Korea

TTS Synthesis One-Step Generative Modeling Conditional Generation

Open paper AskPDF arXiv

Text-to-speech models built on score matching and flow matching produce natural speech, but only by calling the decoder many times, which makes synthesis slow. Existing ways to cut the number of calls either distill from a multi-step teacher or add a stage on top of a pretrained model, leaving single-step TTS without a pretrained TTS model unexplored. We question whether pretraining a TTS model is necessary, and propose DriftTTS, which adopts generative drifting, an objective that moves iterative sampling from inference into training. Drifting assumes samples of fixed shape, whereas speech has variable length; we address this issue with segment drifting. On LJSpeech, DriftTTS matches the multi-step quality of Matcha-TTS and RapFlow-TTS in a single decoder call, while being faster. To our knowledge, this is the first application of generative drifting to text-to-speech.

Abstract

TTS Synthesis

Explore similar work

DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis